NV+ V2 stalling when 80%+ full (20047616)

Converting to EXT3 will help. Again thank you NETGEAR for quality control.

You need to download your data because it will be wiped (from what I've been told). Yes, you need to give them root OR add drives to get below 80%.. I did both ...

I'm looking into this issue, from the logs I see that you converted to ext3 yourself, and you're still running into the issue, right?

I'm chasing down a few things, could you confirm, are you storing a large amount of small files on the device (checked volume stat, if my math is correct, average of 3.5MB per file)?
Are you only using CIFS/SMB to copy files to the NAS?

Skywarp wrote:
I'm looking into this issue, from the logs I see that you converted to ext3 yourself, and you're still running into the issue, right? I'm chasing down a few things, could you confirm, are you storing a large amount of small files on the device (checked volume stat, if my math is correct, average of 3.5MB per file)? Are you only using CIFS/SMB to copy files to the NAS?

I have same problems... converting volume to ext3 solve everything. This problem is related only to filesystem itself not to anything else.
This is propably kernel bug, somewhere at journaling code. I found it on Synology and QNAP forums and people are very angry because of this.
There's nothing in logs only IOWaits raises to 100% after any write is performed and system is not responsive for some seconds/speed drops to 10MB/s or lower.
Maybe disabling journaling in ext4 can help, but I don't trust this filesystem for now.

I saw this problem when volume is aprox. 85% full and I have a lot small files on it.

When you format ext3 volume you can switch to ext4 driver and in that situation everything goes wrong.
Journal must be recreated while you are going back to ext3 - if not, ext3 have same problems as ext4 (100% IOWaits).

Perfomace with nearly full volume on ext3 is ~50MB/s write ~80MB/s read - not bad for ARM platform.

Skywarp wrote:
I'm looking into this issue, from the logs I see that you converted to ext3 yourself, and you're still running into the issue, right? I'm chasing down a few things, could you confirm, are you storing a large amount of small files on the device (checked volume stat, if my math is correct, average of 3.5MB per file)? Are you only using CIFS/SMB to copy files to the NAS?

yeah i converted to ext3 myself "mkfs.ext3 -m 0 /dev/c/c" and still have the issue, last night i upgraded to 5.3.7 as it adds an option in the web GUI to disable write caching and am trying to fill drive again past 80% and hopefully to 99% but am not very hopefull.

yeah im fairly sure most of my files are farily small, only a few larger files.

im only using CIFS/SMB to copy via the lan, im using rsync for some remote backup from another site to here.

have been trying to figgure out if NCQ is enabled on the device as `cat /sys/block/sda/device/queue_depth` results "31" (suggesting it's on) and trying to set it to "1" results in error.
also trying to figgure out why/how to stop the segate drives "chirping" as they load/unload the heads periodaccly as after ~1month my Load Cycle Count is over 10k for each drive as reported in smart.

EDIT:
Currently with firmware 5.3.7 and write cache turned off via the GUI i've managed to copy about 180gb more to the drive with some acceptable speed but after that mark im getting about 1minute of IO Wait before starting to copy a file and then more IO Wait during copying the file to the drive. the test set of data have files between 1gb to 3gb files. currently at about 875GB free space reported.

janforman wrote:

Skywarp wrote: I'm looking into this issue, from the logs I see that you converted to ext3 yourself, and you're still running into the issue, right? I'm chasing down a few things, could you confirm, are you storing a large amount of small files on the device (checked volume stat, if my math is correct, average of 3.5MB per file)? Are you only using CIFS/SMB to copy files to the NAS? I have same problems... converting volume to ext3 solve everything. This problem is related only to filesystem itself not to anything else. This is propably kernel bug, somewhere at journaling code. I found it on Synology and QNAP forums and people are very angry because of this. There's nothing in logs only IOWaits raises to 100% after any write is performed and system is not responsive for some seconds/speed drops to 10MB/s or lower. Maybe disabling journaling in ext4 can help, but I don't trust this filesystem for now. I saw this problem when volume is aprox. 85% full and I have a lot small files on it. When you format ext3 volume you can switch to ext4 driver and in that situation everything goes wrong. Journal must be recreated while you are going back to ext3 - if not, ext3 have same problems as ext4 (100% IOWaits). Perfomace with nearly full volume on ext3 is ~50MB/s write ~80MB/s read - not bad for ARM platform.

i did a fresh format of the volume to ext3 so shoudn't have any left overs from the ext4 setup. i first had issues with ext4 at ~540gb left converting to ext3 started giving issues now at around 1.1tb free, so it has to be something more with a bad kernel/driver build from NETGEAR i would assume. never had any of these issues with any of my other linux systems approaching full disks ever! and i've used ext3/4 for all of them.

edit: i also only get about 20-25mb/sec write to the device typically even for large files ~2gb. but that's good enough write speeds for me, if i wanted faster i would just use internal disks.

hauser wrote:

janforman wrote: Skywarp wrote: I'm looking into this issue, from the logs I see that you converted to ext3 yourself, and you're still running into the issue, right? I'm chasing down a few things, could you confirm, are you storing a large amount of small files on the device (checked volume stat, if my math is correct, average of 3.5MB per file)? Are you only using CIFS/SMB to copy files to the NAS? I have same problems... converting volume to ext3 solve everything. This problem is related only to filesystem itself not to anything else. This is propably kernel bug, somewhere at journaling code. I found it on Synology and QNAP forums and people are very angry because of this. There's nothing in logs only IOWaits raises to 100% after any write is performed and system is not responsive for some seconds/speed drops to 10MB/s or lower. Maybe disabling journaling in ext4 can help, but I don't trust this filesystem for now. I saw this problem when volume is aprox. 85% full and I have a lot small files on it. When you format ext3 volume you can switch to ext4 driver and in that situation everything goes wrong. Journal must be recreated while you are going back to ext3 - if not, ext3 have same problems as ext4 (100% IOWaits). Perfomace with nearly full volume on ext3 is ~50MB/s write ~80MB/s read - not bad for ARM platform. i did a fresh format of the volume to ext3 so shoudn't have any left overs from the ext4 setup. i first had issues with ext4 at ~540gb left converting to ext3 started giving issues now at around 1.1tb free, so it has to be something more with a bad kernel/driver build from NETGEAR i would assume. never had any of these issues with any of my other linux systems approaching full disks ever! and i've used ext3/4 for all of them. edit: i also only get about 20-25mb/sec write to the device typically even for large files ~2gb. but that's good enough write speeds for me, if i wanted faster i would just use internal disks.

Are you using ext3 driver?
If you mount ext3 partition with ext4 driver problem still exist under ext3 driver (you must drop and recreate journal).
I have 50MB/s write performance (my hardrive cannot do more - it's his limit)

My system utilization is now:
10% CPU, 66% System, 3% IO Waits / 50MB/s (3GB data) constant writing / NAS utilization - WD Green AV-GP HDD (when I put this drive to my PC I cannot get more than 50MB/s)
RAIDiator 5.3.7 WriteCache ON pretty cool :-)

But If I accidentaly mount this volume with ext4 driver and go back to ext3 driver without droping journal my system shows IOWaits 100% and freezing and freezing for seconds (same way as in ext4 driver)...

janforman wrote:

hauser wrote: janforman wrote: Skywarp wrote: I'm looking into this issue, from the logs I see that you converted to ext3 yourself, and you're still running into the issue, right? I'm chasing down a few things, could you confirm, are you storing a large amount of small files on the device (checked volume stat, if my math is correct, average of 3.5MB per file)? Are you only using CIFS/SMB to copy files to the NAS? I have same problems... converting volume to ext3 solve everything. This problem is related only to filesystem itself not to anything else. This is propably kernel bug, somewhere at journaling code. I found it on Synology and QNAP forums and people are very angry because of this. There's nothing in logs only IOWaits raises to 100% after any write is performed and system is not responsive for some seconds/speed drops to 10MB/s or lower. Maybe disabling journaling in ext4 can help, but I don't trust this filesystem for now. I saw this problem when volume is aprox. 85% full and I have a lot small files on it. When you format ext3 volume you can switch to ext4 driver and in that situation everything goes wrong. Journal must be recreated while you are going back to ext3 - if not, ext3 have same problems as ext4 (100% IOWaits). Perfomace with nearly full volume on ext3 is ~50MB/s write ~80MB/s read - not bad for ARM platform. i did a fresh format of the volume to ext3 so shoudn't have any left overs from the ext4 setup. i first had issues with ext4 at ~540gb left converting to ext3 started giving issues now at around 1.1tb free, so it has to be something more with a bad kernel/driver build from NETGEAR i would assume. never had any of these issues with any of my other linux systems approaching full disks ever! and i've used ext3/4 for all of them. edit: i also only get about 20-25mb/sec write to the device typically even for large files ~2gb. but that's good enough write speeds for me, if i wanted faster i would just use internal disks. Are you using ext3 driver? If you mount ext3 partition with ext4 driver problem still exist under ext3 driver (you must drop and recreate journal). I have 50MB/s write performance (my hardrive cannot do more - it's his limit) My system utilization is now: 10% CPU, 66% System, 3% IO Waits / 50MB/s constant writing (3GB data) - WD Green AV-GP HDD (when I put this drive to my PC I cannot get more than 50MB/s) RAIDiator 5.3.7 WriteCache ON pretty cool :-)

process i used to drop to ext3:
vi /etc/fstab -- changed: ext4 to ext3 for mountpoint /c
reboot
mkfs.ext3 -m 0 /dev/c/c
reboot

so i would assume that it's a fully fresh ext3 volume journal included as that's what mkfs does -m 0 just sets the root reserved blocks to 0% at time of creation so don't have to tune2fs it down to 0% from the default of 5%

hauser wrote:
process i used to drop to ext3: vi /etc/fstab -- changed: ext4 to ext3 for mountpoint /c reboot mkfs.ext3 -m 0 /dev/c/c reboot so i would assume that it's a fully fresh ext3 volume journal included as that's what mkfs does -m 0 just sets the root reserved blocks to 0% at time of creation so don't have to tune2fs it down to 0% from the default of 5%

That's what I do and after that I have IOWaits on values I posted.

My original path:
First I saw this problem on default ext4 so I reformat it to ext3 and it worked ... after that for testing purposes I mount ext3 with ext4 driver and then system is bad as default.
Okay I'm clever and remounted this volume as ext3 back, but problem persist. WTF - I think ... Why? Then I drop journal and recreate him and voala problem disappeared.

I'm not sure, but I think that there is some bug in journaling, because hardrive permanently writing something (i can hear it by myself HDD LED is flashing) -> high IOWaits
When I touch this volume anyway with ext4 driver I have utilization IOWaits on 100% even when I go back to ext3.

Part with recreating journal I tried many times just to be sure.
default:
ext4 -> problem 100%IO
modified/reformatted:
ext3 driver 5%IO-> ext4 driver 100%IO -> ext3 driver 100%IO (shock for me)
removing journal -> ext3 -> ok 5%IO -> ... and again and again / same results

10% CPU, 66% System, 3% IO Waits / 50MB/s constant writing (3GB data) so this is what I have now with ext3, but of course filescan taking ages.
Maybe NAS can do even more, but with my HDD this is maximum when I benchmarked this HDD in my PC than I have (~120MB/s read, ~50MB/s write)
-> before it's like this 6% CPU, 10% System, 99% IO Waits (not exactly data just what I remember - I don't want to test it again)

Found another bug/glitch/weirdness with the unit, if i've got a backup of my firefox cache folder on the nas (47.9 MB (50,267,577 bytes), 529 Files, 2,799 Folders) IOWait goes really high continously and times out some times. Removing this collection of folders and copy starts again at the normal speed. maybe there's an issue with number of folders on EXT3 or the file system driver in use in the firmware. i've got about ~1.13mil files and about 137k folders on the drive currently.

currently i've gotten to 94% used (336G free of 5.5Tb) still trying to see how far i can fill the unit before it has an issue again.

currently with the nas modifyed as below

Firmware: 5.3.7
Write Cache=OFF
/dev/c/c formatted as EXT3 
tune2fs -O ^dir_index /dev/c/c

tune2fs -O ^dir_index turns off B-tree indexing for the volume. so far it's never had any real advatages to file systems unless you are reading only and you know the file in a directory you haven't browsed to yet and are directly accessing that file.
dir_index can be turned on again by

tune2fs -O dir_index /dev/c/c
umount /dev/c/c
e2fsck -fD /dev/c/c

seems to be working a little better for me than stock so far. waiting to see how full i can get before any more issues, if i an fill to <1gb free i'll be happyer. but if there is a limit to the number of folders able to be stored on the nas, as with having my Firefox profile backup it pushes total folder count over 140k. EXT3 has a limit of 31,998 sub-directories in any single folder so it shouldn't be that and there is still heaps of free inodes (89,921,695 of 91,185,152 used) it would seem like there is yet another bug with NETGEAR's firmware.

EDIT: with all the above changes i've just been able to fill the drive to ~30mb free. at which point (expectably IOWait skyrockets) but during the last ~15% of filling the drive IOWaits have been getting quite long when starting to write largish files (1-3gb)

Hey guys, I started running into this stalling issue at about 85% and pretty much can't add anymore data to the device.
Sounds like I'm going to have to remove all the data before I can fix it correct?
I'm a bit new to this, but how do I convert the file format to ext3? Is that done within the web interface?

Thanks