i've recently purchased one of the Netgear ReadyNAS NV+ V2 units and have become very unhappy with it. it started out ok and could copy files to it at good rate, now that the unit is over 80% full i've been coming accross the smbd stall on IO Wait issue that seems to plague this unit 😞
after coming to these fourms i found that there are many other users with the same issues and none of the solutions they have had for them posted around converting to EXT3 has made the copy issues any better for me.
i have linked the other topics i could find with same issues that have been plagueing me so far. i have logged a support case (20047616) with netgear but their "remote access policy" is very bad and i can't agree to it.
Converting to EXT3 will help. Again thank you NETGEAR for quality control.
You need to download your data because it will be wiped (from what I've been told). Yes, you need to give them root OR add drives to get below 80%.. I did both ...
I'm looking into this issue, from the logs I see that you converted to ext3 yourself, and you're still running into the issue, right?
I'm chasing down a few things, could you confirm, are you storing a large amount of small files on the device (checked volume stat, if my math is correct, average of 3.5MB per file)? Are you only using CIFS/SMB to copy files to the NAS?
I'm looking into this issue, from the logs I see that you converted to ext3 yourself, and you're still running into the issue, right?
I'm chasing down a few things, could you confirm, are you storing a large amount of small files on the device (checked volume stat, if my math is correct, average of 3.5MB per file)? Are you only using CIFS/SMB to copy files to the NAS?
I have same problems... converting volume to ext3 solve everything. This problem is related only to filesystem itself not to anything else. This is propably kernel bug, somewhere at journaling code. I found it on Synology and QNAP forums and people are very angry because of this. There's nothing in logs only IOWaits raises to 100% after any write is performed and system is not responsive for some seconds/speed drops to 10MB/s or lower. Maybe disabling journaling in ext4 can help, but I don't trust this filesystem for now.
I saw this problem when volume is aprox. 85% full and I have a lot small files on it.
When you format ext3 volume you can switch to ext4 driver and in that situation everything goes wrong. Journal must be recreated while you are going back to ext3 - if not, ext3 have same problems as ext4 (100% IOWaits).
Perfomace with nearly full volume on ext3 is ~50MB/s write ~80MB/s read - not bad for ARM platform.
I'm looking into this issue, from the logs I see that you converted to ext3 yourself, and you're still running into the issue, right?
I'm chasing down a few things, could you confirm, are you storing a large amount of small files on the device (checked volume stat, if my math is correct, average of 3.5MB per file)? Are you only using CIFS/SMB to copy files to the NAS?
yeah i converted to ext3 myself "mkfs.ext3 -m 0 /dev/c/c" and still have the issue, last night i upgraded to 5.3.7 as it adds an option in the web GUI to disable write caching and am trying to fill drive again past 80% and hopefully to 99% but am not very hopefull.
yeah im fairly sure most of my files are farily small, only a few larger files.
im only using CIFS/SMB to copy via the lan, im using rsync for some remote backup from another site to here.
have been trying to figgure out if NCQ is enabled on the device as `cat /sys/block/sda/device/queue_depth` results "31" (suggesting it's on) and trying to set it to "1" results in error. also trying to figgure out why/how to stop the segate drives "chirping" as they load/unload the heads periodaccly as after ~1month my Load Cycle Count is over 10k for each drive as reported in smart.
EDIT: Currently with firmware 5.3.7 and write cache turned off via the GUI i've managed to copy about 180gb more to the drive with some acceptable speed but after that mark im getting about 1minute of IO Wait before starting to copy a file and then more IO Wait during copying the file to the drive. the test set of data have files between 1gb to 3gb files. currently at about 875GB free space reported.
I'm looking into this issue, from the logs I see that you converted to ext3 yourself, and you're still running into the issue, right?
I'm chasing down a few things, could you confirm, are you storing a large amount of small files on the device (checked volume stat, if my math is correct, average of 3.5MB per file)? Are you only using CIFS/SMB to copy files to the NAS?
I have same problems... converting volume to ext3 solve everything. This problem is related only to filesystem itself not to anything else. This is propably kernel bug, somewhere at journaling code. I found it on Synology and QNAP forums and people are very angry because of this. There's nothing in logs only IOWaits raises to 100% after any write is performed and system is not responsive for some seconds/speed drops to 10MB/s or lower. Maybe disabling journaling in ext4 can help, but I don't trust this filesystem for now.
I saw this problem when volume is aprox. 85% full and I have a lot small files on it.
When you format ext3 volume you can switch to ext4 driver and in that situation everything goes wrong. Journal must be recreated while you are going back to ext3 - if not, ext3 have same problems as ext4 (100% IOWaits).
Perfomace with nearly full volume on ext3 is ~50MB/s write ~80MB/s read - not bad for ARM platform.
i did a fresh format of the volume to ext3 so shoudn't have any left overs from the ext4 setup. i first had issues with ext4 at ~540gb left converting to ext3 started giving issues now at around 1.1tb free, so it has to be something more with a bad kernel/driver build from NETGEAR i would assume. never had any of these issues with any of my other linux systems approaching full disks ever! and i've used ext3/4 for all of them.
edit: i also only get about 20-25mb/sec write to the device typically even for large files ~2gb. but that's good enough write speeds for me, if i wanted faster i would just use internal disks.
I'm looking into this issue, from the logs I see that you converted to ext3 yourself, and you're still running into the issue, right?
I'm chasing down a few things, could you confirm, are you storing a large amount of small files on the device (checked volume stat, if my math is correct, average of 3.5MB per file)? Are you only using CIFS/SMB to copy files to the NAS?
I have same problems... converting volume to ext3 solve everything. This problem is related only to filesystem itself not to anything else. This is propably kernel bug, somewhere at journaling code. I found it on Synology and QNAP forums and people are very angry because of this. There's nothing in logs only IOWaits raises to 100% after any write is performed and system is not responsive for some seconds/speed drops to 10MB/s or lower. Maybe disabling journaling in ext4 can help, but I don't trust this filesystem for now.
I saw this problem when volume is aprox. 85% full and I have a lot small files on it.
When you format ext3 volume you can switch to ext4 driver and in that situation everything goes wrong. Journal must be recreated while you are going back to ext3 - if not, ext3 have same problems as ext4 (100% IOWaits).
Perfomace with nearly full volume on ext3 is ~50MB/s write ~80MB/s read - not bad for ARM platform.
i did a fresh format of the volume to ext3 so shoudn't have any left overs from the ext4 setup. i first had issues with ext4 at ~540gb left converting to ext3 started giving issues now at around 1.1tb free, so it has to be something more with a bad kernel/driver build from NETGEAR i would assume. never had any of these issues with any of my other linux systems approaching full disks ever! and i've used ext3/4 for all of them.
edit: i also only get about 20-25mb/sec write to the device typically even for large files ~2gb. but that's good enough write speeds for me, if i wanted faster i would just use internal disks.
Are you using ext3 driver? If you mount ext3 partition with ext4 driver problem still exist under ext3 driver (you must drop and recreate journal). I have 50MB/s write performance (my hardrive cannot do more - it's his limit)
My system utilization is now: 10% CPU, 66% System, 3% IO Waits / 50MB/s (3GB data) constant writing / NAS utilization - WD Green AV-GP HDD (when I put this drive to my PC I cannot get more than 50MB/s) RAIDiator 5.3.7 WriteCache ON pretty cool 🙂
But If I accidentaly mount this volume with ext4 driver and go back to ext3 driver without droping journal my system shows IOWaits 100% and freezing and freezing for seconds (same way as in ext4 driver)...
I'm looking into this issue, from the logs I see that you converted to ext3 yourself, and you're still running into the issue, right?
I'm chasing down a few things, could you confirm, are you storing a large amount of small files on the device (checked volume stat, if my math is correct, average of 3.5MB per file)? Are you only using CIFS/SMB to copy files to the NAS?
I have same problems... converting volume to ext3 solve everything. This problem is related only to filesystem itself not to anything else. This is propably kernel bug, somewhere at journaling code. I found it on Synology and QNAP forums and people are very angry because of this. There's nothing in logs only IOWaits raises to 100% after any write is performed and system is not responsive for some seconds/speed drops to 10MB/s or lower. Maybe disabling journaling in ext4 can help, but I don't trust this filesystem for now.
I saw this problem when volume is aprox. 85% full and I have a lot small files on it.
When you format ext3 volume you can switch to ext4 driver and in that situation everything goes wrong. Journal must be recreated while you are going back to ext3 - if not, ext3 have same problems as ext4 (100% IOWaits).
Perfomace with nearly full volume on ext3 is ~50MB/s write ~80MB/s read - not bad for ARM platform.
i did a fresh format of the volume to ext3 so shoudn't have any left overs from the ext4 setup. i first had issues with ext4 at ~540gb left converting to ext3 started giving issues now at around 1.1tb free, so it has to be something more with a bad kernel/driver build from NETGEAR i would assume. never had any of these issues with any of my other linux systems approaching full disks ever! and i've used ext3/4 for all of them.
edit: i also only get about 20-25mb/sec write to the device typically even for large files ~2gb. but that's good enough write speeds for me, if i wanted faster i would just use internal disks.
Are you using ext3 driver? If you mount ext3 partition with ext4 driver problem still exist under ext3 driver (you must drop and recreate journal). I have 50MB/s write performance (my hardrive cannot do more - it's his limit)
My system utilization is now: 10% CPU, 66% System, 3% IO Waits / 50MB/s constant writing (3GB data) - WD Green AV-GP HDD (when I put this drive to my PC I cannot get more than 50MB/s) RAIDiator 5.3.7 WriteCache ON pretty cool 🙂
process i used to drop to ext3: vi /etc/fstab -- changed: ext4 to ext3 for mountpoint /c reboot mkfs.ext3 -m 0 /dev/c/c reboot
so i would assume that it's a fully fresh ext3 volume journal included as that's what mkfs does -m 0 just sets the root reserved blocks to 0% at time of creation so don't have to tune2fs it down to 0% from the default of 5%
process i used to drop to ext3: vi /etc/fstab -- changed: ext4 to ext3 for mountpoint /c reboot mkfs.ext3 -m 0 /dev/c/c reboot
so i would assume that it's a fully fresh ext3 volume journal included as that's what mkfs does -m 0 just sets the root reserved blocks to 0% at time of creation so don't have to tune2fs it down to 0% from the default of 5%
That's what I do and after that I have IOWaits on values I posted.
My original path: First I saw this problem on default ext4 so I reformat it to ext3 and it worked ... after that for testing purposes I mount ext3 with ext4 driver and then system is bad as default. Okay I'm clever and remounted this volume as ext3 back, but problem persist. WTF - I think ... Why? Then I drop journal and recreate him and voala problem disappeared.
I'm not sure, but I think that there is some bug in journaling, because hardrive permanently writing something (i can hear it by myself HDD LED is flashing) -> high IOWaits When I touch this volume anyway with ext4 driver I have utilization IOWaits on 100% even when I go back to ext3.
Part with recreating journal I tried many times just to be sure. default: ext4 -> problem 100%IO modified/reformatted: ext3 driver 5%IO-> ext4 driver 100%IO -> ext3 driver 100%IO (shock for me) removing journal -> ext3 -> ok 5%IO -> ... and again and again / same results
10% CPU, 66% System, 3% IO Waits / 50MB/s constant writing (3GB data) so this is what I have now with ext3, but of course filescan taking ages. Maybe NAS can do even more, but with my HDD this is maximum when I benchmarked this HDD in my PC than I have (~120MB/s read, ~50MB/s write) -> before it's like this 6% CPU, 10% System, 99% IO Waits (not exactly data just what I remember - I don't want to test it again)
Found another bug/glitch/weirdness with the unit, if i've got a backup of my firefox cache folder on the nas (47.9 MB (50,267,577 bytes), 529 Files, 2,799 Folders) IOWait goes really high continously and times out some times. Removing this collection of folders and copy starts again at the normal speed. maybe there's an issue with number of folders on EXT3 or the file system driver in use in the firmware. i've got about ~1.13mil files and about 137k folders on the drive currently.
currently i've gotten to 94% used (336G free of 5.5Tb) still trying to see how far i can fill the unit before it has an issue again.
tune2fs -O ^dir_index turns off B-tree indexing for the volume. so far it's never had any real advatages to file systems unless you are reading only and you know the file in a directory you haven't browsed to yet and are directly accessing that file. dir_index can be turned on again by
seems to be working a little better for me than stock so far. waiting to see how full i can get before any more issues, if i an fill to <1gb free i'll be happyer. but if there is a limit to the number of folders able to be stored on the nas, as with having my Firefox profile backup it pushes total folder count over 140k. EXT3 has a limit of 31,998 sub-directories in any single folder so it shouldn't be that and there is still heaps of free inodes (89,921,695 of 91,185,152 used) it would seem like there is yet another bug with NETGEAR's firmware.
EDIT: with all the above changes i've just been able to fill the drive to ~30mb free. at which point (expectably IOWait skyrockets) but during the last ~15% of filling the drive IOWaits have been getting quite long when starting to write largish files (1-3gb)
Hey guys, I started running into this stalling issue at about 85% and pretty much can't add anymore data to the device. Sounds like I'm going to have to remove all the data before I can fix it correct? I'm a bit new to this, but how do I convert the file format to ext3? Is that done within the web interface?
Hey guys, I started running into this stalling issue at about 85% and pretty much can't add anymore data to the device. Sounds like I'm going to have to remove all the data before I can fix it correct? I'm a bit new to this, but how do I convert the file format to ext3? Is that done within the web interface?
Thanks
for me i did the below to get it to work better for me (after removing/backing up data on the drives)
1) Update firmware to 5.3.7 2) in the webGUI under System -> Settings -> Performance untick "Enable disk write cache" 3) use enable root ssh addon http://www.readynas.com/?p=4203 4) ssh into nas and edit /etc/fstab (vi /etc/fstab) change: EXT4 to EXT3 for mountpoint /c and save file (:wq) 5) reboot 6) mkfs.ext3 -m 0 /dev/c/c 7) tune2fs -O ^dir_index /dev/c/c 😎 reboot 9) copy files to nas
effectivly that's what it summates to of all of what i've done to the device since getting it to get it to the point today where i could fill it untill it had ~30mb left free still have an issue with delays in starting to copy larger files to the drive when it's 85%+ full, but it let me fill the drive.
Found another bug/glitch/weirdness with the unit, if i've got a backup of my firefox cache folder on the nas (47.9 MB (50,267,577 bytes), 529 Files, 2,799 Folders) IOWait goes really high continously and times out some times. Removing this collection of folders and copy starts again at the normal speed. maybe there's an issue with number of folders on EXT3 or the file system driver in use in the firmware. i've got about ~1.13mil files and about 137k folders on the drive currently.
currently i've gotten to 94% used (336G free of 5.5Tb) still trying to see how far i can fill the unit before it has an issue again.
tune2fs -O ^dir_index turns off B-tree indexing for the volume. so far it's never had any real advatages to file systems unless you are reading only and you know the file in a directory you haven't browsed to yet and are directly accessing that file. dir_index can be turned on again by
seems to be working a little better for me than stock so far. waiting to see how full i can get before any more issues, if i an fill to <1gb free i'll be happyer. but if there is a limit to the number of folders able to be stored on the nas, as with having my Firefox profile backup it pushes total folder count over 140k. EXT3 has a limit of 31,998 sub-directories in any single folder so it shouldn't be that and there is still heaps of free inodes (89,921,695 of 91,185,152 used) it would seem like there is yet another bug with NETGEAR's firmware.
EDIT: with all the above changes i've just been able to fill the drive to ~30mb free. at which point (expectably IOWait skyrockets) but during the last ~15% of filling the drive IOWaits have been getting quite long when starting to write largish files (1-3gb)
Wau, so extreme case :lol:, but turning off directory index and write cache isn't good... this must hurt normal performance so much. 30MB free on volume is absolutely crazy situation.
Anyway you can try droping journal in filesystem and mount it as ext2 (without data loss). :rofl:
yeah, i could drop journal as i am running it via UPS, but it's working atm. 30mb was filling as a test to see if i can fill without it freezing like before.
"normal" performance for me currently is ~25mb/sec write ~65mb/sec read. might be nice to have faster write, but for my useage pattern it's enough for now.
Hey guys, I started running into this stalling issue at about 85% and pretty much can't add anymore data to the device. Sounds like I'm going to have to remove all the data before I can fix it correct? I'm a bit new to this, but how do I convert the file format to ext3? Is that done within the web interface?
Thanks
for me i did the below to get it to work better for me (after removing/backing up data on the drives)
1) Update firmware to 5.3.7 2) in the webGUI under System -> Settings -> Performance untick "Enable disk write cache" 3) use enable root ssh addon http://www.readynas.com/?p=4203 4) ssh into nas and edit /etc/fstab (vi /etc/fstab) change: EXT4 to EXT3 for mountpoint /c and save file (:wq) 5) reboot 6) mkfs.ext3 -m 0 /dev/c/c 7) tune2fs -O ^dir_index /dev/c/c 😎 reboot 9) copy files to nas
effectivly that's what it summates to of all of what i've done to the device since getting it to get it to the point today where i could fill it untill it had ~30mb left free still have an issue with delays in starting to copy larger files to the drive when it's 85%+ full, but it let me fill the drive.
Cool, It will be a little bit before I can try this, I have to get some new hard drives to backup the NAS (7 TB). What is dropping the journal? is that something I should do as well?
dropping the journal is something you should only do if you understand the risks involved. most notably if you loose power while writing data to the drive you can risk loosing more than just the file you are writing to at that time. at least with a journal typically you only loose data that hasn't been safely commited to the journal/file system as it creates effectivly a audit trail of what data got changed so the system can replay after power outage to get back to a healthy system.
simplest way woudl be to use ex2 instead of ex3 or you could tell mkfs to not create a journal, but it's probably better to just keep the journal unless your prepaird to nurse a faulty file system back to health yourself as none of what's suggested in this thread is supported by NETGEAR afaik.
info posted within this thread is based on personal experinces and your experince following any of this may not result in the same outcomes that we had. so if your not confident in what your doing and you could somehow "trust" NEGEAR's terrible "remote access policy" they might be able to assist or fix it for you.
dropping the journal is something you should only do if you understand the risks involved. most notably if you loose power while writing data to the drive you can risk loosing more than just the file you are writing to at that time. at least with a journal typically you only loose data that hasn't been safely commited to the journal/file system as it creates effectivly a audit trail of what data got changed so the system can replay after power outage to get back to a healthy system.
simplest way woudl be to use ex2 instead of ex3 or you could tell mkfs to not create a journal, but it's probably better to just keep the journal unless your prepaird to nurse a faulty file system back to health yourself as none of what's suggested in this thread is supported by NETGEAR afaik.
info posted within this thread is based on personal experinces and your experince following any of this may not result in the same outcomes that we had. so if your not confident in what your doing and you could somehow "trust" NEGEAR's terrible "remote access policy" they might be able to assist or fix it for you.
Thanks for the explanation. I'll forget about the journal then.
I have the same issue which has been elevated to Level 3 support still waiting for a reply. From what is reported here I can expect the answer to be "allow root access and let us set the file system up as ext3" So I have started this myself following the "instructions" in this post. I assume that XRAID 2 expansion operates as normal with an ext3 partition on the V2?
I have the same issue which has been elevated to Level 3 support still waiting for a reply. From what is reported here I can expect the answer to be "allow root access and let us set the file system up as ext3" So I have started this myself following the "instructions" in this post. I assume that XRAID 2 expansion operates as normal with an ext3 partition on the V2?
As I know there are always EXT4 on V2 (ARM) devices... only system partitions are EXT3.
I have the same issue which has been elevated to Level 3 support still waiting for a reply. From what is reported here I can expect the answer to be "allow root access and let us set the file system up as ext3" So I have started this myself following the "instructions" in this post. I assume that XRAID 2 expansion operates as normal with an ext3 partition on the V2?
as far as i know XRAID-2 and Flex-RAID should work the same as it's a low level disk raid format and doesn't care what's stored on it, im never planning on putting bigger disks in this nas so i've just gone with Flex-RAID. i figgure i've had this much trouble with it so far im not going to be spending more money on it when a drive fails. that and i'll never be buying another NETGEAR anything after this unit...
I have the same issue which has been elevated to Level 3 support still waiting for a reply. From what is reported here I can expect the answer to be "allow root access and let us set the file system up as ext3" So I have started this myself following the "instructions" in this post. I assume that XRAID 2 expansion operates as normal with an ext3 partition on the V2?
as far as i know XRAID-2 and Flex-RAID should work the same as it's a low level disk raid format and doesn't care what's stored on it, im never planning on putting bigger disks in this nas so i've just gone with Flex-RAID. i figgure i've had this much trouble with it so far im not going to be spending more money on it when a drive fails. that and i'll never be buying another NETGEAR anything after this unit...
But who...? I have IBM Enterprise San, Synology-Rack-NAS and everything seems like joke to me 😄 Thanks to full SSH access to Netgear products - I can fix it by myself 🙂 - no problem... for this price it's not that bad.
OK perhaps I did not make myself clear. I have enabled ssh access , "reformatted" my data partition, c, to ext3 and now am adding additional drives and expanding. I assume that the process does not alter the partition i.e convert back to ext4 for the data partition. Which of the log files will tell me the partition information.
I believe that the Readynas NV+ V1 data partion is ext3 and I have no issue with expansion and filling these to the max!
But who...? I have IBM Enterprise San, Synology-Rack-NAS and everything seems like joke to me 😄 Thanks to full SSH access to Netgear products - I can fix it by myself 🙂 - no problem... for this price it's not that bad.
i'll do what i should have done in the first place, roll my own MAS system using normal PC hardware. that way i not only have full control over it, but i can do so much more with it!
I moved to ext3 couple of months back and this seemed to resolve my issues.
However I have now had a drive failure. On replacing the failed drive the unit will get stuck on 'RAID sync started on volume C.' What then happens is the light for 'no.2' bay starts flashing, even though it is No.1 bay that is out.
The GUI reports the volume as 'redundant' but only 3 of the 4 green lights are on the icon (?). After a restart of the unit *it does not mount my volume and refuses to* - so I have no access to any of my data.
To resolve this it is necessary to remove my new disk and restart the unit again. It then comes back saying 'unprotected' but I can at least access my data.
This unit is 100% stock apart from switching to ext3. I suggest somebody looks into whether using this causes issues for DR scenario. Seems to suggest from my experience this may interfere with this.
Netgear support told me they tested this problem on a unit in their offices and couldn't re produce the problem. Clearly this was lies!
Oh also my time machine stopped working when I went to ext3.
...Netgear support told me they tested this on a unit in their offices and couldn't reproduce the problem. Clearly this was lies!
I understand that you are frustrated, but there is nothing in your post that supports a claim that they are lying to you.
cync wrote:
...To resolve this it is necessary to remove my new disk and restart the unit again. It then comes back saying 'unprotected' but I can at least access my data.
My suggestion is to (a) backup up your data while you are in this unprotected state. Then (b) delete the partitions on the new disk (on a PC), and then reinsert it while the NAS is running.
BTW, what model disk is the new one? Also, did you check with the vendor to see if there is a needed firmware update?
