I see that other users have posted about what looks like a similar issue but I can't seem to find a fix. I must just not be able to find it as I find it extremely hard to believe that Netgear have known about a very serious fault like this for over a year and done nothing about it!
I have 6 ReadyNAS NV+ V2 units all of which I write to on a daily basis with similar amounts of data. 4 of the units are stocked with WD RED WD30EFRX drives and 2 with Barracuda 7200.14 ST3000DM001 drives. All drives have been tested using manufacturers testing tools and come back fine. The problem is that I can now no longer write to any of the units without them crashing and becoming unresponsive. The problems all start when I get down to 600GB of free space. Up until this point if I can write to the unit fine but for some reason this last 12% is impossible to write to without the unit crashing.
Now, if I had one unit or even one unit stocked full of the same disks I could see some kind of one of issue - but 6 and with different disks?! This is a very serious fault that makes this faulty product unfit for market. I'm hoping that someone has an idea of what is going on here or has an idea of how to fix it. I have tried: Factory default Upgrade/downgrading firmware Changing FS version Testing disks
All to no avail, and I need to write data to these units. I have raised a support ticket with Netgear and am eagerly awaiting their response. I have 4 of the ReadyNAS NV+ V1 units and no problems at all, seems like quality control has taken a nose dive.
Any ideas at all?! Don't know what to do with these units!
You can reformat it to ext3 This is very hard situation for Netgear because everyone have ext4 filesystem with data on - so it cannot be easily solved.
What does reformatting to ext3 involve, will I lose all my data? Obviously I backup but 12TBs of data will be a massive inconvenience to restore. What are the disadvantages to moving to this older filesystem, will XRAID still work? Any official comment from Netgear on this or are they just pretending it isnt happening?
What does reformatting to ext3 involve, will I lose all my data? Obviously I backup but 12TBs of data will be a massive inconvenience to restore. What are the disadvantages to moving to this older filesystem, will XRAID still work? Any official comment from Netgear on this or are they just pretending it isnt happening?
You cannot convert ext4 to ext3 (if new features are enabled - and this is default when volume is formatted) ext3 can be mounted by ext4 driver, but when you enabled new features backward compatibility is lost.
As I know everything is same with ext3 except filescan. So Yes propably all RAID functions may work perfectly.
You don't see NETGEAR bug, because this bug appearing on Synology and QNAP devices too.
When you write only little file on ext4 then some IO operations completely eat all resources. - for me this is some problem with journaling operations, but I don't know it exactly. Problem is hidden in kernel driver and nothing is written to system log.
And what is completely crazy is that you can destroy journal of ext3 filesystem by mounting it with ext4 driver.
You don't see NETGEAR bug, because this bug appearing on Synology and QNAP devices too.
When you write only little file on ext4 then some IO operations completely eat all resources. - for me this is some problem with journaling operations, but I don't know it exactly. Problem is hidden in kernel driver and nothing is written to system log.
That was EXT3 file system performance problem. NV+v2 has the same issue with EXT4. The problem is fragmented available space and small embedded memory. To write lage file Linux try to find the best suitable placement of free extents in available fragmented space. Fragmentation map of available space is huge for analysis and NAS has to spent a lot of time for this. Full map investigation doesn't make a sense. This analysis is quite expensive. "Depth" of fragmented map investigation should be restricted. Linux has two tunable parameters for this.
mb_max_to_scan -- The maximum number of extents the multiblock allocator will search to find the best extent
mb_min_to_scan -- The minimum number of extents the multiblock allocator will search to find the best extent
robclarke41, you need find and reduce parameter values inside your NAS. Ask google "mb_max_to_scan" to find the path in file system. Remove your volume from NAS and insert new HDD. Would be better to try these changes on empty HDD first. Hope this will help.
Any ideas at all?! Don't know what to do with these units!
You can reformat it to ext3 This is very hard situation for Netgear because everyone have ext4 filesystem with data on - so it cannot be easily solved.
Snipe3000 can you confirm that Netgear downgraded you to EXT3 and that your unit still crashes when you have around 12% space remaining?! I am about to wipe my units and go through this with support. Will be extremely annoyed if I go through this whole palava only to realise it has done nothing.
robclark41, were you able to change the "mb_max_to_scan" parameter? I have no data to suggest this would or would not be helpful but sotrack's explanation seems plausible. I checked an x86 RN and found '/sys/fs/ext4/dm-0/mb_max_to_scan' was a 4k file populated with a value of '200'. I would hazard editing this file and changing the value in your copy to something smaller might be what is being suggested as a possible workaround. If you were to attempt this you would need to enable root ssh (http://www.readynas.com/?p=4203) and connect via PuTTY or WinSCP.
Thanks for the suggestion ahpsi but I'm determined to get an answered from Netgear as to exactly what the problem is with these units. I don't want to touch the device until the Netgear L3 guys has had a look.
Hi Guys, unfortunately I'm not having a lot of luck with Netgear support. I've realised I need the EXT4 filesystem for 2TB files so downgrading to EXT3 is not an option for me. I told them that I bought the unit as an EXT4 compatible device at the time. They said the only option for me was to take it back to the reseller. I approached the reseller but they will not refund me unless there is an official statement from Netgear saying that there is a fault with these units. And guess what? Netgear are refusing to admit that there is a fault with the NV+ V2! So this leaves me between a rock and a hard place. I've read that QNAP NAS units now have a firmware based solution for this problem so why an earth is there not one from Netgear?! Absoultely disgusting support from Netgear. They really have done nothing to address what is now a common and very widespread problem.
Snipe3000 can you confirm that Netgear downgraded you to EXT3 and that your unit still crashes when you have around 12% space remaining?! I am about to wipe my units and go through this with support. Will be extremely annoyed if I go through this whole palava only to realise it has done nothing.
Sorry Snipe I've never dabbled with the command line interface.
I have just received an official comment from Netgear support:
'Again, this is a problem with this NAS, not a generic one, but we know of it and it happens in very specific situations.'
They have therefore just admitted that they have not tested the unit at all as if they had they would have seen the problem. For them it's easier to dismiss all of us users as individual cases and not link them as then it would become a problem for them. They are a huge company though what can we do 😞 I know my consumer rights have been violated here.
Hi All, I have been in contact with L3 at Netgear who I have finally managed to convince that they have a serious problem with these units. They have given me a firmware patch to test but unfortunately it hasnt made any difference. At least they are working on it now...
Quick update...I am having a nightmare with Netgear support. It is unbelievably bad. The last patch they gave me to fix the problem did not work. They have now said 'accept our remote access policy or we will close the case'. I find their level 2 support guys rude, arrogant and unhelpful. Unless I give them unrestricted access to my unit and all my personal data they will not help me. It is a terrible way to do business, if only there was somewhere official we could complain about this and force a solution out of them. I have conceeded to letting the L3 guys remote onto my PC but said I wanted to be there when they did it so I could see what was going on. This apparently is not aloud, they prefer to snoop, steal and mess with all your data in the privacy of their own offices in their own time. What annoys me the most is that they will not try and reproduce the issue in a test lab. It is very simple and easy to reproduce but they will not do it. This make me furious, with one fell swoop it would tell them that this is a general issue and not restricted to a few cases. I'm at my wits end... what to do with these idiots?!
😞😞 have had to agree to Netgears remote access policy. For some reason the L2 guys revere the L3 guys like some kind of gods. Check this quote out 'I don’t have visibility of their queue but I have upped the priority and contacted the managers directly so I would hope soon but until assigned to an engineer I cannot talk to them directly to check.' They can't even speak to them? WTF kind of IT department is that? Their remote access policy is ridiculous so I have only agreed to it on my terms.
1. No SSH access through the Internet 2. They connect to my PC on a new standard user profile that ill delete after 3. I will be on the phone with them monitoring everything they do 4. No access to my data. I've created a dummy share, when I copy data into it they will see the unit crash. (I got around this by using a program called drive inventory which creates an excel file of all your files on the drive. I then removed the file names).
Hopefully this minimise risk to my systems and privacy. Although still utterly ridiculous and unnecessary. This should have been tested properly by Netgear.
Haha yes chirpa it's getting that way! Guess I should have known better but despite constant assurances and sacrifices made by me to be there I still didn't get a call from L3. What a disgrace.
Thanks for the info cync, ext3 was never an option for me anyway. I tried the firmware patch and it did nothing. Crashed again the second I tried to copy data to it. Absolutely rubbish.
Heres the response from the so-called level 3 engineer, what a joke. He's now trying to say that the unit crashing at 90% full is by design and that its bad practice to use more than 80% of your NAS units space?!!! Where did he become a L3 engineer? Disneyland?! I love the way he patronises me by giving me an IT lesson in spindle drives and then says 'Your vision of the ideal solution may not be practical' - so to expect to be able to copy data to a data storage unit is not practical. What planet are these jokers from. Sooooooo unprofessional its ridicuous, in my company this guy would have been fired.
Robin,
I have taken ownership of your case. I will be the point of contact between you and engineering. I, just now, finished reading the notes logged by you and the several people who have tried to help.
There are a few things, I feel, must be clear before proceeding. · You need to know that we want your issue resolved as much as you do. · We want to know if your issue is systemic, or one that afflicted 2 users-out-of-thousands. · If it is just 2, we still wish to find a solution for you. · Your vision of the ideal solution may not be practical.
From our perspective the issue is related to EXT4 and not uniquely the NV+V2, I will explain. As files are copied and deleted and new files are added, fragmentation occurs. As the system becomes fragmented, the metadata handling and the blocks free to write data may become insufficient to continue writing data. Particularly if the later data is in the form of large files. All spindle drive systems are recommended to be used at less than 90% capacity, unless they hold archival data and fragmentation is virtually nonexistent; in which case you may risk going to 95%. Generally, 20% empty space is recommended to remain trouble free on drives with platters. This is common understanding in the technology world.
Having said that; Keep in mind that we sincerely want to find a proper solution.
Engineering is working on finding a possible solution from their side and I have already initiated contact with them. As things develop I will communicate them with you.
If you have concerns or questions feel free to email me directly.
