RN316 randomly freezing shared folder on 6.4.0

[nybblesandbytes]

Hi,

I upgraded a client's RN316 to firmware 6.4.0 about 2 days ago. The firmware update reported successful and all seemed ok for about half a day.
Now the primary share (2.7TB) of data, has become extremely unstable, randomly freezing up for about 10 minutes when PC's or Macs try to access it.
Protocols enabled are SMB, AFP and NFS, and permissions are open - everyone can read/write.
The unit has 2 x 4TB hard drives running (mirrored).

Approx. 25-30 users may be accessing the share at any one time, and this was fine on 6.2.4.
I had hourly snapshots running, but have reduced them to daily to try troubleshoot - the issue persists.

I have tried using the 2nd NIC instead (Gb LAN), changed the device IP address, really just about exhausted all I can think of.
Is there any way to rollback to 6.2.4?
Will resetting permissions maybe help, or switching off and re-enabling protocols for the relevant share...?

Thanks guys, I'm at a loss here. A great product being let down by these bugs right now 😞

[btaroli · Accepted solution]

Do you have ssh enabled? If so, can you go in and run top to see if "[btrfs-cleaner]" shows up and is using lots of CPU? If so, then you may well be experiencing an issue that's been reported in other threads.

[vandermerwe]

No way to go back, unfortunately.

There is an issue with the built in user group and network permissions but it sounds like your issue is unrelated. 

I also have a 316 and I see an issue where windows explorer gets "stuck" when trying to open a folder in a share, if I cancel the request to open the folder and then double-click the folder to open it it opens the second time.  This is extremely annoying. Possible similar to your issue.  Take note Netgear.

Ir is of course possible that you have a disk problem unrelated to the firmware.  Are the smart stats normal?  Can you remove and test both drives with vendor tools?

[nybblesandbytes]

Thanks for the quick reply vandermerwe!

Gaan ons op Saterdag wen? 😉

Not sure if it's related, but the Web Interface admin page also goes loopy at the same time as the shares lag - it just hangs for about 5 minutes and eventually comes up... slowly.

SMART status within the interface shows both drives as good. I haven't been able to remove the drives and test though...

It all went south after 6.4.0.

I was trying to figure out if something else on the LAN may be the cause - I have been scanning the logs on the switches - found one port with a ton of failed packets, so I am going to see what that is about tomorrow... could be flooding the LAN somehow maybe? Just grasping at straws here...

I will revert in the morning.

Thanks again for your help and time!

[btaroli · Accepted solution]

Do you have ssh enabled? If so, can you go in and run top to see if "[btrfs-cleaner]" shows up and is using lots of CPU? If so, then you may well be experiencing an issue that's been reported in other threads.

[nybblesandbytes]

So it seems there isn't anything even plugged into the suspected LAN port on the switch, so that kinda rules that out.

Sporadic issues again today - share dropping randomly and freezing up PC's.

All I can see happening in the logs is old snapshots deleting on the hour.

Should I manually cleanup the snapshots?

Can I reload the same firmware (6.4.0) manually?

Just thinking of more options here...

[nybblesandbytes]

Howdy,

thx for the reply.

currently it is switched off in settings, but I can enable it.

Thereafter however, I am at a loss - I wouldn't know how to get in via SSH or where to look for btrfs-cleaner... could you possibly advise me?

Many thanks in advance.

[btaroli]

http://kb.netgear.com/app/answers/detail/a_id/23096

Once enabled, you ssh into the NAS as "root" user and use the admin password.

Once you have a shell, use the "top" command, and you'll see something like this:

top - 02:07:03 up 1 day,  5:49,  2 users,  load average: 1.29, 1.42, 1.68
Tasks: 215 total,   2 running, 213 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.1 us, 25.3 sy,  0.0 ni, 74.5 id,  0.2 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  16324816 total, 15779192 used,   545624 free,     3360 buffers
KiB Swap:  2093052 total,        0 used,  2093052 free, 14271244 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND                                                   
 2449 root      20   0     0    0    0 R 100.1  0.0 156:42.25 [btrfs-cleaner]                                           
10075 root      20   0 3121m 275m  21m S   1.0  1.7  41:41.11 /apps/dvblink-tv-server/dvblink_server                    
15643 root      35  15 1872m 181m  10m S   0.3  1.1   0:49.99 Plex Plug-in [com.plexapp.system] /apps/plexmediaserver/B 
17001 root      20   0 28616 3080 2548 R   0.3  0.0   0:30.30 top                                                       
20018 btaroli   20   0  263m  13m  10m S   0.3  0.1   0:00.14 /usr/sbin/smbd                                            
2

Notice the "COMMAND" of the first line? That's what you're looking for. 🙂

To exit top, type "q". To disconnect from ssh session, simply use the command "exit".

If this all sounds scary, maybe wait for Netgear to make suggestions on how to proceed.

[nybblesandbytes]

Thank you, I think I can handle it 🙂

Even so, I am monitoring now for a while, have had a few hours of stability, so wanting to see if it lasts as things are.

[KenD90027]

I am experiencing the same problem with my 314

[nybblesandbytes]

So after another day of troubleshooting, I have what I hope is some good news.

Netgear Support came back to me today after I sent them the full device logs this morning.

They noted something very obvious - my data volume is 90% full, and therefore very much in the red. Apparently they recommend that users stay under 80% for optimal operation.

I then went in and deleted some old snapshots and anything redundant, freeing up almost 100GB in the process.

I have to say that since then, I am cautiously optimistic as all users seemed to be stable at close of business.

I am busy monitoring the device remotely and will continue to do so tomorrow morning, and then report back... and then put an additional drive in of course!

@KenD90027- how full is your RN316...?

[BennyKind]

Hi nybblesandbytes

I am curious to know whether you were able to get on to the web interface for your 316?

I am having a similar issue but I cant get on to the web interface to be able to delete snapshots/disable the snapshot schedule. Mine is also incredibly full so I am hoping this will be a fix if I can get on to it to delete snapshots.

[nybblesandbytes]

@BennyKind Web interface is fine - occasionally it freezes up for a few minutes, but it comes right after a while.

After I deleted some snapshots things seemed to improve a bit, but we went ahead and got an extra 4TB drive today.

I have just installed it and it is resyncing - estimated time to resync is 25 hours, so will likely finish sometime tomorrow.

I will be sure to report back after the storage has been expanded.

[KenD90027]

Problem remains on my RN314, unable to access admin page to delete snapshots. Was at 90% of 3.5TB @nybblesandbytes .

So far I have performed a successful usb recovery for the 6.4.0 and then successfully ran an OS recovery through the boot menu. After rebooting, the unit has an active activity light for several minutes while the LCD shows "Booting..." . Shares are available on the network during that time. Then at some point the shares lose accessibilty and the activity light goes dark. Suggestions anyone? @mdgm-ntgr?

[BennyKind]

@KenD90027 have you made any progress on this?

I managed to get on to the admin page for about 10 minutes, deleted maybe half the snapshots before it kicked me off again and now again I am unable to connect to admin page or access the folders.

Becoming a real big problem for us now.

[nybblesandbytes]

Update as at 18:00 today - the extra 4TB hard drive finished resyncing and I now have over 4TB free space available.

The client has closed for the weekend so I don't have any feedback since then, however I will be taking this up again on Monday morning, to establish if the device is now 100% stable again, and performing normally.

Netgear's official position on this was that the ReadyNAS should have 20% - 25% free space available...

[nybblesandbytes]

Update on Monday morning 26 October 2015:

- NAS is running normally at the moment, after upgrading storage with an additional 4TB last Thursday.

The resync took over 24 hours to complete, but it did finish and the shares were up the whole time.

- Netgear had advised me to run a Balance as well. I first ran a full backup on Saturday, and then I started a balance on Sunday morning.

Bad idea - the balance started out fine, showing up progress as 1% initially, then the device locked up completely (no web interface, no shares available, no ping response).

I decided to wait it out until this morning - still dead, so I had someone onsite pull the power plug on the NAS and switch it back on.

After about 5 minutes the NAS came back up, and the web interface showed the balance had restarted at 0% - there is no way to cancel the balance from within the web interface.

I then used PUTTY to SSH into the device and used the following command to stop the balance:

btrfs fi balance cancel /data

It took a few minutes to cancel, but updating the web interface showed that the balance had been cancelled.

Now the NAS is running normally again, and shares are available. I am just concerned about the fact that I had to cancel the Balance - what effect can this have had on the data?

I can't have the device locking up every time I need to run a balance / defrag / scrub...

[nybblesandbytes]

...and we're back to where we started.

The laggy RN16 issue persists – today it has dropped the share on the LAN several times, but seemingly not on all workstations.

I am trying to pinpoint if the issue is now AFP / SMB based, or maybe based on something else?

Do I switch AFP / SMB off and on for the share in question, and is that safe to do?

Macs seem to be affected more, so I have been refreshing the DHCP leases for the IP addresses of the Macs just to rule that out - I imagine that an IP conflict somewhere could cause issues.

Could the snapshots (hourly) have an impact on the performance of the device?

I noticed in the logs that the device is also deleting old snapshots on the hour… This was never an issue in 6.2.4 - could it be possible that 6.4.0 has somehow made the snapshot feature buggy and slow - slow enough to cripple the device for approx. 10 minutes at a time?

[nybblesandbytes]

Update at the end of the day - 28 October 2015...

The ReadyNAS 316 was up to tricks again all day today.

Since about 08:30, up until about 18:00 this evening, it would drop the share whenever it felt like it.

The web interface also went down a few times.

Something that I did notice, and I am hoping that I am onto something here - I took note of the exact times that users went down, and compared that with the device logs.

Old snapshots were auto-deleted at 17 minutes past each hour - so at 09:17am, then at 10:17am etc. - there must be a schedule to control this.

Users went down at approx. half past each hour - 09:30am, then at 10:30am etc.

Snapshots were then created on the hour, each hour.

I am hoping that there is a correlation here - I have since disabled hourly snapshots, and so the hourly auto deletion of snapshots has stopped as well.

I will know during the course of tomorrow if there is a correlation.

It sucks because hourly snapshots are a powerful thing and have saved my butt many times, but I trust that Netgear will fix this performance hit in the next firmware update, if in fact this is the problem...

I would rather have a stable device and fewer snapshots, than a crazy device with a million snapshots...

Holding every thumb that I possess...! 🙂

[nybblesandbytes]

@btaroli So the troublesome btrfs-cleaner was in fact hard at work taking up 97% CPU a lot of the time.

A user on another thread suggested a way to stop the process by disabling quotas from SSH, and that did the trick.

Now the device is running normally again - only thing is that the volume now shows "Free" and "Snapshots", no data... yet the share is there and working normally.

I worry about what may happen if btrfs-cleaner starts again...

This all became an issue from 6.4.0 onwards...

[nybblesandbytes]

@BennyKind Apparently what happens is that after snapshots are deleted, the btrfs-cleaner process is invoked, which is when all hell breaks loose and it consumes 90+% of CPU resources...

For some reason this is delayed - my snapshots were auto deleting at 17 mins past the hour, and the ReadyNAS would freeze at half past, even though the device logs indicated 17 mins past the hour.

I managed to stop the btrfs-cleaner process in SSH and the device seems to working normally again now - LAN shares, web interface.

The only thing I see is that under Volumes, it now only shows "Free Space" and "Snapshots" - no data... Worrying!

Even so, my shares are there and fully accessible and I have a working fileserver again.

Really hoping that Netgear takes note and pushes out a stable fix for this very soon.

[BennyKind]

@nybblesandbytes this is all such a pain in the ****

I managed to get on to the web interface for a good ten minutes and managed to delete all remianing snapshots and change the snapshot schedule to never.

This seems to have fixed it for now despite the unit not taking snapshots which makes me a little nervous. 4 days on and its still running ok and is accessible by everyone.

Netgear really do need to get this sorted.

[nybblesandbytes]

Apparently 6.4.1 is now out and has fixed this issue...

[BrianL2]

Hi nybblesandbytes,

Thanks for the feedback. You may now tag this thread as resolved by clicking "Accept as Solution" in one of the responses that you received.

Let us know if you have further questions.

Kind regards,

BrianL
NETGEAR Community Team