Problems with 6.1.9, need to restore 6.1.8 #24040549

[smuskiew]

I have a ReadyNAS 314. Foolishly, I decided to update the firmware from 6.1.8 to 6.1.9 when prompted by the Frontview GUI. Since then I have had nothing but problems. The NFS share I have mounted on my VMWare esxi server becomes unresponsive shortly after the ReadyNAS comes up and the afp/Timemachine share is the same. The Frontview GUI also reports that the system is "offline". About the only thing I can do is ssh into the ReadyNAS where I can see a system load of around 17 and "dmesg" output shows that btrfs and nfs threads are hanging.

Netgear "support" (I use that term very loosely) has been wasting my time not actually reading my queries and trying to get me to enable remote SSH access (which I cannot do thru our corp. FW). I just need to get the box rolled back to 6.1.8 so I can get it back online ASAP. The 12+ hour response time of Netgear support in between my queries is really not cutting it.

Is there some way I can initiate a rollback to 6.1.8 myself. I have searched and not found a whole lot of info about this.

Thanks in advance for any info.

-steve

[StephenB]

You should be able to download the firmware yourself - just google the 6.1.8 firmware link (site:readynas.com or site:netgear.com helps), and download it. Then manually install it from the web admin ui.

Sometimes you can't downgrade easily (6.2.0 beta being a current example), but that should not be an issue with 6.1.9 -> 6.1.8.

Always wise to have a backup though...

[mdgm-ntgr]

The firmware update and problem you have encountered is likely coincidental. That is why we requested remote access to diagnose the issue.

Seeing you have SSH access can you check:

# df -h
# btrfs fi show
# btrfs fi df /data
# cat /var/log/frontview/initrd.log

[smuskiew]

StephenB wrote:
You should be able to download the firmware yourself - just google the 6.1.8 firmware link (site:readynas.com or site:netgear.com helps), and download it. Then manually install it from the web admin ui. Sometimes you can't downgrade easily (6.2.0 beta being a current example), but that should not be an issue with 6.1.9 -> 6.1.8. Always wise to have a backup though...

Thanks, I apparently didn't do a very good job of searching the forum the first time. Finally got the right download link, and was able to successfully downgrade to 6.1.8.

(I got the box back up and functioning with a power reset). Seems like the issue is not firmware dependent but is specific to the NFS daemon. The box is fine until we start trying to access files from the NFS share, then we start getting "task nfsd ... blocked for more than 120 seconds" in dmesg output and the system load starts going up:

[Thu Oct 23 15:47:01 2014] INFO: task nfsd:2654 blocked for more than 120 seconds.
[Thu Oct 23 15:47:01 2014] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Thu Oct 23 15:47:01 2014] nfsd D 0000000000000000 0 2654 2 0x00000000
[Thu Oct 23 15:47:01 2014] ffff88005ba55870 0000000000000046 0000000000000000 ffff880061d31a50
[Thu Oct 23 15:47:01 2014] 00000031e8f6e7ce ffff88005ba55fd8 ffff88005ba55fd8 0000000000004000
[Thu Oct 23 15:47:01 2014] ffff88007e46d910 ffff880061d317c0 0000000100000000 0000000000000000
[Thu Oct 23 15:47:01 2014] Call Trace:
[Thu Oct 23 15:47:01 2014] [<ffffffff880ceca4>] ? add_partial+0x24/0x80
[Thu Oct 23 15:47:01 2014] [<ffffffff880d04ce>] ? deactivate_slab+0x4e/0xf0
[Thu Oct 23 15:47:01 2014] [<ffffffff8887adda>] schedule+0x3a/0x50
[Thu Oct 23 15:47:01 2014] [<ffffffff882dc97d>] wait_current_trans.isra.34+0x8d/0xf0
[Thu Oct 23 15:47:01 2014] [<ffffffff88063e80>] ? abort_exclusive_wait+0xb0/0xb0
[Thu Oct 23 15:47:01 2014] [<ffffffff882df0ca>] start_transaction+0x14a/0x3f0
[Thu Oct 23 15:47:01 2014] [<ffffffff882df6d3>] btrfs_start_transaction+0x13/0x20
[Thu Oct 23 15:47:01 2014] [<ffffffff882ee2cd>] btrfs_sync_file+0x10d/0x270
[Thu Oct 23 15:47:01 2014] [<ffffffff880ff058>] generic_write_sync+0x48/0x60
[Thu Oct 23 15:47:01 2014] [<ffffffff882f04e8>] btrfs_file_aio_write+0x378/0x540
[Thu Oct 23 15:47:01 2014] [<ffffffff880eed65>] ? iget5_locked+0x85/0x1d0
[Thu Oct 23 15:47:01 2014] [<ffffffff882df900>] ? btrfs_clean_one_deleted_snapshot+0x140/0x140
[Thu Oct 23 15:47:01 2014] [<ffffffff882e96a8>] ? btrfs_iget+0x48/0x560
[Thu Oct 23 15:47:01 2014] [<ffffffff880ec69a>] ? d_obtain_alias+0x6a/0x1c0
[Thu Oct 23 15:47:01 2014] [<ffffffff882f0170>] ? __btrfs_buffered_write+0x4b0/0x4b0
[Thu Oct 23 15:47:01 2014] [<ffffffff880d6614>] do_sync_readv_writev+0xd4/0x110
[Thu Oct 23 15:47:01 2014] [<ffffffff8806b67d>] ? groups_alloc+0x5d/0x130
[Thu Oct 23 15:47:01 2014] [<ffffffff88352ba8>] ? security_file_permission+0x28/0xb0
[Thu Oct 23 15:47:01 2014] [<ffffffff880d5d7e>] ? rw_verify_area+0x5e/0xf0
[Thu Oct 23 15:47:01 2014] [<ffffffff880d68f3>] do_readv_writev+0xd3/0x1e0
[Thu Oct 23 15:47:01 2014] [<ffffffff880d6a30>] vfs_writev+0x30/0x60
[Thu Oct 23 15:47:01 2014] [<ffffffff881d4d8e>] nfsd_vfs_write.isra.13+0xee/0x3e0
[Thu Oct 23 15:47:01 2014] [<ffffffff880d455c>] ? dentry_open+0x4c/0x90
[Thu Oct 23 15:47:01 2014] [<ffffffff881d5bbd>] ? nfsd_open+0x9d/0x180
[Thu Oct 23 15:47:01 2014] [<ffffffff881d72d9>] nfsd_write+0xf9/0x110
[Thu Oct 23 15:47:01 2014] [<ffffffff881df708>] nfsd3_proc_write+0xb8/0x150
[Thu Oct 23 15:47:01 2014] [<ffffffff881d0d91>] nfsd_dispatch+0x101/0x240
[Thu Oct 23 15:47:01 2014] [<ffffffff8883a813>] svc_process+0x4c3/0x850
[Thu Oct 23 15:47:01 2014] [<ffffffff881d045e>] nfsd+0xbe/0x150
[Thu Oct 23 15:47:01 2014] [<ffffffff881d03a0>] ? exportfs_decode_fh+0x2d0/0x2d0
[Thu Oct 23 15:47:01 2014] [<ffffffff880635f9>] kthread+0x89/0x90
[Thu Oct 23 15:47:01 2014] [<ffffffff8887e0f4>] kernel_thread_helper+0x4/0x10
[Thu Oct 23 15:47:01 2014] [<ffffffff88063570>] ? kthread_worker_fn+0x140/0x140
[Thu Oct 23 15:47:01 2014] [<ffffffff8887e0f0>] ? gs_change+0xb/0xb

Now that I am able to pull logs via the GUI, I have submitted them to Netgear support on my original case, hoping it will actually get addressed at some point.

Thanks for the help!

[mdgm-ntgr]

It sounds to me like the NFS share is quite fragmented. As I said the firmware update was a coincidence. You were going to run into this issue if you had just stayed on 6.1.8 too.

[StephenB]

mdgm wrote:
It sounds to me like the NFS share is quite fragmented. As I said the firmware update was a coincidence. You were going to run into this issue if you had just stayed on 6.1.8 too.

That suggests a defrag might be the next step?

[mdgm-ntgr]

A defrag would be useful, but the real issue here is almost certainly the fact that COW (Copy on Write) and VMs don't go well together. For files that are changed e.g. millions of times COW should be disabled. You can't disable COW on an existing file, but you can disable it on a directory so new files created in it don't have COW enabled.