ReadyNAS 3220 Locking up after upgrading to 6.2

[kplatte]

defrag completed after 12 hours. And it still locked up over night again.
Going to lower NFS count. I see it was set to 12 what do you suggest it be set to?

[kplatte]

I tried to reduce NFS settings and get an error.
"Cannot Start Service without Volume" Code: 15002040001

[mdgm-ntgr]

Can you send a fresh set of logs and the Config Backup?

[kplatte]

It locked up again. After I rebooted the NAS I was able to change NFS settings. I lowered it to 4.

[kplatte]

Still locking up every night with 4 NFS threads set in settings.

[mdgm-ntgr]

You still have a huge amount of metadata on your data volume. This indicates you have a lot of fragmentation. The high NFS thread count was likely related to this.

Sounds like the best solution would be to backup your data, do a factory reset (wipes all data, settings, everything), configure your shares appropriately and a low NFS thread count and restore your data from backup.

[kplatte]

Just an update of what been trying to resolve.
We are still working on getting another server setup that we can use to backup the data from the NAS before we can try a factory reset.
I will be installing the 6.2.3 after are backup finishes running this morning.
Will send over new log file today.

I ran a Disk test came back with no errors
Ran a defrag
Ran a Balance
Was finally able to have a scrub complete.
We stopped all but one backup that runs over night and it still locks up.

[StephenB]

defrag can't help you if the fragmentation is a result of snapshots. So you might try deleting all the snapshots you don't absolutely need, and then re-run the defrag.

[kplatte]

We have no snaps shot. Disabled and deleted them before we upgraded to 6.2.*

[mdgm-ntgr]

Your smart_history.log does show that some disks are having errors. Perhaps a failing disk is causing the problem on your system?

[kplatte]

Looks like those drives with errors are the 2 drives we already replaced in past month.

[mdgm-ntgr]

Ah, Have you tried 6.2.3 beta?: http://www.readynas.com/forum/viewtopic.php?f=154&t=78866

[mgiammarco]

I have the same locking problem with nfs:

viewtopic.php?f=7&t=78073

I think we should start a new thread for NFS related problems.
Because I have already changed hard disks, resetted to defaults like you in the new thread we should explain what steps we have already done.
As a note for example my uptime has increased with more nfs threads.

[mdgm-ntgr]

Increasing the thread count will lead to a lot of fragmentation. Not a good idea in my view.

Have you tried changing to async?

[kplatte]

Were at 8 NFS threads now. when we lowered NFS threads below 8 are backups would not run since it couldn't mount the NFS shares.

[mdgm-ntgr]

Can you try 6.2.3 beta 3: http://www.readynas.com/forum/viewtopic.php?f=154&t=78866

If you are willing you may also wish to consider 6.3.3 beta 3: http://www.readynas.com/forum/viewtopic.php?f=154&t=79670

We have a fix for premature snipping of logs so if your NAS locks up again a fresh set of logs may be more useful. If you send a fresh set of logs it would be helpful if you could state the rough time of the crash.

[kplatte]

I will try the beta 3 first.
But strange thing is our NAS has not locked up the past 2 nights since i started running a backup of the folder that we backup up our server to every night.
This is a backup job I created on the NAS.
This folder is 1.7T and is only about half done after 2 days so i have to wait till it completes before I try new beta. Not sure why backup going so slow besides thats it probably due to the thousands of files it contains.

[kplatte]

It froze up again after we enabled 2nd backup job to run again over night.
Will try new beta tonight.

[kplatte]

Updated to 6.3.3 beta and now keep spamming that PSU 1 and 2 are offline.
Sending you new log file.

[kplatte]

Loading 6.2.3 beta 3 now the PSU offline spam is out of control.

[mdgm-ntgr]

Looking at your smart_history.log I can see you have a disk that has a growing current pending sector count which indicates it may be failing.

I can see that you still have a lot of metadata which suggests you still have a lot of fragmentation.

I have asked our engineers if they can reproduce the erroneous PSU failure messages. I don't have a 3220 so can't attempt to reproduce that here.
'

[kplatte]

Locked up last night running 6.2.3 beta 3
I see that drive with errors i will get that one replaced.
Will send log file for last night.

top - 02:41:24 up 15:17,  1 user,  load average: 7.82, 3.45, 2.55
Tasks: 176 total,  15 running, 161 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us, 75.3 sy,  0.0 ni,  0.0 id, 24.5 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem:   3938488 total,  3141212 used,   797276 free,      600 buffers
KiB Swap:  5232636 total,        0 used,  5232636 free,  2220428 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
  614 root      20   0     0    0    0 R 100.1  0.0   3:47.11 kswapd0
 3714 root      20   0     0    0    0 R 100.1  0.0   4:27.49 nfsd
 3719 root      20   0     0    0    0 R 100.1  0.0   4:20.73 nfsd

[kplatte]

Replaced hard drive this afternoon it is now rebuilding

[kplatte]

After updating to 6.3.3 it didn't lockup the first night during backups. But on Saturday and Sunday night it locked up.
want me to send logs?

[mdgm-ntgr]

Sure. Logs would be good.