NAS Pro Pioneer Edition on OS 6.9.3 freezes/ network dropout

[firefly_242]

Hi,

Update from RAIDiator-x86-4.2.31 to OS6.9.3 went smooth and am happy with the new OS.

However, my system sometimes drops out or freezes. The interval between them ranges from 1 day to 16 days.

When it drops out I am able to initiate a shutdown of the system via the power button. In case of a freeze situation I am forced to keep pressing the power button to shut the system down.

This morning I found my system again in freeze state. System was up for 16 days.

Log files are not showing anything alarms when this drop out/freeze happens.

Have collected the log files and can share them if needed.

Would appreciate to know why this happens and how to solve this.

Thanks

[mdgm-ntgr]

You can send in your logs if you like (see the Sending Logs link in my sig). Ideally logs should be downloaded ASAP after running into the problem.

[firefly_242]

logs shared. thanks for looking into this.

[mdgm-ntgr]

smart_history.log indicates a current pending sector on a few of your disks.

Can you test your disks e.g. using WD Data LifeGuard Diagnostics for WD disks or SeaTools for SeaGate disks?

[firefly_242]

thanks for the fast reply. will have to buy an enclosure first to be able to connect the drive to my laptop.

Will update once drives are tested.

[firefly_242]

Hi,

Took awhile to run WD Data LifeGuard Diagnostics extended test on all 6 HD's, but finally done. All 6 HD's passed the extended test.

So, I assume that means the issue is not with the HD's.

Any idea what to check next?

Thanks

[Alein]

I have this same problem with all NASes with OS6.

2100,2120,4200,ReadyData5200 and with tower models.It must be a bug in the firmware.

It is hard to find the source, but it is something with a network. You will be able to log in through the serial port and for example reboot etc.

I see nothing interesting in logs. If you have more NICs, you even could be able to reach NAS through different NIC. 

I was playing with MTU, but bigger 9000 cause performance drop. The problem can be triggered if you overload a bit NAS with tasks. I use most of them as iSCSi for VM (non critical secondary like backup), and vmotion can kill NAS after few minutes. More snapshots can accelerate freeze.

A few days ago  I was thinking about a script to shut down one nic later up this one and do this same with seconds nic in LAG. Unfortunately, I have to drive to NAS 200KM as it freezes 😞 

OS6 is just a joke, perhaps the problem is BTRFS implementation and some kind of memory leak in NIC drivers.

[Alein]

 I can add. that if you have LACP, and disable ports from the switch site when NAS freeze and later turn them on, they will UP, but LACP will not, there will be no communication.

[mdgm-ntgr]

Well we don't support running OS6 on the Pro, so QA wouldn't test updates against that.

Having said that you could give ReadyNASOS 6.9.4-T8 a try if you like.

[Alein]

Well, that is true that most of them you do not support, but RN2120 as I remember is working with OS6 officially.

It would be nice if QA would try to run firmware on other devices so they could find potential problems even with officially supported models.

I can add another symptom.

When NAS freeze with LACP. You can try to disable ports from switch site. After you enable them, both will be up, but LACP will not renegotiate connection.

Perhaps those settings could help a bit someone.

root@NAS-X:~# cat /proc/sys/net/ipv4/tcp_mem
191826 255769 383652
root@NAS-X:~# cat /proc/sys/net/core/rmem_default
212992
root@NAS-X:~# cat /proc/sys/net/core/rmem_max
212992
root@NAS-X:~# cat /proc/sys/net/core/wmem_default
212992
root@NAS-X:~# cat /proc/sys/net/core/wmem_max
212992
root@NAS-X:~# cat /proc/sys/net/core/optmem_max
20480
root@NAS-X:~# echo 'net.core.wmem_max=12582912' >> /etc/sysctl.conf
root@NAS-X:~# echo 'net.core.rmem_max=12582912' >> /etc/sysctl.conf
root@NAS-X:~# echo 'net.ipv4.tcp_rmem= 191826 255769 12582912' >> /etc/sysctl.conf
root@NAS-X:~# echo 'net.ipv4.tcp_wmem= 191826 255769 12582912' >> /etc/sysctl.conf
root@NAS-X:~# echo 'net.ipv4.tcp_window_scaling = 1' >> /etc/sysctl.conf
root@NAS-X:~# echo 'net.ipv4.tcp_timestamps = 1' >> /etc/sysctl.conf
root@NAS-X:~# echo 'net.ipv4.tcp_sack = 1' >> /etc/sysctl.conf
root@NAS-X:~# echo 'net.core.netdev_max_backlog = 5000' >> /etc/sysctl.conf
root@NAS-X:~# sysctl -p
vm.oom_dump_tasks = 1
vm.dirty_background_bytes = 134217728
vm.dirty_bytes = 536870912
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.tcp_min_tso_segs = 22
dev.raid.speed_limit_min = 30000
kernel.core_pattern = /var/cores/core-%e
kernel.perf_event_max_sample_rate = 25000
net.core.wmem_max = 12582912
net.core.rmem_max = 12582912
net.ipv4.tcp_rmem = 191826 255769 12582912
net.ipv4.tcp_wmem = 191826 255769 12582912
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.core.netdev_max_backlog = 5000
root@NAS-X:~#

[firefly_242]

Hi,

Am aware that OS6 is not supported ion Legacy models.

Decided to install the Beta. System under observation.

[firefly_242]

Hi,

1 day afer the  ReadyNASOS 6.9.4-T8 update I experienced again a freeze of the system:(

Unlike last time no change in the smart_history.log file. Last entry is from 1 May.

[Alein]

On 150% it is not a problem with HDD.

Try one SSD and you will get this same freeze.

Try this network optimization which I send before, but make current config backup.

[StephenB]

@firefly_242 - are you using just one NIC, or are you using some form of link aggregation?  If you are using multiple NICs, can you give us some info on the configuration?

Also, despite @Alein's assertion, @mdgm-ntgr found some evidence of a disk issue with one of your disks.  The vendor tools have fairly high pass/fail thresholds (which IMO is mostly because they want to control their warranty costs).  Can you download the logs again, and look at the SMART stats?  They are in multiple logs, one convenient place is volume.log.  While you are there, you can also check OS partition fullness (look at /dev/md0 in the === df -h === section of the log).

You can also look at network_settings.log and check the packet statistics for TX and RX packets.

[firefly_242]

tried to reply several times, but my reply is not posted

[firefly_242]

looks like the reply was too long:( Let's try the short version.

use only 1 NIC. SMART stats all show Zero for all disks. /dev/md0  is only 17%. TX and RX look good. No errors reported

[StephenB]

Your issue wasn't the post length.  There is a spam filter that sometimes traps legit posts, and it triggered on yours. I can release them if you like, though perhaps it would just make the thread a bit confusing.

---

@firefly_242 wrote:

SMART stats all show Zero for all disks. /dev/md0  is only 17%. TX and RX look good. 

---

Ok, that rules out ethernet cabling, and a filling OS partition.  Disks look fine (I did check your quarantined posts, and saw the stats).  Although SMART isn't infallible, it seems unlikely that they are the problem.

One possibility is just insufficient memory.  A stock Pro has 1 GB, currently shipping OS-6 NAS have at least 2 GB.

It could also be failing hardware.  That could be bad memory, and I have seen a couple of cases where replacing the PSU solved similar freezes.  Those two components can be replaced (as can the CPU), but the rest of the chassis isn't repairable.

The logs might give some more clues - perhaps look in system.log and kernel.log and see if there are any entries related to OOM (out of memory) conditions.  Look also for any evidence that processes are crashing (exceptions for instance).

[firefly_242]

ok, got it:) makes no sense to release those replies as it will indeed overflow this thread.

Can't find any OOM in system.log and kernel.log.

However in kernel.log I see this last entry before I had to reboot:

May 19 17:43:38 nas-storage kernel: Fixing recursive fault but reboot is needed!

[firefly_242]

checked the logs from 1 May crash and can't find the same message when the system hang

[StephenB]

Ok.  Do you have the stock 1 GB of RAM?

If so, a risk buy of more RAM might be a reasonable step.  It eliminates one possibility, and might help overall performance/stability too  4 GB total gives you the same amount of RAM as the RN500 series, so it should be enough..

This memory should work, and if you are in the US would cost about $30.   https://www.amazon.com/dp/B002N52ZO4

FWIW, I went with 2x4GB Patriot RAM in my pro-6 some years ago, but there weren't a lot of PCs that could use that memory, and at this point anything over 2GB/slot is scarce/expensive.  

Of course you could simply revert back to OS 4.2, though it is a fair amount of work, and you might discover that the reliability issues remain.

[firefly_242]

ok, will give the memory update a shot. Did a internet check and parts are not available in the Philippines. will have to sort them from the US. Will take a some time.

BTW: already tried reverting to OS4.2 after my first experienced theses freezes when I first update to OS6. Also on OS4..2 I have the sam behaviour now. So, decided to stick with OS6

[firefly_242]

Update: took awhile to get the memory delivered to the Philippines, but finally got it. Bought the recommended one in the thread.

Strange thing is that after the installation of the 2 moduels the storage did not start up anymore. Only get the "READYNAS" screen.

Tried to go in to the boot menu, but also this is not possible.

Removed them again and reinserted the original 1GB memory and system boots up without any issues

Strange that the recommended memory blocks the system from booting. Any idea why?

[Sandshark]

First, verify they didn't send you the wrong memory.  Make sure it is DDR2 PC2 6400.

Then, there is always the possibility that one of the sticks or one of the slots on your NAS are bad.  Try each of the new sticks individually.

BTW, all it takes is 5V for the unit to say "ReadyNAS", so that's no clue as to where the problem lies.

[firefly_242]

Hi,

correct memory modules received.

Already tried using 1 new module yesterday and the system did not boot. Assumed that the module is not compatible. Did not try the second module. My bad. Turns out that the first module I use is faulty:( The second one works. Thanks for the hint.

Currently using the old 1GB and 1 new 2GB module.

[Sandshark]

The unit does not use dual-channel memory, so there is no advantage to making sure both sticks are the same, in case you just want to get a refund on the bad one.