Re: ReadyNAS Pro 6 crashed again

tony359 · ‎2023-06-06

Thank you for your input, really much appreciated.

I cannot remember if I tried accessing via the second port - it's been many months of tests! I'll set it up and keep it disconnected and try accessing only when it hangs.

My Linux skills are limited - but I learn fast 🙂

is this https://www.debian.org/CD/live/ ok to make a live USB install? Do I boot it by changing the boot order in the BIOS?

Once booted does it take me to terminal or GUI?

So I'd imagine I boot into Debian CD, then I issue

btrfs-check --readonly /dev/sda (or whatever drives I want to check).

and I repeat for all the other drives?

I guess this should not affect the NAS but as usual nobody can guarantee that 🙂

This is the only reference of CSUM I have in my logs

D:\Downloads\System_log-Enterprise-NAS-20230531-191137\btrfs.log (21 hits)
	Line 130: csum_type		0 (crc32c)
	Line 131: csum_size		4
	Line 132: csum			0xa36bf3b8 [match]
	Line 195: 		backup_csum_root:	34406400	gen: 621032	level: 1
	Line 206: 		backup_csum_root:	39092224	gen: 621033	level: 1
	Line 217: 		backup_csum_root:	32161792	gen: 621029	level: 1
	Line 228: 		backup_csum_root:	32538624	gen: 621030	level: 1
	Line 236: csum_type		0 (crc32c)
	Line 237: csum_size		4
	Line 238: csum			0x881ca9bc [match]
	Line 301: 		backup_csum_root:	17009663148032	gen: 166700	level: 1
	Line 312: 		backup_csum_root:	17009663148032	gen: 166700	level: 1
	Line 323: 		backup_csum_root:	17009663148032	gen: 166700	level: 1
	Line 334: 		backup_csum_root:	17009663148032	gen: 166700	level: 1
	Line 342: csum_type		0 (crc32c)
	Line 343: csum_size		4
	Line 344: csum			0xf5e3c214 [match]
	Line 407: 		backup_csum_root:	17009663148032	gen: 166700	level: 1
	Line 418: 		backup_csum_root:	17009663148032	gen: 166700	level: 1
	Line 429: 		backup_csum_root:	17009663148032	gen: 166700	level: 1
	Line 440: 		backup_csum_root:	17009663148032	gen: 166700	level: 1

tony359 · ‎2023-06-06

Ok good news. I can access the NAS via the second NIC when it goes down. It was very slow authenticating but then it sped up.

Here it shows up my lack of linux knowledge to be honest! I wanted to reset the network (or maybe just one of the NICs) but ifdown and ifup are not available?

Also traceroute is not available?

fdisk -l correctly lists all the drives.

What would you suggest I checked once I've got access?

tony359 · ‎2023-06-06

Update.

I found

ifconfig ETH1 up|down

Disabling ETH1 and bringing it back up, made the NAS appear again on my other network. At that point, logging in via SSH did not show any delay as before and everything seems to be working normally.

So it looks like it could be a NIC issue - even though I know that sometimes when I plug a keyboard, the CAPS LOCK button does not respond, showing a complete unresponsive system.

Two scenarios I guess

1. Faulty NIC HW

2. Software issue involving the NIC and when the NIC is in that weird state, everything which is using the network hangs

Any ideas?

StephenB · ‎2023-06-06

@tony359 wrote:

Two scenarios I guess

1. Faulty NIC HW

2. Software issue involving the NIC and when the NIC is in that weird state, everything which is using the network hangs

Are both NICs always connected to the switch? Or did you move the cable to the second NIC???

tony359 · ‎2023-06-06

The second NAS NIC would directly connect to the second NIC on my desktop. NIC1 and NIC2 are on a different network range of course.

For this test, the secondary NIC was physically unplugged - but configured and ready to go. When the NAS disappeared from the main network, I plugged the secondary cable to the NAS and I could see the NAS from my desktop.

I thought of keeping the second NAS NIC disconnected rather than always connected. I guess I'll leave them both on for a test - but one NIC is going through the switch and to the main network, the other is only directed connected to the desktop.

Under normal circumstances, I would only use one NIC on the NAS.

tony359 · ‎2023-06-07

Just for the Hardware Geeks, I decided to replace the capacitors of the PSU anyways. I know, waste of time. And massive one as those large traces with wired made the process very difficult.

HOWEVER, 90% of the caps were bulged and were reading either nothing or a fraction of the original capacitance!! But the NAS was working, I wonder how.

And I wonder how is it even possible that the replacement (temporary) PSU hasn't fixed the issue!

I think I'll replace the Seasonic anyways but a small PSU is always handy so I'll fix it anyways.

The NAS didn't disappear over the past couple of days.

KDS · ‎2023-06-09

Just on an off chance you haven't tried yet.

CPU & RAM swap out?

tony359 · ‎2023-06-09

Considering it’s happening very rarely it’s not such a bad idea.
I’ve run overnights of ram tests but maybe it didn’t catch it because it happens very rarely.
I still have the original CPU so I could try that too.

That said, the fact that just the network went down last time is suspicious. A ram or cpu issue would have much bigger impact I reckon. I might want to put a switch in between the nas and the main switch. It’s always been that switch and maybe it’s faulty. After all the nas stopped crashing when I took it off the main network - which takes the main switch out of the equation.
And it worked for a while while connected to my main desktop, again no main switch involved.

Uhm… I like this idea 🙂

KDS · ‎2023-06-09

I have my router dishing out DHCP addresses>>>Unmanaged 2.5G switch>>>both NICs into switch.

Static IP's on both Netgear NIC settings (IPV4) and router address.

Router set to static IP addresses for both NICs.

Since doing that both NICs are very stable.

Ram is 2 x 2GB PC800.

CPU is now E7600, Just upgraded from E5300, find this much faster than the Q6600, though my NAS is mainly used for backup, and file server, not really serving any Apps. E7600 runs faster and much cooler than Q6600.

StephenB · ‎2023-06-09

@tony359 wrote:
I might want to put a switch in between the nas and the main switch. 🙂

Makes sense. You could also swap the two connections, and see if the problem moves.

Sandshark · ‎2023-06-09

Is it a "green" switch? I've had a couple issues with ReadyNAS and green switches, though I've believed the problem units already had partly damaged LAN ports. My main switch has a "green" on/off selection. Try turning off power saving mode if yours does. Otherwise, a non-green switch in between might be the answer.

tony359 · ‎2023-06-09

It's a Netgear! 🙂

GS108Ev2. "partly" managed. V1.00.12 (latest). DHCP disabled. DHCP is handled by the router (Fritzbox) which issues the same IP to the NAS MAC address. All settings are default to be honest.

I have tried a static IP in the past with no change - though, I'm confident those swollen capacitors might have contributed to SOME of the issues I was having.

Today's new issue is... the NAS is online, I can see the files. I can SSH into it. But web interface shows an "500 - internal server error". This is on both ports. Sigh 🙂

Before I just reboot the box, how would I restart the web interface from SSH?

I'll install a dumb switch between the NAS and the main switch - with new cables.

The 7600 seems to be a good option. It's only 2 cores but it's faster than the cores in the 6600. I wonder how much a NAS used as a "file system" is actually using a multi-core CPU. And the 7600 as you say is cooler.

I think I'll fix this issue first then I might try the 7600 as well, thanks for the hint!

tony359 · ‎2023-06-09

I feel that the below is relevant with my issue. Again, the NAS is accessible, I can write a file on the data folder via nano. I just lost the web interface.

These weird failures are incredibly annoying. I'd like to test what @itachi2 recommended, can someone possibly point me to the right direction? See https://community.netgear.com/t5/Using-your-ReadyNAS-in-Business/ReadyNAS-Pro-6-crashed-again/m-p/23...

Thanks 🙂

root@Enterprise-NAS:/# systemctl status apache2
Failed to get properties: Activation of org.freedesktop.systemd1 timed out
root@Enterprise-NAS:/#
root@Enterprise-NAS:/#
root@Enterprise-NAS:/#
root@Enterprise-NAS:/#
root@Enterprise-NAS:/# systemctl restart apache2
Failed to restart apache2.service: Activation of org.freedesktop.systemd1 timed out
See system logs and 'systemctl status apache2.service' for details.
root@Enterprise-NAS:/# sudo systemctl status apache2
-bash: sudo: command not found
root@Enterprise-NAS:/# su
root@Enterprise-NAS:/# systemctl status apache2
Failed to get properties: Activation of org.freedesktop.systemd1 timed out
root@Enterprise-NAS:/# systemctl status readynasd
Failed to get properties: Activation of org.freedesktop.systemd1 timed out
root@Enterprise-NAS:/# ps aux | grep readynasd
root     26625  0.0  0.0  17836  1008 pts/2    S+   19:58   0:00 grep readynasd
root@Enterprise-NAS:/# service ctscand stop
Failed to stop ctscand.service: Connection timed out
See system logs and 'systemctl status ctscand.service' for details.
Failed to get load state of ctscand.service: Connection timed out
root@Enterprise-NAS:/# systemctl restart readynasd
Failed to restart readynasd.service: Activation of org.freedesktop.systemd1 timed out
See system logs and 'systemctl status readynasd.service' for details.
root@Enterprise-NAS:/# systemctl status readynasd.service
Failed to get properties: Activation of org.freedesktop.systemd1 timed out

KDS · ‎2023-06-09

Just another hardware thing that has probably already happened.

1. After good PSU installed was CMOS cleared?

2. Has CMOS battery been checked?

3. Are you keeping it simple with just 1 HDD, possibly 2 (raid 1), with HDDs especially raid arrays cleaned and cleared on another PC prior to installing. Granted you may have data on your system, though remove those HDDs and start fresh, with known clean and good drives? I tested with some old 320GB junk drives I had kicking about. I also encountered NIC, web access, and HDD problems prior to replacing the PSU. My original 7200 WD HDDs were only seen as 5900, then when I added a newer 7200 WD HDD it was seen as 7200, it did not like the mismatch in HDD speed that it saw.

Though finally did clean HDDs. I think Web interface may be associated with what is already on the HDDs.

My HDD and hardware issues were resolved when I replaced PSU. Both types drives 3 x 7200 seen as 5900, and 3 x 7200 seen as 7200, and running together fine.

4. BTW are you using RAIDar 6.5.0.

tony359 · ‎2023-06-09

Just another hardware thing that has probably already happened.

1. After good PSU installed was CMOS cleared?

--

No, I did not update the BIOS so I didn't think of clearing the CMOS. I can try.

2. Has CMOS battery been checked?

--

No. Good point.

3. Are you keeping it simple with just 1 HDD, possibly 2 (raid 1), with HDDs especially raid arrays cleaned and cleared

--

No. Reason is: last time the system behaved, it lasted for 2 months. I cannot stay without my data for 2 months.

The only two options here are

a. Fix it with the current setup

b. try a factory default and migrate a backup

Testing with 2 random HDDs is likely not gaining any evidence I'm afraid.

I also encountered NIC, web access, and HDD problems prior to replacing the PSU. My original 7200 WD HDDs were only seen as 5900, then when I added a newer 7200 WD HDD it was seen as 7200, it did not like the mismatch in HDD speed that it saw.

--

Unfortunately the replacement PSU did not solve all the problems. I'm confident some of the issues I experienced were caused by the bad PSU but the NAS is still misbehaving I'm afraid.

All my HDDs are WD RED, 5400-ish (4TB are a bit slower than the 6TB).

4. BTW are you using RAIDar 6.5.0.

--

No. I am on OS6.

I appreciate a factory reset would be a good idea but I have 13TB on that NAS and I don't know where to store them for a backup. Yes, the NAS is more or less fully backed up (locally and online) but it would take me forever to restore those backups so I'd consider that as an emergency option only.

I could see if I could hire another NAS, transfer the data, reset and restore. But somehow I am not confident my problems would go away 🙂

Thanks for your input!

tony359 · ‎2023-06-10

Little update.

I checked the battery, it's ok, 3.1V. I replaced it some time ago when I serviced the box.

I re-reset the BIOS (only thing I change is the default fan speed!)

I swapped position of HDD0 with HDD4. I sprayed dry contact cleaner on the backplane and on the HDDs, cleaned with a small q-tip.

Once the NAS was powered up again, HDD0 failed to show up on the BIOS splash page straight away. So it's not the HDD and, to be honest, I feel that that might be a red herring. I never had issues with HDD0 so maybe it's a BIOS bug which then does not affect the software. No idea. But I now know it's not the drive.

I've added a TP-Link switch between the main switch and the NAS.

Next: throwing the NAS out of the window.

tony359 · ‎2023-06-11

And no, the NAS disappeared again.

Solution: SSH into other port and ifconfig the other port DOWN and then UP again.

I could try swapping the config but I think I tried that in the past already.

If someone could give me some directions for checking the HDDs offline as mentioned above, that would be great! 🙂

Thanks

StephenB · ‎2023-06-11

@tony359 wrote:

And no, the NAS disappeared again.

Solution: SSH into other port and ifconfig the other port DOWN and then UP again.

Have you tried swapping the NIC ports?

tony359 · ‎2023-06-11

That's what I meant with "swapping the config" sorry. As in swap the IP addresses between ports.

I'll try but I think I tried that in the past already. 100% worth a try.

StephenB · ‎2023-06-12

@tony359 wrote:

That's what I meant with "swapping the config" sorry. As in swap the IP addresses between ports.

I meant connecting the ethernet going to the PC to the switch, and vice versa. Then seeing if the problem was limited to NIC 1.

tony359 · ‎2023-06-12

The NICs are on two different IP range - one main network, one PC only.

What used to be on main network is now directly connected to the PC and what used to be connected to the PC is now connected to the main network and I've swapped the IP addresses accordingly.

I did that yesterday and I've just checked: NAS has disappeared. Sigh!

I SSH'd through the other NIC, restarted it and it worked as usual.

So

- It's not the specific NIC

- It's not the switch

It's curious that it's always the NIC on the main network failing and not the other.

Help 🙂

schumaku · ‎2023-06-12

As you are in the lucky situation having an alternate LAN interface (and IP subnet) available. what does the kernel output show when the device "disappeared", ...?

# dmesg

The risk that a network adapter does become flakey is very small. More typical, the adapters resp. the data connectivity does completely disappear completely, and the UPnP OS does no longer detect the adapter.

Most problems on such NASes are caused by RAID becoming inoperable, due to aged or breaking storage blocks.

Do you have a known working, reliable SATA storage block at hand to set-up the NAS with one single device volume, or two on a RAID 1 volume? Remove the potentially unhealthy storage blocks, and restart a test from scratch.

tony359 · ‎2023-06-12

@schumaku

Thanks, I'll test next time.

Many have (rightly) recommended a test with a couple of random HDDs. I have plenty so that wouldn't be an issue.

My concern is that sometimes the NAS stays online for weeks without issues and I really cannot keep my data offline for so long.

Is there a way to do an offline test of my drives? Someone recommended booting from a Debian Live-USB but I would need some minor guidance on that. I know how to make the USB, I'm just making sure (as much as possible) I don't do anything that can destroy my data.

Thanks! 🙂

schumaku · ‎2023-06-12

Start with retrieving the SMART data from the storage block (aka. disk). Next trigger a full SMART check (rapid, then full) of the storage block. Then retrieve the SMART data again.

You can do this on any platform, without erasing or re-partition or re-format the storage block - if done carefully of course.

tony359 · ‎2023-06-12

Thanks.
I’ll Google how to do that. 🙂

Just to double check: do you mean doing those checks on the NAS itself while it’s online?