× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

ReadyNAS Pro 6 crashed again

tony359
Apprentice

Re: ReadyNAS Pro 6 crashed again


@StephenB wrote:

@tony359 wrote:

and what could that be? As I said, the NAS is on a different IP address so all my network shares point to a different IP. 

 


Might not be linked to network activity at all.  There are background processes that kick in from time to time.  If you use indexing, that will get updated periodically, and of course AV (if enabled) will run scans from time to time.  Snapshot processing, and other btrfs stuff might also be kicking in.

 

 

 


@tony359 wrote:

 

Do we know where the temperature sensor is? Could that just be the NIC IC warming up a bit because it's being used and the temp sensor happens to be nearby? 


The one that's spiking appears to be the CPU temp, not the system temp.  The CPU sensor is part of the chip. In many

 

 

(Sorry not sure how to "break" the quote line...)

 

The temp spike is 100% linked to me plugging the ethernet cable to the NAS. I've taken notes of when I plug and unplug and the time matches the temperature spikes. 

 

The graph colours are misleading when they overlap. It's the system temp (see attached).

Message 51 of 191
Sandshark
Sensei

Re: ReadyNAS Pro 6 crashed again

But, clearly, the CPU temp is rising and falling before the system temp, and likely being it's main driver.  And as @StephenB noted, there appaers to be NIC activity when it does so.  Are you running ant-virus, ReadyCloud, and/or any other app that might be trying (and failing) to "phone home" on a regular basis?

Message 52 of 191
tony359
Apprentice

Re: ReadyNAS Pro 6 crashed again

@Sandshark 

Thanks.

Ignore the rise in CPU temp at 10am. That's me opening the patio door to get some fresh air in and then the heating starting. 🙂

 

If you check 2pm and 4.30pm temps, there is no CPU temp increase. Maybe a few degrees which I would should be the consequence of running the GUI on my browser. 

That, in my opinion, does not explain a 10C+ system temperature rise.

 

No, I literally have nothing running on this NAS. Only file services. 

 

Attached another example: that was me unplugging the ethernet for like 5 minutes and plugging it back again. CPU temp actually goes down a bit (I closed the GUI) but the system temp goes down 10C in a few minutes and up 10C again in another few minutes.

 

Again, this might be a red hearring but I find it weird. I will check with my thermal camera to see if there's anything weird happening on the mainboard. 

 

But most importantly,  the NAS has been up 3 days and 14 hours since I've disconnected it from the main network. As the NAS was never disappearing when being heavily used, I think this removes the "hardware failure" out of the equation. 

Message 53 of 191
Sandshark
Sensei

Re: ReadyNAS Pro 6 crashed again

Poor cooling could be a part of the equation, though maybe only because a component has become more sensitive to temperature than it should be.  I don't remember if one of the heat sinks on the Pro6 motherboard is on the NIC.  But there are some "stick-on" ones that can get knocked off when you service the power supply.  Any chance that happened?  A chip with residual thermal pad for heat sync attachment but missing the sink is going to heat faster than just a bare one, and much more than one with the sink.

 

When you hooked the NAS up differently, did you also move it to where it might cool differently?  One thing some ignore is that there needs to be sufficient space behind the unit to reduce back pressure on the fan and keep heat from building up locally.

 

What you are experiencing looks mostly like hysteresis in the fan profile.  Especially on the "quiet" setting, the profile allows the fan speed to stay low and the temperature to rise until a threshold is reached.  Then, it boosts the fan speed to cool it, but does so such that it drops lower than the point at which it would be increased again, so as not to constantly go back and forth between the settings.  And if your replacement for the PSU power supply has a lower CFM ratting than the original, that could affect how well the standard profile controls temp since the NAS can't control that fan.

 

I don't remember exactly how my Pro6 units ran WRT that, and I know I always used the "Balanced" setting and all had the stock fans or an exact replacement.  I really wish Netgear would expose the fan control parameters more in the GUI.  Especially on legacy units that have a CPU fan (newer units don't have one), I find that the temperature can be allowed to get rather high before the speed is increased, causing the speed boost increase to be a larger step than would have been necessary had it done so earlier.  The configuration for newer units often contain a hysteresis setting, where legacy units don't.  But you can only edit them via SSH and they get replaced at every OS update if you do.  The segment below is from the RN3220.conf profile.  Granted a rack-mount unit, but I have added a similar statement to my legacy RN4200's profile and the result calmed the speed switching down when it starts to reach the max temperature, though that is always driven by the CPU temp on mine.  Of course, the 4200 is also a rack-mount unit, so behavior may differ on a desktop model.

 

        label temp1 "System"
        set temp1_max 80
        set temp1_max_hyst 72

 

Try changing the cooling profile (Cool/Balanced/Quiet) and see if it changes the behavior more to your liking.  If that doesn't work, try adding a hysteresis statement to the configuration file.  I don't know if the Netgear fan control looks at the difference between the max and max_hyst and uses that hysteresis all the time or if it just affects when it approaches max.  The profile for the Pro6. is /etc/frontview/sensors/NV6.conf.  Or you can replace the symbolic link at ./etc/sensors.d/system.conf with a modified copy of the profile itself and not touch the default one.

Message 54 of 191
tony359
Apprentice

Re: ReadyNAS Pro 6 crashed again

The NAS normally lives in a cabinet which doesn't have a door. Because the back of the NAS was getting a bit on the warm side, I installed a fan at the back of the cabinet which blows air OUT of the cabinet itself. Right now it's sitting on the floor, so definitely better airflow but I feel it's never been a problem anyways. 

 

NAS is normally on Balanced. When I plug/unplug the ethernet, I don't hear the fans spinning down or anything changing. That said, I noticed the PSU fan is blowing inwards which doesn't make much sense but I must have checked when I replaced it. Anyways, the PSU airflow is independent from the NAS. The air is sucked in at the back and blown out at the bottom of the NAS. There is a very small vent on a side inside the NAS but I doubt that could make a big difference. That said, I replaced that fan months ago. 

 

Heatsink: anything is possible but I'd say no. I can check though. Still, I'd be curious to know where the System temperature sensor is. 

 

I can try playing with the fan controls but I honestly don't think it's a fan issue. I've carefully listened and "felt" the fans when I unplugged the RJ45 (as I had the same idea, fans slowing down) but I didn't feel anything. The temp is going up 10C in a few minutes so I'd expect the fans to change their speeds by a lot to justify that change in temperature. I'll measure the temperature of the exhausted air to see if there is any change when I plug/unplug the RJ45.

 

In the end, the NAS is staying up even when connected to my main desktop (still no internet). That is good news. This whole temperature thing is just a bell ringing for attention, it might be a bug or something unrelated or totally normal. Who knows. 

Message 55 of 191
tony359
Apprentice

Re: ReadyNAS Pro 6 crashed again

I found something weird 🙂

 

I got the thermal camera out. And indeed there is an IC under a heatsink which immediately warms up as soon as the RJ45 is plugged. I can see it warms up some of the adjacent PCB so I guess that's the cause of our temp spike.

 

However.

This only happens when the bottom ethernet port is used - which I don't normally use. If I plug the network cable in the top one, the IC does not warm up.

 

We could conclude that that ethernet port is faulty but... it works! I have a feeling this is normal behaviour to be honest. 

But to be on the safe side I'll configure my usual ethernet port and use that. After all when the box "crashes" the first thing that happens is that it goes offline. A faulty network IC could the the cause of all of this. 

Message 56 of 191
Sandshark
Sensei

Re: ReadyNAS Pro 6 crashed again

I think I see the other heat sink in your photo -- it's in the rear, left corner of the NAS (upper left in your photo).  And I suspect you'll find that one heats up when you use the other port.  That other one is the one that's easy to knock loose when servicing the power supply, so make sure it seems to be solidly attached.

Message 57 of 191
tony359
Apprentice

Re: ReadyNAS Pro 6 crashed again

I really don't see anything warming up when I plug the other port, I did look for that. But for now I shall assume that what I see is normal and that somehow the NIC controller of the bottom port is closer to where the temperature sensor of the mainboard is.

 

I've switched to the upper port, which is the one I normally use. Let's see if the NAS stays online. If it does, I'll move to the next step: link my shared folder to the new IP and use it - just not on the main network. 

Finger crossed. 

Message 58 of 191
tony359
Apprentice

Re: ReadyNAS Pro 6 crashed again

Not sure how much the community is interested to hear an update but the NAS has been up for 6 days now with no issues. Next step is to set my Desktop to use the NAS as previously but on the second NIC I have on my system - hence the NAS is only reachable by one computer. 

 

This is weird though. Clearly not an HW issue but I really don't have anything running on that box so it shouldn't make any difference if it's online or not. We shall see.

Message 59 of 191
StephenB
Guru

Re: ReadyNAS Pro 6 crashed again


@tony359 wrote:

 

This is weird though. Clearly not an HW issue but I really don't have anything running on that box so it shouldn't make any difference if it's online or not. We shall see.


Do you have any ports forwarded to the NAS in the router?  Maybe if uPNP is enabled in the router, you could try disabling it.

Message 60 of 191
tony359
Apprentice

Re: ReadyNAS Pro 6 crashed again

I do, as I said I have FTP forwarded from the internet and my ISP gives me a fixed IP address. The port is not the standard one so it should be safe but you never know.

 

I need to educate myself on uPNP, I know it's there but I don't know what it does. What are your suspects on uPNP? 

Message 61 of 191
StephenB
Guru

Re: ReadyNAS Pro 6 crashed again


@tony359 wrote:

I do, as I said I have FTP forwarded from the internet and my ISP gives me a fixed IP address. The port is not the standard one so it should be safe but you never know.

 


Maybe try connecting the NAS tn the full network again, after you disable the port forwarding, and disable uPNP if it is enabled.

 

Something seems to be causing the NAS to crash, and it would be good to know if it is a device connected to your network, or something outside.

 

 


@tony359 wrote:

 

I need to educate myself on uPNP, I know it's there but I don't know what it does. What are your suspects on uPNP? 


uPNP (universal plug and play) allows any device on your network to ask the router to open a port.  Convenient in some ways, but also dangerous.  Best to disable it if you don't really need it.

Message 62 of 191
tony359
Apprentice

Re: ReadyNAS Pro 6 crashed again

Sure, if the NAS works now it's back on my small network, actively used as a NAS by my desktop only, the next step would be to put it back where it was but with no internet access whatsoever, including external access. Good to know about uPNP, which I can disable. 

 

Finger crossed 🙂

Message 63 of 191
tony359
Apprentice

Re: ReadyNAS Pro 6 crashed again

Another update: NAS is still up (10 days now) and been on my desktop network (isolated by the main network) for 4 days now. It works. 

 

The next step is to put it back where it was, but with port forwarding disabled.

 

Good progress though. From "you need a new one" to "it's been up 10 days" 🙂 These things are still pretty expensive!

I was looking into TrueNAS and I was told the 8GB requirements has been recently moved to 16GB - which feels silly for a NAS. Even 8GB feels quite a lot if you only intend to use it as a file server.

Message 64 of 191
tony359
Apprentice

Re: ReadyNAS Pro 6 crashed again

Well, the NAS is back where it was, main network with internet access. Only difference is that I disabled the port forwarding and Universal PnP.

 

It works. It's been online for 18 days in total - including the initial tests off network. 

 

If this works, that's great! But if it does, it might indicate a firmware issue. I wouldn't expect a NAS to crash because it's being accessed from the internet. I appreciate Netgear does not seem to be very busy developing for the ReadyNAS (and my NAS is unsupported) but is there a way to forward this to their engineers? I have a feeling this might not be just my problem, it's just that not many people have port forwarding on their network I suppose 🙂

 

Now, I can VPN on my network but every now and then I used the FTP to give people access to large files stored on my NAS. Is there a safe alternative than opening ports on my router?

 

I'll update again in a while. For now, thanks to everybody for helping me. 

 

Message 65 of 191
StephenB
Guru

Re: ReadyNAS Pro 6 crashed again


@tony359 wrote:

 

If this works, that's great! But if it does, it might indicate a firmware issue. I wouldn't expect a NAS to crash because it's being accessed from the internet.

 


Obviously it shouldn't crash, but hacking attempts could result in crashes.

 


@tony359 wrote:

Well, the NAS is back where it was, main network with internet access. Only difference is that I disabled the port forwarding and Universal PnP.

 


uPNP is pretty dangerous, because if any device on your network is compromised, it can tell your router to forward traffic to any other device - making it much easier for your entire network to become compromised.  I don't know why vendors put it their router firmware to begin with.

 


@tony359 wrote:

Now, I can VPN on my network but every now and then I used the FTP to give people access to large files stored on my NAS. Is there a safe alternative than opening ports on my router?

 


A safer method would be to use a cloud repository for what you want to share, though if the files are large that wouldn't be free. Then copy things that you want to share into the cloud repository.

 

Not sure how you are set up now.  But if you want to use FTP, you can set up the NAS to require FTPS ("Enable Force FTPS" in system->settings->services->FTP).  Limit the passive port range (the NAS default is far too big).  Generally I've found that 4 passive ports per simultaneous connections is enough.  So if you are sharing with a couple of people you'd need to forward only 5 ports total.  Best to use dynamic/private ports (any set of five in the range 49152-65535).  If you are sharing specific files, then don't leave the ports forwarded all the time.

 

That will somewhat complicate the sharing - you'd need to create a user account/password for shared access, and change the password from time to time.  And you'd need to provide more instructions on how to set up their FTP client.  For instance giving them

  • the non-standard connection port.  
  • the username/password
  • telling them to use "require explicit FTP over TLS" (FileZilla) or ""TLS/SSL explicit encryption" (WinSCP).

 

Message 66 of 191
tony359
Apprentice

Re: ReadyNAS Pro 6 crashed again

Well from my point of view an OS should be solid enough to be able to withstand a hack attempt without crashing. 

I'll keep Upnp disabled, I read online it was safe nowadays but I see your point.

 

Online services for sharing are good but that implies me sending the file there and the other person downloading it. It takes more time. Also, I might not feel comfortable using those services for certain more sensitive data.

 

I don't have force SFTP currently so that's a start. I only have 5 passive ports enabled. And yes, I've always given those info to those who needed to access my NAS. 

 

I'm not sure I want to test this now, given the headache I've been through! 🙂 Thanks for your input though. 

Message 67 of 191
tony359
Apprentice

Re: ReadyNAS Pro 6 crashed again

After 1.5 month of solid rock performance... the NAS disappeared from the network yesterday. And again today.

 

For the life of me I cannot understand what's going on. I guess I'll have to build that VGA cable after all 😞

Message 68 of 191
Sandshark
Sensei

Re: ReadyNAS Pro 6 crashed again

Those cables are still available for purchase, though they are becoming more scarce as VGA becomes more obsolete.  Examples: https://www.ebay.com/itm/184609738277?hash=item2afb990e25:g:2yEAAOSwKo5j8TBh or https://www.pccables.com/VGA-Port-HD15F-Adapter-to-IDC16.html .

 

As for file transfer, OwnCloud and NextCloud are also options.

Message 69 of 191
tony359
Apprentice

Re: ReadyNAS Pro 6 crashed again

Thank you,

 

I already have the bits to make my own, I got them when the NAS kept crashing two months ago. 

Thanks for suggesting the file transfer services, I'll take a look for sure!

Message 70 of 191
tony359
Apprentice

Re: ReadyNAS Pro 6 crashed again

Ok, I made the cable. I plugged it into the NAS - it had disappeared again - and I had a black screen, but the monitor reported a signal, 640x480@67hz.

 

Nothing would show up so at some point I issued CTRL-ALT-DEL twice and the box rebooted within seconds. 

 

The BIOS appeared on screen so I know the adaptor is correct. How can I display a terminal on screen when it's loaded? The monitor should be able to handle weird resolutions.

 

The CTRL-ALT-DEL trick per se is a life changer at least I don't have to power cycle the box - even though it rebooted very quickly but it cannot be worse than a power switch. That also tells me - correct me if I am mistaken - that the core of the OS was still working? Had it been fully frozen it wouldn't have responded to the keyboard.

 

Thanks

Tony

Message 71 of 191
Sandshark
Sensei

Re: ReadyNAS Pro 6 crashed again

You saw no messages during boot at all?  AFAIK, you have to have SSH enabled to get a terminal once the OS is fully booted, but you should see some messages from the boot process.  The BIOS will respond to the keyboard, but I don't know if CTL-ALT-DEL is something it understands.  But I would expect it to display some sort of error message if it can't see the boot code on the flash.

Message 72 of 191
tony359
Apprentice

Re: ReadyNAS Pro 6 crashed again

I see the BIOS and the early stages of boot process of SYSLINUX 3.31. 

 

Then black screen - black screen, NOT no signal.

 

SSH is enabled. 

 

Well, a modern OS should initiate a software reboot if CTRL-ALT-DEL is issues. I wouldn't expect the BIOS to take over and forcefully reboot the system. But this is an old system and also non-standard. Still, a forced reboot is better than powe cycling the whole thing!

 

Here is all I see on screen: https://www.icloud.com/photos/#0eel-87kln4OEJQ0UlY8hhMNg

Message 73 of 191
tony359
Apprentice

Re: ReadyNAS Pro 6 crashed again

The NAS keeps crashing as a couple of months ago...

 

This time no response from CTRL-ALT-DEL. In fact, plugging the keyboard does not light up any of the LEDs and CAPS-LOCK does not work. It did work earlier on when the NAS was gone from the network.

 

I power cycled and got into the BIOS, not sure why. There I found a clue. Only 4HDD were being detected! The first one - the oldest - was not. 

 

I saved and exit, got in the BIOS again and all 5 drives were there.

 

I'm wondering whether that old HDD is dying? There is no evidence of that though. SMART is clean. But it could be one of those weird faults the system cannot handle and just crashes. It wouldn't be the first time I see that. 

 

Ideas? I have a feeling that if the drive is the culprit, it would pass any tests I throw at it until it stops working altogether. 

Message 74 of 191
StephenB
Guru

Re: ReadyNAS Pro 6 crashed again


@tony359 wrote:

 

I'm wondering whether that old HDD is dying? There is no evidence of that though. SMART is clean.


It could be - I've sometimes seen failing drives with clean smart stats.

 

If you have ssh enabled , try smartctl with the -x option.  That can show failures that aren't in the stats.

Message 75 of 191
Top Contributors
Discussion stats
  • 190 replies
  • 4285 views
  • 7 kudos
  • 7 in conversation
Announcements