ReadyNAS RN104 shutdown because disks exceed safe temperature after defrag - follow up thread
Hello @StephenB , hello all, I'm "reopening" my old thread from April 2020 because I have new details which may help to track down the issue. @StephenB sorry for tagging you directly but you were involved most in the original thread: RN104 shutdown because disks exceed safe temperature
-
My RN104 is currently running at firmware level 6.10.7. Yesterday, when I observed the "disks exceed safe temperature" after defrag, it was running at firmware level 6.10.6. - Changes since April 2020: I erased and recreated the RAID5 volume some months ago, following a suggestion in the original thread. Besides this, I disabled the alarm option to shut down the RN104 in case a disks exceeds the safe temperature.
-
Yesterday, by chance, I was able to monitor what's going on.
I received the notification that the defrag process was started on the RAID5 volume.
I logged into the admin web interface and changed to the system: performance page. I could see that the fan was rotating with about 1250rpm. The page did not show a disk temperature for disk 1, 2, and 3. The temperature of disk 4 was 27°C.
While defrag was still running, I tried to get the temperatures of disks 1, 2, and 3. I refreshed the browser, I changed the admin language to reload the pages, but no change. No temperature was displayed for disks 1, 2, and 3. I was opening the disk details by placing the mouse pointer on the green dot behind the disk, but the disk details window did not contain a temperature for disk 1, 2, and 3; not even an empty field for the temperature. During all the time, disk 4 was showing a temperature of 27°C and the fan was rotating with about 1250rpm.
At the same time, when the defrag process ended, all disks suddenly showed their temperature. For the disks in the middle, disk 2 and 3, the temperature was 60°C. The "Disk in channel 3 (Internal) exceeded safe temperature threshold (60 C)." was issued and at about the same time the fan speed increased to maximum which is 3233rpm. The disk temperatures very quickly decreased to normal values.
For me it looks like the RN104 is not getting the temperature values of the disk drives while defrag is running. Because of this, the RN104 firmware is not increasing the fan speed when the disk drives' temperature raises.
I will attach the screen shots I have to this thread.
I did some research in the forum already and found that disks may not return their temperature value when they sleep. I enabled disk sleep when idle for 10 minutes.
So it's possible the disks where sleeping and waking up when defrag started. I'm wondering why the RN104 firmware failed to get the disk temperatures while defrag was running? I downloaded the log files after observing the issue and before updating the RN104 from 6.10.6 to 6.10.7. I would be happy my observation helps to resolve the issue permanently. Regards, Holger
For me it looks like the RN104 is not getting the temperature values of the disk drives while defrag is running. Because of this, the RN104 firmware is not increasing the fan speed when the disk drives' temperature raises.
Possible - I don't know how often it reads the temps (and it might be a lower priority process than the defrag).
I did some research in the forum already and found that disks may not return their temperature value when they sleep. I enabled disk sleep when idle for 10 minutes.
So it's possible the disks where sleeping and waking up when defrag started.
Also possibly part of the puzzle. You could try disabling spindown for little while, and wait for the next defrag.
Some users have reported that keeping the NAS door open lowered their disk temps - so maybe try that also.
Though I am wondering if it's simply one of the disks starting to fail (and overheating). Though I have no idea why this only happens when defragging.
Re: ReadyNAS RN104 shutdown because disks exceed safe temperature after defrag - follow up thread
Hello StephenB, thanks for your prompt reply.
Yes, I cleaned everything inside the RN104 and the disk drives + trays, etc.
The RN104 has it's door open all the time, nothing is blocking the Air flow front and rear side. It's placed in a rack just for the RN104.
It's located in the cellar, room temperature about 15°C all the year. Although please keep in mind that the replacement fan I'm using has a very high air pressure, much more then the original fan. Nevertheless, the high air pressure does not help if the RN104 firmware is not detecting the disk temperature is raising up and therefore not increasing the fan speed. None of the disks is failing. I already rotated the disks and always the middle 2 disk drives reach the 60°C.
It seems to be not depending on the disk load, because it ONLY happens when defrag is running/ ended. It never happened for data scrubbing nor any other workload like host backups, data movements, whatever, only defrag.
I don't know who's maintaining the firmware or who's writing firmware fixes, but will it be possible to find out why the firmware is not able to refresh the disk temperature when defrag is running? Maybe defrag is locking out other processes, maybe because of the slow Marvel processor used here?
Although I'ld like to get this resolved in the RN104 firmware, my mitigation is clear: I'll keep the RN104 running, even if the disk temperature raises up to 60°C, because I know that after defrag it will cool down very quickly.
Re: ReadyNAS RN104 shutdown because disks exceed safe temperature after defrag - follow up thread
Hello, I have an update on this situation. Defrag is currently running. The performance website is showing all drives temperature and the fan speed, but it's not refreshing. It's not refreshing by itself nor it's possible to refresh the website manually. It's the only website not refreshing. All other admin pages refresh correctly.
It's not even possible to adjust the fan speed (cool, balanced, quiet).
To me it looks like the device access if completely blocked when defrag is running. Maybe defrag is locking low level device access to ensure data integrity?
-
I changed the defrag start to ensure the disk drives are spinned up when defrag starts.
I power on the ReadyNAS by scheduler at 4pm and start the defrag process by scheduler at 4:05pm.
The disks are going to sleep after 10 minutes inactivity.
Means, the disk show their temperature on the admin interface/ website when defrag starts.
Nevertheless, the disk temperature values never change as long as defrag is running.
-
Just in case someone's still interested in this issue.
Re: ReadyNAS RN104 shutdown because disks exceed safe temperature after defrag - follow up thread
I have similar issues with defrag and temps over 60 degrees. It started after I upgraded from 6.10.6 to 6.10.8. Prior that everything worked with no issues.
I have similar issues with defrag and temps over 60 degrees. It started after I upgraded from 6.10.6 to 6.10.8. Prior that everything worked with no issues.
My setup is 4x WD RED 10tb.
Are you also seeing the temp problem when you aren't running defrag?
Re: ReadyNAS RN104 shutdown because disks exceed safe temperature after defrag - follow up thread
It sounds like the process that reads the temperature and and adjusts the fan speed is getting locked out (no CPU time available) until the defrag process completes. Those processes hogging CPU time and making other things (like the admin interface) not work well has been an issue for a while, and it seems it's gotten worse with the latest updates.
Frankly, I'd quit running defrag. It's usefulness in a RAID system is limited. But that won't help you with a balance (which, hopefully, won't run long enough to matter) or a scrub (which will run long).
I have similar issues with defrag and temps over 60 degrees. It started after I upgraded from 6.10.6 to 6.10.8. Prior that everything worked with no issues.
My setup is 4x WD RED 10tb.
Are you also seeing the temp problem when you aren't running defrag?
No. Even 4 days scrub does not increase temps that much.
