This topic has been marked solved and closed to new posts due to inactivity. We hope you'll join the conversation by posting to an open topic or starting a new one.
OS 6.6 on Ultra 6:. Not ready for prime time? Do I need more memory? Or...?
TL;DR version: OS 6.6 is worryingly unpredictable/unstable on my Ultra 6. Do I just need to expand the memory past the stock 1 GB, or is OS 6 still not ready for prime time, at least on these older boxes?
I've been running two stock Ultra 6s for the past few years on OS 4.2.x, and they've been rock-solid stable. WD Green and Red drives (Greens with the load-cycle fix enabled) in 2 TB, 3TB, and 4 TB sizes. Both NASes have the 6/10/2010 BIOS. Gigabit links to a WNDR3700 router.
Two weeks ago I updated my backup NAS to 6.6.0, and a couple of days ago updated it to 6.6.1.
I should mention that I've been swapping out old 2 TB drives for new 4 TB drives and resyncing/expanding. I'm running AFP, SMB, and NFS, but no apps and no antivirus.
I've had several issues.
1) One full lockup, after 2 days of operation: not visible on the network via AFP/SMB/ping, etc.; couldn't connect to the web UI (do we still call it Frontview?); no power button response. Had to hard-cycle power. An initial backup from the main NAS to the backup via rsync was in progress at the time.
2) Partial lockup, 2 days after a reboot: shares accessible, web UI usable, but the performance graphs stopped showing data a couple of hours before I noticed the issue. Resync in process, so front-panel display showed "data DEGRADED", but steadily, not flashing on and off as is normal in this mode. Could not shutdown or reboot through web UI; power button unresponsive; via SSH I got timeouts:
root@NAS2:~# reboot --reboot &
[1] 20240
root@NAS2:~# Failed to start reboot.target: Connection timed out
See system logs and 'systemctl status reboot.target' for details.
"systemctl status reboot.target" also timed out. I was able finally to hard-shutdown the system with "reboot -f". A resync was in process at the time (and resumed after restarting); a Really Big Backup Job (see below) had been running, but I'd manually cancelled it about ten hours prior to the lockup.
Despite both of these occurring after about two days of uptime, the NAS has also run for many days without a hang or lockup, so it's not a hard-wired two-day-fail.
3) A couple of instances where the NAS temporarily refused to allow AFP connections from my Macs. Entering username/password worked, but the Finder would sit for tens of seconds before reporting "Connection Failed". In this case, using the Connect To Server window usually worked, but other times, after selecting a share, I'd get "The share does not exist on the server". When this happens, sometimes it's for all users, sometimes just for one or two (out of four users set up), and it'll be consistent regardless of which Mac is used to connect, and regardless of whether it's running macOS 10.11 or 10.12. Meanwhile, these same Macs will happily connect to my NAS running OS 4. In all cases so far, the refusal to allow logins clears itself up after a period of time, without my doing anything.
4) My Really Big Backup Job (RBBJ), which rsyncs about 6.5 TB from the main NAS (OS 4) to the backup (OS 6) has failed about 24 or 36 hours in (for the initial backup, and then a backup tried after 4 days of changes on the main NAS). Usually, when there's less to be copied, the job runs fine, and my other, smaller jobs always run to completion. I notice that when the RBBJ executes, the system starts swapping after a few hours. The NAS has the stock 1 GB memory and when idle it's using 70%–95% of that memory and no swap space, but after a few hours minutes of rsync the swap starts getting used. Not a lot necessarily; only 76 KiB or so (at least that I've seen so far; I just started looking at this yesterday). I just started another backup running about ten minutes ago—my 2nd biggest job, about 2.5 GB—and the swap space is now up to around 335 KiB.
6) The CPU and SYS temperatures reported on OS 6 web UI are the reverse of what they were on OS 4:
On OS 4, CPU ~50º, SYS ~35º On OS 6, CPU ~35º, SYS ~50º
On my OS 4 NAS, the fan speed changes with workload and the temperatures remain within a degree or two of their normal values. On OS 6, the fan speed doesn't change much and the CPU reading is constant, but the SYS temp varies from 43º to 55º depending on workload. I wonder: did the CPU and SYS values get swapped in OS 6, and "SYS" is really the CPU, which heats up as it works harder (because the fan isn't ramping up since all it "sees" is the stable "CPU" reading)?
Through all this, I've been running volume expansions (one of the reasons to move to OS 6, as I was already maxed out on OS 4 without doing a factory reset). These hiccups/hangups have occurred with the volume 80%–90% full, and with snapshots being purged automatically to make room.
The first 4 TB drive I stuck in resynced and expanded in a day or so. The 2nd one initially reported a similar time estimate but, after a couple of hours, jumped to 200–300 hours (!); it's currently 41% done and reporting about 183 hours to go. Ouch! But at least, periodically, the volume grows; my 9 TB of data now occupy only about 71% of the volume instead of 90%.
Overall, OS 6 just doesn't feel as solid as OS 4. The reversed SYS and CPU temps; the display corruptions; the hangs/timeouts; the denied connections that resolve themselves... The occasional failure of the RBBJ makes me suspect memory (rsync is said to be memory-hungry in direct proportion to the number of files to be synced); current 6-drive ReadyNAS boxes seem to run 2–4 GB RAM. If I bump the RAM to 2–3 GB, will that make all my problems magically go away?
Or is OS 6.6 still just a wee bit unstable/unproven/un-debugged? Should I roll back to 4.2.28 or 4.2.30?
Or is something else going on that I've foolishly overlooked?
Re: OS 6.6 on Ultra 6:. Not ready for prime time? Do I need more memory? Or...?
Final update (I hope): I re-ran the upgrade from 4.2 to 6.6.1. The NAS has been working perfectly for over a week. What was different this time? (a) all disk were in place already, so no RAID expansion was needed; and (b) I went straight to 6.6.1, not 6.6.0.
I did order more memory, but haven't installed it; I'm still running the stock 1 GB. I wanted to replicate my previous issues first so I could verify whether increasing RAM alone would fix it. But the box has resolutely failed to show any issues at all: no hangups, no unexpected slowness, no failures to respond. I've tried things like running a scrub while launching backups and hitting the NAS with clients, and the system reacted more slowly when all that was going on, but even so: it all worked, and the scrub of 8.8 TB finished within 48 hours (as did the initial full backup, before I started the stress test). Incremental backups run as quickly now as they did on OS 4.
So I either had a subtle corruption during the initial upgrade, or stuffing two new disks in and expanding the volume triggered the problems. But it wasn't low RAM, apparently, and OS 6 seems to be running fine despite my best attempts to break it. I'm going to label it "bug report closed: cannot replicate."
As for the apparent mislabeling of the System and CPU temperatures? I edited /etc/frontview/sensors/ULTRA6.conf and swapped the "System" and "CPU" labels for temp2 and temp3, ran "systemctl restart readynasd", and now the readings in the web UI match the values I expected from version 4.2 (I got the idea from this thread: <https://community.netgear.com/t5/Using-your-ReadyNAS/ReadyNAS-4200V2-OS6-6-1-fan-control/m-p/1219655...
Re: OS 6.6 on Ultra 6:. Not ready for prime time? Do I need more memory? Or...?
I also have upgraded my Ultra 6 to 6.6.1 - CPU temp is locked at 35 degrees and doesn't budge. The system temperature is varying over time based on conditions - but seems incorrect.
Re: OS 6.6 on Ultra 6:. Not ready for prime time? Do I need more memory? Or...?
I found a file:
/etc/frontview/sensors/ULTRA6.conf
This is my conf. CPU temp is everything low (max 15-16 celsius), i'm sure it is wrong. (heatsink is warm-hot, with Arctic MX4 paste and a 40x40x10mm Noiseblocker XM2 fan, what is plugged the connector near chipset):
Re: OS 6.6 on Ultra 6:. Not ready for prime time? Do I need more memory? Or...?
When my Ultra6 frozed again and again, and sometimes couldnt boot up (just the "ReadyNAS" text, no BootMenu, and I needed to reset by cmos-battery out-waitabit-back), i tested the 2 memory module (Kingston KHX6400D2B1/2G HyperX blu), and I found one is bad (memtest86 on an old PC found the error, the NAS memtest just frozen on it.
Now I use the other HyperX (who passed memtest) and the factory module (passed test too), 3GB is enough for a home NAS, I am sure.
All fans replaced, Arctic Cooling F12 on back and a Cooling SWIF2-801 in the PSU, and now whisper-quiet. OS 6.6.1 on it.
Now I have problem with CPU-temp sensor and the fan controll only. (no changes between quiet-balanced-cool, 1087-1090 RPM everytime...)
Re: OS 6.6 on Ultra 6:. Not ready for prime time? Do I need more memory? Or...?
Another hang, after half a day of operation: front panel display frozen, power button inoperative, can't restart or shutdown from web UI, no updates to performance graphs since about 2:50am. Yet resync continues, shares are accessible (though very slowly) and backup jobs ran, completing at 5:21am. Swap topped out at 42196 KiB. "reboot -f" restated the NAS.
Re: OS 6.6 on Ultra 6:. Not ready for prime time? Do I need more memory? Or...?
On all of my 6-bay systems (which includes one Ultra 6) the temperatures have always seemed swapped to me, on 4.2.x as well as 6.x. Or at least I think they all were. The Ultra6 has only ever been a backup system, so it may have been reversed and I never noticed. They are quite similar to what you are experiencing with 6.6. Since my Ultra6 is just a backup, I booted it up 10 minutes ago to watch what happens. Drives are at 33°C to 35°C, CPU at 30°C, and system at 55°C. The system temperature started at about 43°C and rose quickly to the current temperature. I have only one 6-bay system not running 6.x. It's an Ultra6Plus. I booted it a couple minutes after the Ultra6. Note that it's not co-located, the environment may be a bit cooler, and it's got some older Seagate enterprise drives where the Ultra6 has WD reds.. The drives are 39° or 41°C, CPU at 24°C and system at 55°C. But here is the kicker -- the chart on the U;ltra6 OS6.6.0 system and the reported current temperature don't agree. The chart says the CPU temp stared at 0°C and is now at 27°C. The reported current CPU temperature has not moved from the original 30°C.
30°C may be a valid CPU temp at the point it took the first measurement. 0° clearly is not, nor is 27°C likely. And it's not that there is a single bad reading at 0° and it jumps to something valid -- it has been slowly rising. There is definately something amiss with the reporting of Ultra6 CPU temperature in OS6.6.0. The reported current temperature stays fixed at the boot temperature and the charted temperature has the wrong scale factor and bias. Yes, it's "unsupported", but I thought they had this right at some point.
Re: OS 6.6 on Ultra 6:. Not ready for prime time? Do I need more memory? Or...?
Two more hangups today, with frozen front-panel display, inop power button, cessation of data to performance graphs (and, I discovered, changing fan setting does nothing in these cases). Web UI still runs, resync runs (48% done, and only 160 hours to go!), shares still served on the network, SSh works, but no updating of fan speed or temperatures, no shutdown via front panel or web UI, no dowloading of logs from the web UI. I can still start/stop backup jobs from the web UI, though.
In the latest case, no backup jobs had run and no swap was ever used. Between that and mdgm's response, I'm ruling out low memory as the culprit.
kekesgm: I ran the memory test for 6 passes today; no errors. On my Ultra 6, setting the fan to "Cool" does ramp up the fan (as long as the box hasn't hung up, that is) and cools down "SYS" to 39º or 40º, but "Balanced" and "Quiet" either set the same condition around 43º or so, or are reversed (e.g., Balanced runs quieter and hotter than Quiet does), but I wasn't able to repeat my testing to see which was the case before the most recent hangup occurred.
Jophus: My disks run 35º–38º; SYS varies from 39º to 50º depending on fan setting and workload; CPU is, like yours, fixed at 35º no matter what.
Sandshark: On my Ultra 6 the CPU temp reading and the CPU temp graph agree: both are as fixed to 35º as if they'd been nailed there, and then had Gorrilla Glue poured over them to be on the safe side.
bedlam1: there are no dumb questions, only dumb answers (grin). I have no idea where the "System" sensor is on a Pro 4; that's a very different box than the Ultra 6, and whatever we learn here may or may not apply there. Mind you, I have no idea where the System sensor is on an Ultra 6 either.
I'm going to leave the hung system running until resync finishes or it locks up completely, just to see what happens, but my gut feeling at this point is that OS 6 isn't solid enough yet on an Ultra 6 to be trusted for deployment. 😞 Stay tuned...
Re: OS 6.6 on Ultra 6:. Not ready for prime time? Do I need more memory? Or...?
Try to test memory in a PC. Or just try to put back the original 1GB. It need to be enough, maybe slower. (at my workplace we have a half dozen of 8bay and 4pcs 12bay NAS from another brand, they both use 1GB Ram, enough for Fileserver, DNS server, Backup server, everything.)
Check the hardware. Is your PSU cleaned? My was full with dirt, the full is real full,after clean I can feel the PSU lighter (the weight). Not a joke, the hoover tray was half full after the cleaning. (didnt tried with compressed air, because a global warming start, if it goes into the atmosphere...)
Do you have antivirus ON? Maybe try without it.
Quiet-balanced-cool: On my box, it is mad. Sometimes the Quiet work on full RPM, Cool work on slow (700RPM, warning in frontview). Sometimes the opposite. Weird... It needs a fix in an update. Maybe Quiet is 7V for fan, Cool is 12V for fan, Balanced can be an automatic, by temperature.
CPU temp: If you see my config file, what i wrote, there is a line (compute....), what set a default minimum to the cputemp (35Celsius by factory default.) It is bad. the Atom CPU-s have the own sensor, if a PC or netbook can use it, a NAS can too, i am sure. Just the OS doesnt handle it fine.
I already thought to use the Arctic F12 TC fan, with own temp sensor (this one), but it can sense on the heatsink only, not in the core of CPU. But I think, a good fan and a good placement enough for keep the NAS in life. If you worry about the temp, a 40x40x10mm fan (like my Noiseblocker XM2) can help some degree.
System Temp Sensor is in Northbridge, I think. (not sure). I touched the chips, when the NAS worked on a big copy. (yeah, i am an extreme and horroristic sysadmin, beyond borders, checker T-shirt, drink cola&coffee, StarWars-fan, etc). The Northbridge was hot, all other warm or roomtemp.
I ordered a heatsink to Northbridge already (this one), and the conductive glue (this one), if I arrive, and used and tested, I will write the results.
Re: OS 6.6 on Ultra 6:. Not ready for prime time? Do I need more memory? Or...?
I have been running a variant of os6 on my Ultra 6 since mid 2015 with no stability issues (just mainly feature or bug issues). I ran 6.4.2 for 9 months and recently upgraded to 6.6.1
6.4.2 was the best experience I had after 6.4.0 debacle. I only upgraded to 6.6.1 so I could use kernel drivers for TV stick, otherwise would have stayed on 6.4.2.. which more accurately measured temps.
I have had no stability issues with 6.6.1 so far. 4gb ram is in mine and 2x1.5TB+4x3TB in dual redundancy.
Without being an expert on the things, power supplies are notorious for causing instability when the start to fail... I have had to change psu's in a couple PCs because of this. Or perhaps try 6.4.2 (downgrading not supported, but possible).
Re: OS 6.6 on Ultra 6:. Not ready for prime time? Do I need more memory? Or...?
Thanks very much to all who replied! It's good to know that OS 6 can run well on Ultra 6s, even if that hasn't been my experience.
Logs were sent to mdgm, but there weren't any apparent smoking guns as I never heard anything back.
I kept getting hangups where various services stopped responding, typically within a few hours of a reboot.
Several times, the "reboot -f" required to restart the system resulted in the message "Remove inactive volumes to use the disk. Disk #1, 2, 3, 4, 5, 6." and all the drives on the Volumes page were shown in red. Rebooting again from the web UI restored normal ops, but the overall experience was not encouraging.
Finally, a hangup occurred such that even ssh access was lost. A back-panel power-switch reboot was done, resulting in the "Remove inactive volumes" state; a subsequent reboot resulted in two inactive volumes: data, and data-0, which further rebooting didn't resolve. Oh, dear!
I could have continued to collect data in an attempt to troubleshoot further, but I had real work to do and NAS-nannying was taking up too much of my time, so four days ago I rolled back to OS 4.2, restored my saved configuration, and restarted my backups. The required resyncs and volume expansions ran under six hour each (not the 180-some hours that OS 6 predicted, but never had a chance to run to completion); the full backup of 8.8 TB of data ran in about two days, and the box is rock-solid stable again.
Might more memory have made the difference? The existing 1 GB memory tested fine, and that amount has worked well in both my Ultra 6s for years. I re-checked memory utilization on both boxes running OS 4 when running large backup jobs and both use about as much swap under OS 4 as the test box did under OS 6, so it's unlikely to be memory exhaustion and swapping per se that's the issue, unless OS6 has introduced new bugs in that regard.
Why my test box failed to behave well on OS 6 while others work fine remains a mystery. But I need my NASes to Just Work, and on OS 4 they do Just Work. Yes, I'll have to do a Factory Reset when I hit certain expansion limits, but I'll live with that inconvenience in return for the rock-solid reliability I've come to expect from OS 4 (not only on my two Ultra 6s, but on the two Pro 6s I set up at my former place of work). "If it ain't broke, don't fix it" turned out to be the right answer in my case.
The good news is that the rollback from OS 6 to OS 4 went flawlessly and my backup NAS is restored to its role as a solid, stable server. It's pretty cool that this box could be updated to an unsupported OS, and even cooler that it could be rolled back, none the worse for wear: mad props are due to mdgm, AMRivlin, and others involved in making these upgrades/rollbacks possible.
Thanks again, all!
Model: ReadyNASRNDU6000|ReadyNAS Ultra 6 Chassis only
Re: OS 6.6 on Ultra 6:. Not ready for prime time? Do I need more memory? Or...?
Nothing stood out to me looking at your logs. Of course the Ultra 6 does have half the memory that we shipped with the OS6 x86_64 unit with the least memory.
Re: OS 6.6 on Ultra 6:. Not ready for prime time? Do I need more memory? Or...?
More memory might be the solution; indeed, it's the one consistent thing emerging as a difference between the people running OS 6 successfully on Ultras and myself, who was not so successful.
Sadly I wasn't able to find any information during my test period to indicate that more memory was useful other than for performance reasons: no one said, "you do need more memory to run OS 6", nor could I find any posts that indicated that someone had actually done the experiment of running OS 6 on both 1 GB and on 2+ GB, and seen a difference. So, rather than easter-egg it further—I needed to get the NAS back into service so I could expand my other one—I went back to what I knew worked.
Now I'm getting close to running into the 8 TB expansion limit on my main NAS, so I may swap them in the near future once my schedule allows for a Factory Reset, and try the experiment again with more RAM. If so I'll report back.
Re: OS 6.6 on Ultra 6:. Not ready for prime time? Do I need more memory? Or...?
Final update (I hope): I re-ran the upgrade from 4.2 to 6.6.1. The NAS has been working perfectly for over a week. What was different this time? (a) all disk were in place already, so no RAID expansion was needed; and (b) I went straight to 6.6.1, not 6.6.0.
I did order more memory, but haven't installed it; I'm still running the stock 1 GB. I wanted to replicate my previous issues first so I could verify whether increasing RAM alone would fix it. But the box has resolutely failed to show any issues at all: no hangups, no unexpected slowness, no failures to respond. I've tried things like running a scrub while launching backups and hitting the NAS with clients, and the system reacted more slowly when all that was going on, but even so: it all worked, and the scrub of 8.8 TB finished within 48 hours (as did the initial full backup, before I started the stress test). Incremental backups run as quickly now as they did on OS 4.
So I either had a subtle corruption during the initial upgrade, or stuffing two new disks in and expanding the volume triggered the problems. But it wasn't low RAM, apparently, and OS 6 seems to be running fine despite my best attempts to break it. I'm going to label it "bug report closed: cannot replicate."
As for the apparent mislabeling of the System and CPU temperatures? I edited /etc/frontview/sensors/ULTRA6.conf and swapped the "System" and "CPU" labels for temp2 and temp3, ran "systemctl restart readynasd", and now the readings in the web UI match the values I expected from version 4.2 (I got the idea from this thread: <https://community.netgear.com/t5/Using-your-ReadyNAS/ReadyNAS-4200V2-OS6-6-1-fan-control/m-p/1219655...
