My RN102 appears to be broken after updated to 6.10.4. Not sure which came first, I updated the OS to 6.10.4 and noticed very slow transfer speed when I copied hundreds of JPEG images from the SD card on my PC to the NAS. I know the RN102 is not exctly a speed demon but it was annoying slow, only 25 MB/s as measured by NASTester 1.7. Other posts in the community suggests around 60 to 70 MB/s. The reported negotiated speed is 1 Gps as reported by Windows 10.
To rule out problems with LAN hardware, I connected the PC directly to the NAS with a CAT 6e cable but the speed did not change much. I then ran the same test from a laptop connecting to the LAN via 5GHz WiFi and the reported speed was similar, albeit a bit slower. I concluded the bottle neck is unlikely to be LAN hardware.
I suspect the next step is probably what broke the camel's back. In an attempt to improve read/write speed, I disabled Bit Rot Protection on all Shares and things took a dive from there. The NAS hangs up whenever reading/writing multiple files but single file read/write is fine. When it hangs up, the Admin page is not accessible but I can still ping the NAS. To shut it down requires disconnecting power, it will not respond to the power button shutdown sequence, or it will take so long that I do not bother to wait.
Honeslty I do not know where to go from here. I hate the idea of reinstalling the OS because I will lose all data, as I do not have a back up. The file system appears to be intact and the files are still accessible but they can only be accessed one by one and that is not terribly useful. I have not re-enabled Bit Rot Protection to see what will happen.
I downloaded the log System_log-nas-36-1C-14-20210429-072924.zip this morning but I don't know exactly what to look for. The Log page in Admin looks fine to me; no error, no warning. By the way, the OS was updated on 25/4 6:39pm but the log was not retrieved until 29/4 4:7:29am so there is big chunk of information missing. The only app installed is Transmission.
I downloaded the log System_log-nas-36-1C-14-20210429-072924.zip this morning but I don't know exactly what to look for. The Log page in Admin looks fine to me; no error, no warning. By the way, the OS was updated on 25/4 6:39pm but the log was not retrieved until 29/4 4:7:29am so there is big chunk of information missing. The only app installed is Transmission.
I redacted your download link (the full log zip shouldn't be posted publicly). But I did take a look.
kernel.log has a lot of errors on disk 1 (sda). For instance, this segement
I downloaded the log System_log-nas-36-1C-14-20210429-072924.zip this morning but I don't know exactly what to look for. The Log page in Admin looks fine to me; no error, no warning. By the way, the OS was updated on 25/4 6:39pm but the log was not retrieved until 29/4 4:7:29am so there is big chunk of information missing. The only app installed is Transmission.
I redacted your download link (the full log zip shouldn't be posted publicly). But I did take a look.
kernel.log has a lot of errors on disk 1 (sda). For instance, this segement
I suggest powering down the NAS and testing it in a Windows PC (either connected with SATA or a USB adapter/dock) with Lifeguard.
Alternatively, you could try rebooting the NAS w/o it, (leaving disk 2 in slot 2), and see if you get a more normal speed.
When you replace the disk, avoid the WD20EFAX - it is an SMR drive. A Red Plus drive (another WD20EFRX or WD20EFZX) would be fine.
Do you think the NAS will boot up with Disk 1 taken out? Did you observe similar errors in Disk 2? How likely do you think the problem with the NAS hanging when reading/writing multiple files is related to errors in Disk 1?
What is preferred way of posting a file (other than an image) on the forum?
Will the NAS boot up with disk 1 taken out? Did you observe similar errors in disk 2? How likely is the NAS hanging up when reading/writing multiple files caused by errors in disk 1?
Re HDD dock, do you mean something like this? What is Lifeguard? Can Windows read the files directly?
No, and it won't mount the disk. But Lifeguard will see it, and you can test it.
If you boot the NAS with just disk 2, and then later on want to reinsert disk 1: You will need to resync disk 1, which rebuilds it's contents from scratch. So you might as well also run the destructive "erase disk" (might be called "write zeros") test as well as the non-destructive tests I suggested.
If you want the option to reinsert disk 1 w/o resync, then you need to keep the NAS powered down until you test the disk. But given the errors, I'd just replace the disk.
Please confirm the following action plan is sound with low risk:
Remove Disk 1 - should I remove the disk when the NAS is live or is it safer to remove it when it is powered down?
Power on the NAS with just Disk 2. Run speed test using NASTester and confirm the system is operating properly without hanging up.
Power down the NAS and insert Disk 1 with a new HDD into the bay.
Power on the NAS and let it resync and rebuild Disk 1, assuming this is all automatic.
When resync is complete, run speed test again and confirm the system is operatng properly without hanging up.
Insert the failed disk into a HDD dock and examine it with the WD utility as an optional exercise.
Some HDD docks have two disk slots and can do disk cloning. Is it okay to clone a disk from Disk 2 using the dock or is it better to leave this to the NAS?
How come the disk_info log attached shows no disk errors but the kernel log logged disk errors? Does Channel 0 corresponds to Disk 1?
Lowest risk would be to add a step to back up the data on the NAS (for instance to a USB drive). This could be done before step 1, but it likely will go faster between step 2 and 3.
3. Power down the NAS and insert Disk 1 with a new HDD into the bay.
No need to power down here. Hot-insertion is fine.
Generally I test new disks before I add them to the NAS. What I do is test them with vendor diags in a Windows PC - running the full non-destructive "long" test, followed by a full erase/write zeros test. I've received some new disks that pass one of those tests but fail the other.
Some HDD docks have two disk slots and can do disk cloning. Is it okay to clone a disk from Disk 2 using the dock or is it better to leave this to the NAS?
I wouldn't. If you run the NAS with only disk 2 in place, you still need to do the resync. Cloning is sometimes a necessary part of data recovery - but if there are sectors that can't be read, there will be some garbage on the clone. Since there are no read errors on the new good disk, the NAS has no way to tell which sectors are garbage.
How come the disk_info log attached shows no disk errors but the kernel log logged disk errors? Does Channel 0 corresponds to Disk 1?
That is an excellent question. You'd think it would show up in the SMART stats. But I've seen this before, so I know errors don't always end up in those stats.
SMART stats are kept by the disk itself - the NAS just asks the disk for them. So this is either a limitation in SMART itself, or something not quite right in the disk's statistics collection.
The disk is over 6 years old, so you have gotten your money's worth out of it.
The Volume log has entries for both RAID 0 and RAID 1. What does that mean? My understanding is that RAID 0 offers no redunancy and both disks are required to operate. If disk 1 is removed, does it mean the system will fail to boot with only disk 2?
mdstat.log shows info on the RAID levels more concisely
md0 is the 4GB root volume and uses RAID-1. The system will still boot if a disk is removed however any RAID-0 data volume using that disk won’t be able to be mounted as the RAID array won’t be able to be started.
Based on the logs, do you think the data in disk 1 is recoverable using WD disk utility? Can you tell which Share will be affected by the loss of disk 1?
Based on the logs, do you think the data in disk 1 is recoverable using WD disk utility?
I don't recommend trying that. In my experience such "recovery" is more of a bandaid than a cure. When I've tried it, the disk very quickly starts throwing errors again.
Another aspect is that you are experiencing read errors. The disk utility can't preserve the data if it can't read the disk. So the volume would still need to be resynced, otherwise you won't have a proper mirror.
A replacement drive would cost ~$80 (current USD pricing). IMO attempting a "repair" isn't worth the risk.
But you are jumping ahead. The next step is to test the disk. I suspect it will fail.
Can you tell which Share will be affected by the loss of disk 1?
Your volume is RAID-1 (mirrored disks), so there should be no data loss. All shares will lose redundancy when you remove disk 1 (since there will be no mirror). The volume status will change to "degraded" to reflect that.
But you've actually already lost some redundancy, due to the read errors. The logs show that the system is relying on disk 2 to get your data (the log shows the read for disk 1 failing, and the RAID system failing over to read from disk 2). If disk 2 were to fail, there would be at least some data loss.
The Volume log has entries for both RAID 0 and RAID 1.
FYI, you are misreading the log. Volume.log simply numbers the RAID arrays starting from 0. So that entry isn't telling you the RAID mode - it's just a label. A few lines after that label you should see
Removed disk 1 after powering down the NAS and it boots up fine with just disk 2.
I ran NASTester all Shares. The Share with BitRot protection enabled measured 25MB/s. The remaining Shares with BitRot protection disabled measured 36MB/s on average. Is this speed normal at all? I think it is underpar even with BitRot protection off. If so, what else could be wrong? No problem reading or writing multiple files though. Is BitRot protection worth the sacrifice in speed? I have problem with 25MB/s in accessing photos on my photo management software.
I ran NASTester all Shares. The Share with BitRot protection enabled measured 25MB/s. The remaining Shares with BitRot protection disabled measured 36MB/s on average. Is this speed normal at all?
Both speeds sound low to me. Are you testing over wifi or gigabit ethernet? On gigabit you can get ~70 MB/s read and ~50 MB/s write.
There were some improvements made to the BTRFS setup some years ago. But to take advantage of them you'd need to do a factory reset, rebuild the NAS, and restore all the data from backup. I tested that some time ago, and it did make a significant difference on my RN102.
Is BitRot protection worth the sacrifice in speed?
Up to you really. I do leave it on my main NAS (an RN526x), but I've never had bitrot protection kick in and recover a file. But it's not an RN100 series, and even with bitrot on, it can saturate a fast gigabit connection.
I ran NASTester all Shares. The Share with BitRot protection enabled measured 25MB/s. The remaining Shares with BitRot protection disabled measured 36MB/s on average. Is this speed normal at all?
Both speeds sound low to me. Are you testing over wifi or gigabit ethernet? On gigabit you can get ~70 MB/s read and ~50 MB/s write.
Testing was done over GigE. I have confirmed the LAN hardware is sound before using a direct Cat 6e cable connection (no router and switch) between NAS and PC and verifed with another PC over 5GHz WiFi and the speed in both tests were about the same. It was somewhat lower over WiFi but this was expected. I concluded the common element is the NAS and is the likely culprit. What issues could possibly drag the speed down by that much? Are there any clues in the logs?
A backup of the NAS was run overnight and completed without issues.
I ran WD Dashboard Diganostic Extended Test on disk 1 and it failed with error code 2. Don't know what that means.
I will intall a new disk into the NAS today after checking it with WD Dashboard.
I ran NASTester all Shares. The Share with BitRot protection enabled measured 25MB/s. The remaining Shares with BitRot protection disabled measured 36MB/s on average. Is this speed normal at all?
There were some improvements made to the BTRFS setup some years ago. But to take advantage of them you'd need to do a factory reset, rebuild the NAS, and restore all the data from backup. I tested that some time ago, and it did make a significant difference on my RN102.
When was BTRFS introduced? I had the RN102 for 6 years and I should have checked the speed when I first installed it and defintely would not have accepted it if the speed was so slow back then, which is not a lot better than my old ReadyNAS Duo. BTRFS is unlikely to be the real issue, we are talking about 60% reduction in speed so something fundamental must be wrong.
I also spotted the following entries in the kernel log around the time when I ran the first test with flow control off:
May 01 12:10:43 nas-36-1C-14 kernel: mvneta d0074000.ethernet eth0: bad rx status 0e830000 (overrun error), size=1024 May 01 12:10:43 nas-36-1C-14 kernel: mvneta d0074000.ethernet eth0: bad rx status 0e830000 (overrun error), size=512 May 01 12:10:43 nas-36-1C-14 kernel: mvneta d0074000.ethernet eth0: bad rx status 0e830000 (overrun error), size=128 May 01 12:23:58 nas-36-1C-14 kernel: nr_pdflush_threads exported in /proc is scheduled for removal
What does your initrd.log look like in the logs zip? That will show the firmware update history since the last factory reset of your system.
BTRFS has always been used as the filesystem for the data volume for OS6 and the filesystem for the root volume on the x86 NAS units (the 102 uses EXT4 for the root volume).
Bit-rot protection was added back in 6.2.x
If you ever used bit-rot protection or snapshots, disabling bit-rot protection isn't going to undo all the CoW that has happened. It will just limit what happens going forward.
There are things you could check like CPU usage, whether the memory is using a lot of swap etc. Also turn off services and apps that you don't need.
Over time as updates are released software may consume more resources and the RN100 series is the least powerful systems that run OS6 so any performance hit is going to be most noticeable on those systems.
If you have a bad disk that can impact performance. If you only have the one disk that can also impact performance, and if your NAS is doing a resync of the RAID whilst adding a replacement disk performance will be reduced until that resync completes.
IMO you should be careful to tackle one issue at a time. So wait to address the speed issue until you have a full RAID-1 volume back.
I also suggest making a backup of your files before you add the replacement disk back. The rebuild of the RAID will stress the disk 2 (every sector is read and copied to the replacement to create the mirror). If the disk 2 fails during the process, you will lose your data.
My own RN102 goes back to 2013. By 2016 the read speed had dropped to about 35 MB/sec. This was using jbod volumes (so bitrot protection was off, since it needs RAID redundancy). That speed sounds very similar to yours.
A factory reset restored the read speed to around 70 MB/s. I posted the results here, but it was a long time ago, and search isn't finding them. I did another benchmark in 2019, checking all three versions of SMB. The second test was using different disks, and the NAS was set up for XRAID/RAID-1 - not jbod. SMB 2.1 speeds were still around 70 MB/s, though SMB 3 was somewhat slower (63 MB/sec). https://community.netgear.com/t5/Using-your-ReadyNAS-in-Business/Average-transfer-speed-read-and-wri...
So based on my own experience, a factory reset will almost certainly increase your speeds substantially. Yes, it is painful, but the performance increase is likely worth it.
Other things you can try:
Disable strict sync in the SMB settings
disable IPv6 in your NAS network settings. In some cases the PC will use ipv6 link local addresses for data transfer, and that can be slower (esp if your router doesn't have ipv6 enabled).
Disabling all services you don't actually need (and uninstalling any apps you no longer use). The RN102 doesn't have much memory, and over the years the memory footprint of the firmware has increased. Removing services and apps can help mitigate that.
Run the maintenance functions on the volume settings wheel. Balance and Scrub can both help improve performance. Personally I schedule one maintenance function per month - completing the cycle every four months (balance, defrag, scrub, disk test)
While I get your desire to solve your original speed problem, I still suggest taking things one step at a time. Data safety comes first, performance comes second.
If I were in your position now (waiting for the new disk to be inserted and resync), I'd be making sure I had an up-to-date backup. Once that was taken care of, I'd run the disk test from the volume settings wheel - in order to confirm that disk 2 didn't have any issues. The disks are about the same age, and it isn't unusual to have multiple disks in a RAID array fail in rapid succession.
I also spotted the following entries in the kernel log around the time when I ran the first test with flow control off:
May 01 12:10:43 nas-36-1C-14 kernel: mvneta d0074000.ethernet eth0: bad rx status 0e830000 (overrun error), size=1024 May 01 12:10:43 nas-36-1C-14 kernel: mvneta d0074000.ethernet eth0: bad rx status 0e830000 (overrun error), size=512 May 01 12:10:43 nas-36-1C-14 kernel: mvneta d0074000.ethernet eth0: bad rx status 0e830000 (overrun error), size=128 May 01 12:23:58 nas-36-1C-14 kernel: nr_pdflush_threads exported in /proc is scheduled for removal
RX overuns happen when the NAS isn't keeping up with the transfer requests. So it's not surprising you'd see them with flow control off, since your NAS isn't capable of transferring at full gigabit speed. The NAS uses flow control requests to pause the traffic when the buffering on the requests starts to get full. The router then will hold them (and at some point will pause the traffic from the PC).
Note that no traffic will be lost due to the overruns - the NAS will request retransmission. TCP (network) speeds will drop though, as the network protocols are designed to back off speeds when any loss occurs.
.png){kind=link}
.png){kind=link}