This topic has been marked solved and closed to new posts due to inactivity. We hope you'll join the conversation by posting to an open topic or starting a new one.
After install of 6.10.7 (apparently trouble free) on a RN314 (4*6TD WD drves) that has been in use for years, the gui degrades to the point where no data appears in the admin page.
In the short time after boot where admin page can be read, it shows that we are in resync at about 34.xx% complete. Using ssh and cat /proc/mdstat it shows 5.9% complete and verrrry slowly increments over the past few days. Converting the minutes left to days shows more that a month to complete the resync!
The log, when available, shows increasing disk errors on one of the four drives.
Can I replace the faulty drive with a new one, or must I wait for the resync to complete?
If I can replace the drive, is the proper procedure to hot swap the drive, or should I shutdown, replace the drive and power up?
Given the lack of normal behaviour of the admin gui, I am cautious about attempting to either:
upgrade to 6.10.8
downgrade to 6.10.6
reinstall 6.10.7
top shows no stopped or zombie processes, cpu time is approx 50% idle and 50% wait. No swap usage. The only significant accumulation goes to md126_raid5
I'd appreciate advice from anyone more knowledgeable than me as to what steps to take.
So should I hot swap sdb (drive 2 as 1,2,3,4) or is it sbd (drive 2 as in 0,1,2,3) The output from mdstat infers drive numbers are 0,1,2,3
And are these numbers from left to right as looking from the front??
One caution is that the error in the screen shot is a read error - normally if mdadm is rebuilding a disk, it is writing to it. So there still is some uncertainty on exactly what is going on.
sdb is normally the second disk from the left (the disk in the first slot normally is sda).
If you aren't certain, you can also get the serial number with smartctl -x /dev/sdb (along with a lot of other information). Then you could power down the NAS, check the disk serial, and boot up the NAS read-only without it. If the volume is there, but degraded, then you could reboot again normally. Then do a hot-insert of the replacement.
Instructions on booting the NAS read-only are on pages 74-75 here:
If the drive that is resyncing is the faulty one, then you don't need to wait.
But if it is a different drive, then you do.
I am not sure which drive is resyncing. I will try to copy the output of /proc/mdstat to show you what I know. It shows md126 with the activity, but no indication that I can see of drive 1 or 2 or 3 or 4
You are correct that those are RAID groups, composed of partitions on the physical drives. The commands you issued show what partitions are in each RAID group and what "RAID device" is assigned to each.
If you also look at the results of cat /proc/mdstat, you'll see a list of the current members of the RAID and a string that indicates what's missing. For example, this is mine for md126:
Do you see an underscore in place of one of the U's, indicating the "missing" drive and one of the drives from the mdadm --detail command not showing up in the mdstat list?
volume_util -s raid and/or volume_util -s disk will help you match each md device with it's physical location. Note that channel labels start with 0, so the drive in channel 0 is in bay 1.
If that's not the case, then it must never have dropped a partition completely from the array, which may be confusing the issue.
Thanks for the reply. There are NO "_" in the lists for all four (md126, md127,md0,md1) all show four "U"s.
I have a new 6Tb drive to replace when the resync completes. But the resync for md126 is still just over 7% after 3 days, according to cat /proc/mdstat . A sample of the output is included in previous attachments.
In this attachment I show the only significant differences in the output of four mdadm --detail commands, one for each of md0,md1,md126,md127
I have a new 6Tb drive to replace when the resync completes. But the resync for md126 is still just over 7% after 3 days, according to cat /proc/mdstat . A sample of the output is included in previous attachments.
The puzzle is figuring out exactly what is being rebuilt. There are some operations (for instance a RAID scrub) that look like a resync, but aren't rebuilding any drive.
But if you remove the wrong drive during an actual resync, then you will lose all the data.
Have you been able to download the log zip file? There could be errors in there that would be helpful.
You can also try looking at the log from ssh using journalctl -r
So should I hot swap sdb (drive 2 as 1,2,3,4) or is it sbd (drive 2 as in 0,1,2,3) The output from mdstat infers drive numbers are 0,1,2,3
And are these numbers from left to right as looking from the front??
One caution is that the error in the screen shot is a read error - normally if mdadm is rebuilding a disk, it is writing to it. So there still is some uncertainty on exactly what is going on.
sdb is normally the second disk from the left (the disk in the first slot normally is sda).
If you aren't certain, you can also get the serial number with smartctl -x /dev/sdb (along with a lot of other information). Then you could power down the NAS, check the disk serial, and boot up the NAS read-only without it. If the volume is there, but degraded, then you could reboot again normally. Then do a hot-insert of the replacement.
Instructions on booting the NAS read-only are on pages 74-75 here:
Yes the journalctl output seems to be read errors only.
The smartctl output from sda, sdc, and sdd are "snappy" as in they finish momentarily after pressing enter.
The smartctl output from sdb takes minutes to finish and does indicate a large number of read errors. The serial number as reported by smartctl matches the serial number in the error log.
Re " If the volume is there, but degraded, then you could reboot again normally. " Since my access to the gui is unreliable, is
Re " If the volume is there, but degraded, then you could reboot again normally. " Since my access to the gui is unreliable, is
cat /proc/mdstat
a good place to verify "the volume is there'?
The web ui should respond normally after you boot up read-only, as it is pretty clear that the performance issues are related to the disk errors.
So one way is to look at the volume page in the web ui when you boot up read-only.
If the volume is marked as degraded, you can then make sure the data is accessible from a PC. If it is, just reboot normally
If the volume is marked as "inactive" or dead, then the system is trying to rebuild another disk. In that situation, you need some form of data recovery. I don't think that's likely, as the commands we've already run normally will show you what disk is rebuilding. While this scenario is possible, I don't think it makes sense to wait for the resync to complete.
You could also check with ssh:
cat /proc/mdstat will tell you the status of the raid groups
Examining the data volume will let you know that the data volume is mounted and that the data is accessible - for instance ls /data should show you the shares (if you are using data as the volume name). Similarly, ls /data/sharename should show the files and folders in the share. If ls /data doesn't show the shares, then the volume failed to mount.
StephenB: I will proceed with the steps you outlined.
Sandshark: output of btrfs command
root@CTS-NG314:/proc# btrfs scrub status /data scrub status for 9f66362a-5347-4ccb-9950-77ab32406926 no stats available total bytes scrubbed: 0.00B with 0 errors root@CTS-NG314:/proc#
The replacement of the drive and rebuild completed successfully around 03:30 am, and elapsed time of 15 hours. Reliable access to the gui right after drive replacement.
In summary, perusing the log, showed drive had been reporting errors since July, but email notifications were not working so I missed them!
My inference in the subject line that the upgrade to 6.10.7 was implicated was totally off base!
The replacement of the drive and rebuild completed successfully around 03:30 am, and elapsed time of 15 hours. Reliable access to the gui right after drive replacement.
Thx for following up, and I'm glad the problem is resolved.
