Re-adding disk to volume

Question

I seem to have gotten myself in some trouble...

I've got a RN104 with 4x3TB disks in it. It's running RAID-5 and encryption. v6.6.0.

I got emails saying disk #4 was getting errors and that I should replace it. So I bought a new disk and replaced disk #4. It started resyncing (just 44 hours to go...). But just a few hours later I started getting emails that disk #1 was getting errors and should be replaced. An hour after that I got an email that disk #1 had gone offline. Doh.

Disk #1 seems to be completely broken, but as #4 hadn't died yet I'm currently running ddrescue on disk #4, copying it to the new disk I bought. My reasoning is that as disk #2 and disk #3 are both fine, if I can get a good copy of #4 I should have 3 good disks and be able to have a degraded volume. From there I can add a new disk in slot #1, have the volume resync and be back in business.

Disk #2 and disk #3 are quite new (much newer than #1) so they shouldn't break from resyncing.

Should this approach work? While the volume was resyncing I may have added 1 or 2 small files, and I'm guessing a couple of files may have been modified as well. So disk #4 is probably a little behind #2 and #3. Will that be OK? mdadm indicates that if the event count is fairly close one can --re-add devices that were previously removed from the array, but I don't know if mdadm is used in ReadyNAS.

Should I just stick in the new copy of #4 and leave #1 empty and hope that my NAS boots up and reports a degraged volume, at which point I can stick a new disk in #1 and it will resync? What are my options?

Yes, I had a backup job on the important folders. Turns out that backup job was last run in 2015... :(
Should have put some sort of monitoring on that backup job. But that's too late now, and it's not the thought that counts.

StephenB · Answer

ckabo wrote:
&nbsp;
...Disk #1 seems to be completely broken, but as #4 hadn't died yet I'm currently running ddrescue on disk #4, copying it to the new disk I bought. My reasoning is that as disk #2 and disk #3 are both fine, if I can get a good copy of #4 I should have 3 good disks and be able to have a degraded volume. From there I can add a new disk in slot #1, have the volume resync and be back in business.
&nbsp;
...Should this approach work? While the volume was resyncing I may have added 1 or 2 small files, and I'm guessing a couple of files may have been modified as well. So disk #4 is probably a little behind #2 and #3. Will that be OK? mdadm indicates that if the event count is fairly close one can --re-add devices that were previously removed from the array ,,,.

It's probably your only option short of data recovery. &nbsp;The changes might be a problem - not so much the files themselves, but the metadata will be inconsistent. You might want to mount the volume read-only and copy off all the data you can. &nbsp;Leaving slot 1 empty makes sense to me, since you certainly wouldn't want the system to attempt another resync.
&nbsp;
ckabo wrote:
... but I don't know if mdadm is used in ReadyNAS.
&nbsp;

It is.

ckabo · Answer

Is LUKS used for encryption? And how is it used, do I first run mdadm to get the raid online and then run LUKS, or do I first unlock with LUKS and then run mdadm? Are there some lvm commands I need to run in there as well?

Comments and links on this would be appreciated :)

ckabo · Answer

Status:After cloning disk #4 I popped the clone in, started the NAS and logged in with SSH.root@MyNAS:~# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md1 : active raid1 sdc2[2] sdb2[1] sda2[0]
      523712 blocks super 1.2 [3/3] [UUU]

md0 : active raid1 sda1[6] sdc1[5] sdb1[4]
      4190208 blocks super 1.2 [4/3] [UU_U]

unused devices: &lt;none&gt;

root@MyNAS:/var/log/readynasd# mdadm --assemble --scan -v
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/md1
mdadm: no recogniseable superblock on /dev/md/0
mdadm: /dev/sdc2 is busy - skipping
mdadm: /dev/sdc1 is busy - skipping
mdadm: No super block found on /dev/sdc (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc
mdadm: /dev/sdb2 is busy - skipping
mdadm: /dev/sdb1 is busy - skipping
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb
mdadm: /dev/sda2 is busy - skipping
mdadm: /dev/sda1 is busy - skipping
mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda
mdadm: No super block found on /dev/mtdblock4 (Expected magic a92b4efc, got 06101831)
mdadm: no RAID superblock on /dev/mtdblock4
mdadm: No super block found on /dev/mtdblock3 (Expected magic a92b4efc, got 5726f12c)
mdadm: no RAID superblock on /dev/mtdblock3
mdadm: No super block found on /dev/mtdblock2 (Expected magic a92b4efc, got e580c000)
mdadm: no RAID superblock on /dev/mtdblock2
mdadm: No super block found on /dev/mtdblock1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/mtdblock1
mdadm: No super block found on /dev/mtdblock0 (Expected magic a92b4efc, got 2b023063)
mdadm: no RAID superblock on /dev/mtdblock0
mdadm: /dev/sdc3 is identified as a member of /dev/md/data-0, slot 1.
mdadm: /dev/sdb3 is identified as a member of /dev/md/data-0, slot 2.
mdadm: /dev/sda3 is identified as a member of /dev/md/data-0, slot 3.
mdadm: no uptodate device for slot 0 of /dev/md/data-0
mdadm: added /dev/sdb3 to /dev/md/data-0 as 2
mdadm: added /dev/sda3 to /dev/md/data-0 as 3 (possibly out of date)
mdadm: added /dev/sdc3 to /dev/md/data-0 as 1
mdadm: /dev/md/data-0 assembled from 2 drives - not enough to start the array.
mdadm: looking for devices for further assembly
mdadm: /dev/sdc2 is busy - skipping
mdadm: /dev/sdc1 is busy - skipping
mdadm: /dev/sdb2 is busy - skipping
mdadm: /dev/sdb1 is busy - skipping
mdadm: /dev/sda2 is busy - skipping
mdadm: /dev/sda1 is busy - skipping
mdadm: No arrays found in config file or automatically

root@MyNAS:/var/log/readynasd# mdadm --examine /dev/sd[a-g]3
/dev/sda3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b1340f07:7ebeb430:31028483:bfd6dd5b
           Name : 0e35246e:data-0  (local to host 0e35246e)
  Creation Time : Sat Dec 28 10:22:45 2013
     Raid Level : raid5
   Raid Devices : 4

Avail Dev Size : 5850833742 (2789.89 GiB 2995.63 GB)
     Array Size : 8776250112 (8369.68 GiB 8986.88 GB)
  Used Dev Size : 5850833408 (2789.89 GiB 2995.63 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=334 sectors
          State : clean
    Device UUID : a063c204:8a5493bb:5193b40d:d7ac0533

Update Time : Wed Nov 30 19:47:35 2016
       Checksum : 349c514c - correct
         Events : 25926