Start a New Discussion

Discussion stats
  • 5 replies
  • ‎2016-11-30 11:59 PM
  • 3281 views
  • 1 kudo
  • 2 in conversation
Announcements

Top Contributors
See All
Reply
Highlighted
ckabo
Aspirant
Aspirant
‎2016-11-30 11:59 PM
‎2016-11-30 11:59 PM

Re-adding disk to volume

I seem to have gotten myself in some trouble...

 

I've got a RN104 with 4x3TB disks in it. It's running RAID-5 and encryption. v6.6.0.

 

I got emails saying disk #4 was getting errors and that I should replace it. So I bought a new disk and replaced disk #4. It started resyncing (just 44 hours to go...). But just a few hours later I started getting emails that disk #1 was getting errors and should be replaced. An hour after that I got an email that disk #1 had gone offline. Doh.

 

Disk #1 seems to be completely broken, but as #4 hadn't died yet I'm currently running ddrescue on disk #4, copying it to the new disk I bought. My reasoning is that as disk #2 and disk #3 are both fine, if I can get a good copy of #4 I should have 3 good disks and be able to have a degraded volume. From there I can add a new disk in slot #1, have the volume resync and be back in business.

 

Disk #2 and disk #3 are quite new (much newer than #1) so they shouldn't break from resyncing.

 

Should this approach work? While the volume was resyncing I may have added 1 or 2 small files, and I'm guessing a couple of files may have been modified as well. So disk #4 is probably a little behind #2 and #3. Will that be OK? mdadm indicates that if the event count is fairly close one can --re-add devices that were previously removed from the array, but I don't know if mdadm is used in ReadyNAS.

 

Should I just stick in the new copy of #4 and leave #1 empty and hope that my NAS boots up and reports a degraged volume, at which point I can stick a new disk in #1 and it will resync? What are my options?

 

Yes, I had a backup job on the important folders. Turns out that backup job was last run in 2015... Smiley Sad
Should have put some sort of monitoring on that backup job. But that's too late now, and it's not the thought that counts.

Model: RN104|ReadyNAS 100 Series
Message 1 of 6
0 Kudos
Reply
Highlighted
StephenB
Guru Guru
Guru
‎2016-12-01 07:10 AM
‎2016-12-01 07:10 AM

Re: Re-adding disk to volume

@ckabo wrote:

 

...Disk #1 seems to be completely broken, but as #4 hadn't died yet I'm currently running ddrescue on disk #4, copying it to the new disk I bought. My reasoning is that as disk #2 and disk #3 are both fine, if I can get a good copy of #4 I should have 3 good disks and be able to have a degraded volume. From there I can add a new disk in slot #1, have the volume resync and be back in business.

 

...Should this approach work? While the volume was resyncing I may have added 1 or 2 small files, and I'm guessing a couple of files may have been modified as well. So disk #4 is probably a little behind #2 and #3. Will that be OK? mdadm indicates that if the event count is fairly close one can --re-add devices that were previously removed from the array ,,,.

It's probably your only option short of data recovery.  The changes might be a problem - not so much the files themselves, but the metadata will be inconsistent. You might want to mount the volume read-only and copy off all the data you can.  Leaving slot 1 empty makes sense to me, since you certainly wouldn't want the system to attempt another resync.

 

@ckabo wrote:

... but I don't know if mdadm is used in ReadyNAS.

 

It is.

Message 2 of 6
Reply
Highlighted
ckabo
Aspirant
Aspirant
‎2016-12-02 12:50 PM
‎2016-12-02 12:50 PM

Re: Re-adding disk to volume

Is LUKS used for encryption? And how is it used, do I first run mdadm to get the raid online and then run LUKS, or do I first unlock with LUKS and then run mdadm? Are there some lvm commands I need to run in there as well?

 

Comments and links on this would be appreciated Smiley Happy

Message 3 of 6
0 Kudos
Reply
Highlighted
ckabo
Aspirant
Aspirant
‎2016-12-04 11:26 AM
‎2016-12-04 11:26 AM

Re: Re-adding disk to volume

Status:

After cloning disk #4 I popped the clone in, started the NAS and logged in with SSH.

root@MyNAS:~# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md1 : active raid1 sdc2[2] sdb2[1] sda2[0]
      523712 blocks super 1.2 [3/3] [UUU]

md0 : active raid1 sda1[6] sdc1[5] sdb1[4]
      4190208 blocks super 1.2 [4/3] [UU_U]

unused devices: <none>

root@MyNAS:/var/log/readynasd# mdadm --assemble --scan -v
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/md1
mdadm: no recogniseable superblock on /dev/md/0
mdadm: /dev/sdc2 is busy - skipping
mdadm: /dev/sdc1 is busy - skipping
mdadm: No super block found on /dev/sdc (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc
mdadm: /dev/sdb2 is busy - skipping
mdadm: /dev/sdb1 is busy - skipping
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb
mdadm: /dev/sda2 is busy - skipping
mdadm: /dev/sda1 is busy - skipping
mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda
mdadm: No super block found on /dev/mtdblock4 (Expected magic a92b4efc, got 06101831)
mdadm: no RAID superblock on /dev/mtdblock4
mdadm: No super block found on /dev/mtdblock3 (Expected magic a92b4efc, got 5726f12c)
mdadm: no RAID superblock on /dev/mtdblock3
mdadm: No super block found on /dev/mtdblock2 (Expected magic a92b4efc, got e580c000)
mdadm: no RAID superblock on /dev/mtdblock2
mdadm: No super block found on /dev/mtdblock1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/mtdblock1
mdadm: No super block found on /dev/mtdblock0 (Expected magic a92b4efc, got 2b023063)
mdadm: no RAID superblock on /dev/mtdblock0
mdadm: /dev/sdc3 is identified as a member of /dev/md/data-0, slot 1.
mdadm: /dev/sdb3 is identified as a member of /dev/md/data-0, slot 2.
mdadm: /dev/sda3 is identified as a member of /dev/md/data-0, slot 3.
mdadm: no uptodate device for slot 0 of /dev/md/data-0
mdadm: added /dev/sdb3 to /dev/md/data-0 as 2
mdadm: added /dev/sda3 to /dev/md/data-0 as 3 (possibly out of date)
mdadm: added /dev/sdc3 to /dev/md/data-0 as 1
mdadm: /dev/md/data-0 assembled from 2 drives - not enough to start the array.
mdadm: looking for devices for further assembly
mdadm: /dev/sdc2 is busy - skipping
mdadm: /dev/sdc1 is busy - skipping
mdadm: /dev/sdb2 is busy - skipping
mdadm: /dev/sdb1 is busy - skipping
mdadm: /dev/sda2 is busy - skipping
mdadm: /dev/sda1 is busy - skipping
mdadm: No arrays found in config file or automatically








root@MyNAS:/var/log/readynasd# mdadm --examine /dev/sd[a-g]3
/dev/sda3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b1340f07:7ebeb430:31028483:bfd6dd5b
           Name : 0e35246e:data-0  (local to host 0e35246e)
  Creation Time : Sat Dec 28 10:22:45 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 5850833742 (2789.89 GiB 2995.63 GB)
     Array Size : 8776250112 (8369.68 GiB 8986.88 GB)
  Used Dev Size : 5850833408 (2789.89 GiB 2995.63 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=334 sectors
          State : clean
    Device UUID : a063c204:8a5493bb:5193b40d:d7ac0533

    Update Time : Wed Nov 30 19:47:35 2016
       Checksum : 349c514c - correct
         Events : 25926

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b1340f07:7ebeb430:31028483:bfd6dd5b
           Name : 0e35246e:data-0  (local to host 0e35246e)
  Creation Time : Sat Dec 28 10:22:45 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 5850833742 (2789.89 GiB 2995.63 GB)
     Array Size : 8776250112 (8369.68 GiB 8986.88 GB)
  Used Dev Size : 5850833408 (2789.89 GiB 2995.63 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=334 sectors
          State : clean
    Device UUID : 75ea7b81:9c6e1b73:636dd3bc:bfddbcc4

    Update Time : Thu Dec  1 08:54:34 2016
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : a9b8beea - correct
         Events : 28678

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 2
   Array State : .AA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b1340f07:7ebeb430:31028483:bfd6dd5b
           Name : 0e35246e:data-0  (local to host 0e35246e)
  Creation Time : Sat Dec 28 10:22:45 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 5850833742 (2789.89 GiB 2995.63 GB)
     Array Size : 8776250112 (8369.68 GiB 8986.88 GB)
  Used Dev Size : 5850833408 (2789.89 GiB 2995.63 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=334 sectors
          State : clean
    Device UUID : 3d749b6f:9e210956:aac73191:571ded87

    Update Time : Thu Dec  1 08:54:34 2016
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 12549594 - correct
         Events : 28678

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 1
   Array State : .AA. ('A' == active, '.' == missing, 'R' == replacing)




root@MyNAS:/var/log/readynasd# mdadm --assemble --force /dev/md/data-0 /dev/sdc3 /dev/sdb3 /dev/sda3 -v
mdadm: looking for devices for /dev/md/data-0
mdadm: /dev/sdc3 is identified as a member of /dev/md/data-0, slot 1.
mdadm: /dev/sdb3 is identified as a member of /dev/md/data-0, slot 2.
mdadm: /dev/sda3 is identified as a member of /dev/md/data-0, slot 3.
mdadm: NOT forcing event count in /dev/sda3(3) from 25926 up to 28678
mdadm: You can use --really-force to do that (DANGEROUS)
mdadm: no uptodate device for slot 0 of /dev/md/data-0
mdadm: added /dev/sdb3 to /dev/md/data-0 as 2
mdadm: added /dev/sda3 to /dev/md/data-0 as 3 (possibly out of date)
mdadm: added /dev/sdc3 to /dev/md/data-0 as 1
mdadm: /dev/md/data-0 assembled from 2 drives - not enough to start the array.
root@MyNAS:/var/log/readynasd# mdadm --assemble --really-force /dev/md/data-0 /dev/sdc3 /dev/sdb3 /dev/sda3 -v
mdadm: looking for devices for /dev/md/data-0
mdadm: /dev/sdc3 is identified as a member of /dev/md/data-0, slot 1.
mdadm: /dev/sdb3 is identified as a member of /dev/md/data-0, slot 2.
mdadm: /dev/sda3 is identified as a member of /dev/md/data-0, slot 3.
mdadm: forcing event count in /dev/sda3(3) from 25926 upto 28678
mdadm: clearing FAULTY flag for device 2 in /dev/md/data-0 for /dev/sda3
mdadm: Marking array /dev/md/data-0 as 'clean'
mdadm: no uptodate device for slot 0 of /dev/md/data-0
mdadm: added /dev/sdb3 to /dev/md/data-0 as 2
mdadm: added /dev/sda3 to /dev/md/data-0 as 3
mdadm: added /dev/sdc3 to /dev/md/data-0 as 1
mdadm: /dev/md/data-0 assembled from 3 drives - not enough to start the array.
root@MyNAS:/var/log/readynasd# mdadm --assemble --really-force --run /dev/md/data-0 /dev/sdc3 /dev/sdb3 /dev/sda3 -v
mdadm: looking for devices for /dev/md/data-0
mdadm: /dev/sdc3 is identified as a member of /dev/md/data-0, slot 1.
mdadm: /dev/sdb3 is identified as a member of /dev/md/data-0, slot 2.
mdadm: /dev/sda3 is identified as a member of /dev/md/data-0, slot 3.
mdadm: no uptodate device for slot 0 of /dev/md/data-0
mdadm: added /dev/sdb3 to /dev/md/data-0 as 2
mdadm: added /dev/sda3 to /dev/md/data-0 as 3
mdadm: added /dev/sdc3 to /dev/md/data-0 as 1
mdadm: /dev/md/data-0 has been started with 3 drives (out of 4).

root@MyNAS:/var/log/readynasd# mdadm --examine /dev/sd[a-g]3
/dev/sda3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b1340f07:7ebeb430:31028483:bfd6dd5b
           Name : 0e35246e:data-0  (local to host 0e35246e)
  Creation Time : Sat Dec 28 10:22:45 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 5850833742 (2789.89 GiB 2995.63 GB)
     Array Size : 8776250112 (8369.68 GiB 8986.88 GB)
  Used Dev Size : 5850833408 (2789.89 GiB 2995.63 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=334 sectors
          State : clean
    Device UUID : a063c204:8a5493bb:5193b40d:d7ac0533

    Update Time : Wed Nov 30 19:47:35 2016
       Checksum : 349c5c0c - correct
         Events : 28678

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b1340f07:7ebeb430:31028483:bfd6dd5b
           Name : 0e35246e:data-0  (local to host 0e35246e)
  Creation Time : Sat Dec 28 10:22:45 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 5850833742 (2789.89 GiB 2995.63 GB)
     Array Size : 8776250112 (8369.68 GiB 8986.88 GB)
  Used Dev Size : 5850833408 (2789.89 GiB 2995.63 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=334 sectors
          State : clean
    Device UUID : 75ea7b81:9c6e1b73:636dd3bc:bfddbcc4

    Update Time : Thu Dec  1 08:54:34 2016
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : a9b8beea - correct
         Events : 28678

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 2
   Array State : .AA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b1340f07:7ebeb430:31028483:bfd6dd5b
           Name : 0e35246e:data-0  (local to host 0e35246e)
  Creation Time : Sat Dec 28 10:22:45 2013
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 5850833742 (2789.89 GiB 2995.63 GB)
     Array Size : 8776250112 (8369.68 GiB 8986.88 GB)
  Used Dev Size : 5850833408 (2789.89 GiB 2995.63 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=334 sectors
          State : clean
    Device UUID : 3d749b6f:9e210956:aac73191:571ded87

    Update Time : Thu Dec  1 08:54:34 2016
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 12599593 - correct
         Events : 28678

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 1
   Array State : .AAA ('A' == active, '.' == missing, 'R' == replacing)

root@MyNAS:/var/log/readynasd# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md127 : active raid5 sdc3[5] sda3[3] sdb3[4]
      8776250112 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [_UUU]

md1 : active raid1 sdc2[2] sdb2[1] sda2[0]
      523712 blocks super 1.2 [3/3] [UUU]

md0 : active raid1 sda1[6] sdc1[5] sdb1[4]
      4190208 blocks super 1.2 [4/3] [UU_U]

unused devices: <none>

So I think I managed to get the array up and running again... However in the ReadyNAS web interface the disks still show up as red. Should I run cryptsetup luksOpen or something like that? Or should I just try restarting the NAS?

Message 4 of 6
0 Kudos
Reply
Highlighted
ckabo
Aspirant
Aspirant
‎2016-12-04 11:41 AM
‎2016-12-04 11:41 AM

Re: Re-adding disk to volume

Ah, it seems cryptsetup was run automatically. /dev/mapper/data-0 points to /dev/dm-0.

 

root@MyNAS:~# mount /dev/dm-0 /data
mount: wrong fs type, bad option, bad superblock on /dev/mapper/data-0,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
root@MyNAS:~# dmesg | tail
[ 3169.231414] UBIFS (ubi0:0): un-mount UBI device 0
[ 3169.474448] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 3169.474888] NFSD: starting 90-second grace period (net c097fc40)
[ 3173.890476] BTRFS: device label 0e35246e:data devid 1 transid 558760 /dev/dm-0
[ 3174.259285] BTRFS (device dm-0): parent transid verify failed on 2474687627264 wanted 558760 found 558769
[ 3174.266676] BTRFS (device dm-0): parent transid verify failed on 2474687627264 wanted 558760 found 558769
[ 3174.290469] BTRFS: open_ctree failed
[ 3487.554478] BTRFS (device dm-0): parent transid verify failed on 2474687627264 wanted 558760 found 558769
[ 3487.556355] BTRFS (device dm-0): parent transid verify failed on 2474687627264 wanted 558760 found 558769
[ 3487.580507] BTRFS: open_ctree failed

So it seems that BTRFS is f*cked. I'll try to google for solutions later, see if I can save it.

Message 5 of 6
0 Kudos
Reply
Highlighted
ckabo
Aspirant
Aspirant
‎2016-12-05 01:57 AM
‎2016-12-05 01:57 AM

Re: Re-adding disk to volume

The BTRFS file system was damaged to such an extent that I couldn't mount it. But I'm able to use btrfs restore to get my stuff out. Mailing list thread here.

Message 6 of 6
0 Kudos
Reply