× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Replaced drive, but second drive failed during resync (NV+ v2)

bjt23
Aspirant

Replaced drive, but second drive failed during resync (NV+ v2)

Hi all, managed to get myself into a bad situation.  I have a ReadyNAS NV+ v2, with four 2TB drives in it.  I decided to purchase four new 4TB drives to increase my storage space.  I pulled out old drive #1, waited for the notification that the drive was removed, and then inserted one of the new drives.  The new drive was recognized, and resync started.

 

Next morning, I wake up to find a notification that resync finished.  Awesome.  I ssh in and poke around on the file system, but I start seeing I/O errors in some places where there definitely were files before.  I check the kernel log, and find that, while the new disk #1 was syncing, disk #2 failed with some bad sectors.

 

Shame on me: I don't have anywhere near a complete backup, and I didn't do any checks (like a full SMART check, or even a full fsck) before replacing the first disk.  Lesson learned.

 

I started looking around, and tried using 'hdparm --read-sector' to read the bad sectors reported in the kernel log.  I did get a "sense error" while reading, but no I/O errors, and 'hdparm' did dump the contents of those sectors.  I'm also doing a full 'dd if=/dev/sdb of=/dev/null' to run through the whole disk to see if I can get through it without errors.

 

So I'm hoping this was a transient error, and if I could convince the NAS to allow drive #2 back into the array, I could try to resync drive #1 from scratch.  I tried using mdadm to "--remove" and "--re-add" on drive #2, but now it's showing up in the list as a spare (sda is new drive #1 and sdb is old drive #2):

 

 

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       0        0        1      removed
       2       8       35        2      active sync   /dev/sdc3
       3       8       51        3      active sync   /dev/sdd3

       4       8       19        -      spare   /dev/sdb3
       5       8        3        -      spare   /dev/sda3

But nothing seems to be happening, and I'm not sure how to 'activate' the spare drive.  The event counts are close for the three old drives:

 

# mdadm --examine /dev/sd[bcd]3 |grep Event
         Events : 334914
         Events : 334917
         Events : 334917

... so I'm hoping it could maybe just work, if only the kernel would allow it.

 

 

I did find https://www.linuxquestions.org/questions/linux-server-73/mdadm-re-added-disk-treated-as-spare-750739... where the poster suggests stopping the array and re-creating it, but I'm not sure if that's going to confuse the ReadyNAS's software; not sure if it's doing anything special.  Also the following post from someone else said the suggestion I reference did not work (no data after the rebuild)... but then the post _after_ that said the suggestion worked.

 

My other thought is to pull new drive #1, and replace it with original old drive #1, and see if I can get somewhat back to the original state of things, after which I could start by replacing disk #2 with a new disk (with old disk #1 back in place).  But I'm afraid that when I do so, the ReadyNAS will think of it as a brand-new disk and I'll lose the data on it.

 

Any thoughts as to the best course of action?  I know I've certainly gotten myself into this bad situation (and will be more careful with backups and pre-flight checks in the future), but I'm hoping things aren't completely unrecoverable.

 

Model: RND4000v2|ReadyNAS NV+ v2 Chassis only
Message 1 of 3
Sandshark
Sensei

Re: Replaced drive, but second drive failed during resync (NV+ v2)

If you want the best chance of recoverying data, you should stop tinkering and contact Netgear support for paid recovery.

 

I don't have a lot of exp[erience messing around under the hood on OS5.x.  On other OS's, a volume created external to the GUI will be recognized by the GUI if the proper entry is in /etc/fstab.  So, if the recreation works at all, I would think that getting the GUI to recognize it should not be too difficult.  The underlined text being the key, of course.

Message 2 of 3
bjt23
Aspirant

Re: Replaced drive, but second drive failed during resync (NV+ v2)

I ended up getting _really_ lucky.

 

I tried putting back in old drive #1, but unfortunately the system treated it as a new drive and started resync.

 

I was of course assuming that resync would fail part-way through, since drive #2 still had damaged sectors.  So, while drive #1 was resyncing, I did a `dd` of drive #2, ignoring errors, so I could see a list of bad sectors in the kernel log.  I was incredibly lucky: looks like there was only actually one bad sector.  I used `dd` again to force a write of the entire bad sector, which forced the disk to reallocate the sector to spare space.  Yes, that caused me to lose 4k of data, but the resync of old drive #1 finished successfully.

 

I then did a full fsck and set of SMART checks, and then started replacing again, this time starting with drive #2.  After a few days, all four drives were replaced, and the NAS expanded the partition to account for the extra space.

 

So, as I said, I got incredibly lucky.

Message 3 of 3
Top Contributors
Discussion stats
  • 2 replies
  • 560 views
  • 0 kudos
  • 2 in conversation
Announcements