Dead volume after disc replacement

[JFMM]

Hello,

I'm using ReadyNAS 2100 v1. Firmware: RAIDiator 4.2.30. RAID level X-RAID2, 4 disks of 3Tb. 

Disk 1 was dead, and disk 4 had increasing SMART errors.

Before replacing disk 1:

Disk 1    -- 2794 GB, Dead   
Disk 2    -- 2794 GB, Ok    
Disk 3    -- 2794 GB, Ok
Disk 4    -- 2794 GB, Ok

Logs also showed SMART errors on disk 4. I replaced disk 1.

After replacing disk 1:

Volume Status: Dead. This volume is no longer available.
Disk 1    -- 2794 GB, Spare    
Disk 2    -- 2794 GB, Ok    
Disk 3    -- 2794 GB, Ok
Disk 4    -- 2794 GB, Dead

Last log entries are:
- RAID sync started on volume C. (it started five days ago and hasn't finished)
- Data volume will be rebuilt with disk 1.

Kernel.log and system.log shows continously:

...

Mar 17 19:12:32 NAS-xxx kernel: EXT4-fs (dm-0): previous I/O error to superblock detected
Mar 17 19:12:32 NAS-xxx kernel: Buffer I/O error on device dm-0, logical block 0
Mar 17 19:12:32 NAS-xxx kernel: lost page write due to I/O error on dm-0
Mar 17 19:12:32 NAS-xxx kernel: EXT4-fs error (device dm-0): __ext4_get_inode_loc:4758: inode #80347898: block 1285554255: comm proftpd: unable to read itable

...

I suppose if 2 disks have failed simultaneously, data might have lost. But:

1.- Is there any possibility of recovering any data?
2.- Should I replace Disk 4 before doing anything else?

3.- Should I perform a factory reset?

Any advice appreciated.

Thank you very much.

[jak0lantash · Accepted solution]

Looks like a dual disk failure indeed. When you replace a disk, the NAS has to rebuild it, that is a stressful operation for the other disks. If the disks are healthy, then there shouldn't be any problem. If you have a dodgy disk, then it becomes a risky situation.

A good way to avoid running into this is (only applicable if the disk is dying but still in sync with the RAID array - not dead):

1. Shutdown the NAS.

2. Pull the first disk you want to replace.

3. Clone the disk.

4. Insert the cloned disk in place of the dead one.

5. Boot the NAS.

6. Check that everything is fine.

In your earlier situation, when you know two disks are dying, the best approach I can think of would be:

1. Shutdown the NAS.

2. Pull the first disk you want to replace.

3. Clone the disk.

4. Insert the cloned disk in place of the dead one.

5. Pull the second disk you want to replace.

6. Clone the disk.

7. Re-insert the second disk you wanted to replace back in the chassis, not the clone, the original one.

8. Boot the NAS.

9. Check that everything is fine. If so, continue.

10. Shutdown the NAS.

11. Replace the second disk with its clone.

12. Boot the NAS.

13. Check that everything is fine.

That's what I would do, but it's open for debate 😉

Anyway, you're past this situation.

What you should do is try clone the disk 4 as it's the dead disk that it's the most ouf of sync.

So:

1. Shutdown the NAS.

2. Clone disk 4.

3. Insert the clone of disk 4.

4. Boot the NAS.

5. Hope that the RAID will start.

If the RAID doesn't start, there may be a need to force it, but I can't advise you to do that.

If it does, it will rebuild disk 1.

You can contact a data recovery company, or even NETGEAR offers this type of service, but it isn't free. They can do more than me (a user on a forum), they can assess the situation, take precautions and confirm the necessary steps according to the exact context.

[jak0lantash · Accepted solution]

Looks like a dual disk failure indeed. When you replace a disk, the NAS has to rebuild it, that is a stressful operation for the other disks. If the disks are healthy, then there shouldn't be any problem. If you have a dodgy disk, then it becomes a risky situation.

A good way to avoid running into this is (only applicable if the disk is dying but still in sync with the RAID array - not dead):

1. Shutdown the NAS.

2. Pull the first disk you want to replace.

3. Clone the disk.

4. Insert the cloned disk in place of the dead one.

5. Boot the NAS.

6. Check that everything is fine.

In your earlier situation, when you know two disks are dying, the best approach I can think of would be:

1. Shutdown the NAS.

2. Pull the first disk you want to replace.

3. Clone the disk.

4. Insert the cloned disk in place of the dead one.

5. Pull the second disk you want to replace.

6. Clone the disk.

7. Re-insert the second disk you wanted to replace back in the chassis, not the clone, the original one.

8. Boot the NAS.

9. Check that everything is fine. If so, continue.

10. Shutdown the NAS.

11. Replace the second disk with its clone.

12. Boot the NAS.

13. Check that everything is fine.

That's what I would do, but it's open for debate 😉

Anyway, you're past this situation.

What you should do is try clone the disk 4 as it's the dead disk that it's the most ouf of sync.

So:

1. Shutdown the NAS.

2. Clone disk 4.

3. Insert the clone of disk 4.

4. Boot the NAS.

5. Hope that the RAID will start.

If the RAID doesn't start, there may be a need to force it, but I can't advise you to do that.

If it does, it will rebuild disk 1.

You can contact a data recovery company, or even NETGEAR offers this type of service, but it isn't free. They can do more than me (a user on a forum), they can assess the situation, take precautions and confirm the necessary steps according to the exact context.

[JFMM]

Thank you for the help and the tips.

I'll follow those steps as soon as I'll get the new disk to replace disk 4.

In the meantime, new disk 1 is now marked as spare, so I could try to use it to clone the replaced dead one, if that is possible. Is that appropiate?
I mean:

1.- Shutdown NAS.

2.- Pull out again the new disk 1 marked as spare, and use it to clone the replaced dead one, if possible.

3.- Insert the cloned disk 1.

4.- Boot the NAS

Thanks

[jak0lantash]

It's completely out of sync so technically yes. You need to keep around the dead disks, just in case.

1. Shutdown the NAS.

2. Remove the new disk 1, label it.

3. Remove the dead disk 4, label it.

4. Clone the dead disk 4 on the previously new disk 1.

5. Insert the clone of disk 4 (so: missing, disk 2, disk 3, clone of disk 4).

6. Boot the NAS.

[JFMM]

I'll try cloning one dead disk.

 - Thank you -