× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Data Volume Lost after Disk Replace #25483643

azerva
Tutor

Data Volume Lost after Disk Replace #25483643

Hello ReadyNAS Community,

 

after a disk failure (disk 3), I replaced this disk. A volume resynchronization was started automatically. However, this resynchronisation failed with no obvious errors and now the entire data volume is unavailable. Can anyone please provide any help as to how to try to rebuid this volume or make it available again?

 

Regards

Alex

Message 1 of 18
JennC
NETGEAR Employee Retired

Re: Data Volume Lost after Disk Replace

Hello azerva,

 

Do you still have access to the admin page of the NAS? Please attach system logs here if you do. Please let us know the model of the NAS you have and how many disks are inserted.

 

Regards,

Message 2 of 18
vandermerwe
Master

Re: Data Volume Lost after Disk Replace

I don't suppose you have a backup of the data on the volume?

Message 3 of 18
azerva
Tutor

Re: Data Volume Lost after Disk Replace

Hi,

 

I still have access via HTTP GUI and SSH. I don't know how good the logs will be, but I will provide then nonetheless. I'm not sure how tho, can't seem to find an attachment button around here. 

Message 4 of 18
azerva
Tutor

Re: Data Volume Lost after Disk Replace


@vandermerwe wrote:

I don't suppose you have a backup of the data on the volume?


Hi,

 

well no. It's quite a lot of data, was around 5 TB (I had 4 x 2 TB drives), so not actually a lot of space to back it up twice. The NAS was supposed to be the "redundant" backup 😉

 

So I'm open to any suggestions.

Message 5 of 18
vandermerwe
Master

Re: Data Volume Lost after Disk Replace

Sending logs is described here, you just download them from the nas then email them in:

 

http://kb.netgear.com/app/answers/detail/a_id/21543

 

There are many scenarios that a redundant volume does not offer protection against, therefore backups are essential.

 

What model readynas is it?

 

 

I think you need to ask Netgear support to look to look at this, open a support case and be prepared to pay.

Message 6 of 18
azerva
Tutor

Re: Data Volume Lost after Disk Replace


@vandermerwe wrote:

Sending logs is described here, you just download them from the nas then email them in:

 

http://kb.netgear.com/app/answers/detail/a_id/21543

 

There are many scenarios that a redundant volume does not offer protection against, therefore backups are essential.

 

What model readynas is it?

 

 

I think you need to ask Netgear support to look to look at this, open a support case and be prepared to pay.


Thanks for this. I have sent the logs. I have already contacted Netgear, but as you probably know / assume they are not very eager to help. At the moment they are looking into the case. However, I assume that the Netgear implementation of RAID is based on existing technolgoies, and that there is maybe at least one or two rebuild or salvage commands I can try. I was hoping I could maybe find an answer here.

 

The model is RNS-104, was bought diskless, and has 4 drives in it.

 

Thanks in advance for your time,

Alex

Message 7 of 18
JohnRo
NETGEAR Employee Retired

Re: Data Volume Lost after Disk Replace

Hi azerva, 

 

I looked at some information on the logs you have sent and on my understanding, it appears that Drive 4 may have also failed. To confirm this, please download disk diagnostic tools from Western Digital (WD lifeguard) or Seagate's(Seatools) webiste and check your drives. My hopes are still high that your data will be recovered.

 

I'll look forward to your response. 

 

 

Thanks,

Message 8 of 18
mdgm-ntgr
NETGEAR Employee Retired

Re: Data Volume Lost after Disk Replace

Your logs show that disk 4 has been giving errors for quite some time.

 

It would have been good to have been asked before there was a big problem whether the errors were a cause of concern.

 

When you replace a failed disk, the resulting resync that puts heavy stress on all disks can finish off another disk that is failing.

 

  ATA Error Count:                90
  Reallocated Sectors:            696
  Reallocation Events:            696
  Spin Retry Count:               0
  End-to-End Errors:              0
  Command Timeouts:               0
  Current Pending Sector Count:   48
  Uncorrectable Sector Count:     48
  Temperature:                    47
  Start/Stop Count:               1554
  Power-On Hours:                 14041
  Power Cycle Count:              232
  Load Cycle Count:               1555

Those ATA errors, reallocated sectors, current pending sectors and uncorrectable sector counts considered together indicate very clearly that this disk is bad.

 

An attempt to clone this disk e.g. using dd_rescue may be necessary.

 

This looks like a data recovery situation.

 

You may need to consider purchasing data recovery services. Support could explain the cost and the scope of what is covered.

Message 9 of 18
azerva
Tutor

Re: Data Volume Lost after Disk Replace

Hi mdgm,

 

thanks for your reply. I remember only getting one e-mail notification from the system regarding disk 4, somehow I was not aware that there were so many errors reported by the disk already, otherwise I would have mentioned it. Also, disk 4 is online now, that's what I find a little strange. When disk 3 failed a week ago, it was really offline, i.e. red and could not start. I even tried putting in a usb docking port and it didnt boot up. Disk 4 however "seems" to be fine now. This is why I still have some hope left that there is something I can do.

 

Fri Jul 24 2015 2:15:52
Disk: Disk in channel '4' (Internal) changed state from ONLINE to FAILED.
Fri Jul 24 2015 2:15:49
Disk: Disk in channel '3' (Internal) changed state from RESYNC to ONLINE.
Fri Jul 24 2015 2:15:46
Volume: The rebuild operation has finished on volume 'data'. However, the volume is still degraded.

By loooking at the timestamps of the events, it would seem that the resync finished just seconds (02:15:46) before disk 4 failed (02:15:52), but I guess this is a reporting error and the events probably happened the other way around, and the "failure" of disk 4 caused the resync to fail and it was just reported the other way around. What do you think, is this a correct assumption? Otherwise, If the resync did finish before disk 4 failed, then I would expect to be able to use the device like I did after disk 3 failed, i.e. volume in a degraded state with disk 4 offline.

 

All in all it seems that disks 3 and 4 were of poor quality, since both were bought at the same time and seem to have failed at almost the same time, even before warranty ran out.

Message 10 of 18
azerva
Tutor

Re: Data Volume Lost after Disk Replace


@JohnRo wrote:

Hi azerva, 

 

I looked at some information on the logs you have sent and on my understanding, it appears that Drive 4 may have also failed. To confirm this, please download disk diagnostic tools from Western Digital (WD lifeguard) or Seagate's(Seatools) webiste and check your drives. My hopes are still high that your data will be recovered.

 

I'll look forward to your response. 

 

 

Thanks,


Hi JohnRo,

 

I connected disk 4 to my PC and I can see it using SeaTools. Which Check / Repair should I try to do?

 

Thanks in advance.

Alex

Message 11 of 18
mdgm-ntgr
NETGEAR Employee Retired

Re: Data Volume Lost after Disk Replace

The problem with the disk caused the resync to fail.

 

In the status log and elsewhere I see a number of mentions of errors dating back some months.

 

Before replacing a disk that has failed it is advisable to check to see if any other disks have problems as well as consider where you are regarding backups.

 

For new disk purchases I would go with e.g. WD RED disks or SeaGate NAS disks.

Message 12 of 18
azerva
Tutor

Re: Data Volume Lost after Disk Replace


@mdgm wrote:

The problem with the disk caused the resync to fail.

 

In the status log and elsewhere I see a number of mentions of errors dating back some months.

 

Before replacing a disk that has failed it is advisable to check to see if any other disks have problems as well as consider where you are regarding backups.

 

For new disk purchases I would go with e.g. WD RED disks or SeaGate NAS disks.


Hi,

 

thanks for your reply. I am also pretty sure that this is what happened. However, and I'd really like to come back to this, unlike disk 3 which failed completly, disk 4 is still accessible (hooked up to my PC right now) and I would like some advice or opinions about what I could at least try before giving up on the data. I have the Seagate Tools installed (since its a Seagate Disk), a SHORT DST Test failed. The tool recommends to use the DOS version to try and repair the bad sectors.

 

Should I give this a go or does anyone have any better ideas?

 

Thanks

Alex

Message 13 of 18
vandermerwe
Master

Re: Data Volume Lost after Disk Replace

No.

Wait for Netgear support to see if data recovery is an option.  If they haven't offered it, ask them. 

If you start altering the data on the disks then it is likely you will make data recovery more difficult.  It is possible that cloning disks and trying to reassemble the raid structure may be what is needed, but leave this to someone with expertise, and be prepared to pay for it.  

Message 14 of 18
JennC
NETGEAR Employee Retired

Re: Data Volume Lost after Disk Replace

Hello azerva,

 

I replied to your PM. Like what mdgm and vandermerwe advised you to do, contact support and leave this to the experts.

 

The data you have in that disk 4 cannot be retrieved without the other 2 original disks. Put the disk back to the NAS bay where you removed it and have the system logs ready as you will have to give that to the support center.

 

You may need to prepare a spare disk to clone disk 4. But let support check first if it is still possible before getting one (if you do not a have spare disk and will have to buy).

 

here's what mdgm replied yesterday to this topic.

 

Those ATA errors, reallocated sectors, current pending sectors and uncorrectable sector counts considered together indicate very clearly that this disk is bad.

 
An attempt to clone this disk e.g. using dd_rescue may be necessary.

 
This looks like a data recovery situation.

 
You may need to consider purchasing data recovery services. Support could explain the cost and the scope of what is covered.

 

Regards,

Message 15 of 18
azerva
Tutor

Re: Data Volume Lost after Disk Replace #25483643

Hi guys,

 

update on the whole situation. Netgear Support recommended cloning the disk with ddrescue. I did this, the process finished with 4 errors. Below is the log file.

 

# Command line: ddrescue -fv /dev/sdb /dev/sda ddrescue.log
# current_pos  current_status
0x14419EC7C00     +
#      pos        size  status
0x00000000  0x139A33C4000  +
0x139A33C4000  0x00001000  -
0x139A33C5000  0x00004000  +
0x139A33C9000  0x00001000  -
0x139A33CA000  0xA76AF6000  +
0x14419EC0000  0x00001000  -
0x14419EC1000  0x00005000  +
0x14419EC6000  0x00002000  -
0x14419EC8000  0x8DA724E000  +

If I am reading this correctly, there are 4 relatively small areas (designated by the - at the end which are bad). I tried to start ReadyNAS with disks 1, 2 and the cloned disk 4, however this didn't work.

 

Are there any additional steps I need to consider? Should I consider re-running ddrescue using the -r option for more retries? Or is there a command to try to re-detect the RAID volume?

 

Thanks for any hints.

 

Alex

 

 



Message 16 of 18
vandermerwe
Master

Re: Data Volume Lost after Disk Replace #25483643

Are support not following up on this case?

Did they offer you data recovery services?

Message 17 of 18
JennC
NETGEAR Employee Retired

Re: Data Volume Lost after Disk Replace #25483643

Hello azerva,

 

I strongly recommend not to touch anything but just follow thier instructions. I don't think booting up the NAS with the remaining disks along with the cloned one will rebuild the RAID and get you access the data right away, there are still few steps they will have to do once you have the disk cloned. Please contact the support center to continue with it.

 

Regards,

Message 18 of 18
Top Contributors
Discussion stats
  • 17 replies
  • 3716 views
  • 2 kudos
  • 5 in conversation
Announcements