Re: ReadyNAS 716X data- Two volumes after Resync

RN716x · ‎2023-01-04

System: Readynas 716X

F/W: 6.10.8

Old HDDs: 6TBx6 (WD Red)

New HDD: 8TBx1 (WD Red)

Raid 5 (X-Raid)

I have recently experienced some issues with my Readynas system. I found out that I had to replace the drive in slot4, which was working fine but about to fail. During the Resync process, apparently after it has completed successfully, slot0 drive failed and now I have two volumes data0 and data (image attached). I went through the KB and there are a couple of posts there about similar issues. I cannot enable SSH, therefore, still unable to explore any of what's been suggested.

New Volume

Please note that I've initially decided to replace this drive because the system started showing degraded Raid, specifically slot 4 in red from time to time. So I thought it would be better to replace this drive before it's too late. This problem (not able to access my nas) started when the resync was completed (per the system log) but with 1 drive (slot 0 this time) showing disk failure (the disk is surprisingly healthy now). I was still -then- able to explore the directories and they were all there and it seemed everything was normal, thus, -because the system was showing red Raid- I decided that if I reboot the nas system all the reds will disappear and it'll go back to normal then will be in peace and I'll be able to access the nas with no issues, alas that wasn't the case. It seems this is exactly similar to this kb post:

https://community.netgear.com/t5/Using-your-ReadyNAS-in-Business/ReadyNas-526X-quot-Data-dead-quot/m...

Now it's asking to 'remove inactive volumes to use the drives'.

In an effort to try to resolve this, -when I restarted the Nas system and found out of the problem-, I tried to remove/ isolate the drive that was reported as failure during the resync (now working with no issue) in slot 1, inserted the old drive that was replaced back in its old slot (slot 4), hoping everything will go back to where we started and should have worked normal, but nothing, it's exactly the same red Raid and two data volumes data (27.7TB) and data-0 (0TB). After going through the relevant KBs, Now I probably understand why that may be the case, since the BTRF journal could have got corrupted as a result of the sudden interruption in the sync, and therefore the system could not mount back to the original data volume.

From the KB threads I've also come across these threads, which I believe share similar symptoms/ issues:

https://community.netgear.com/t5/Using-your-ReadyNAS-in-Business/Test-posting/td-p/2119388/page/2

https://community.netgear.com/t5/Using-your-ReadyNAS-in-Business/No-volume-exists/td-p/1729434

Please note that I could not get the SSH service enabled, could be to do with the volume mounting issue.

image(1).png

I've tried my luck with Netgear support, but it looks like they no longer provide support to this system.

Would appreciate any support that could allow me to mount back/ access the data volume.

StephenB · ‎2023-01-06

@RN716x wrote:

Is it asking for an input here?

root@MaQBaLiNAS:~# cat /etc/mdadm/mdadm.conf

CREATE owner=root group=disk mode=0660 auto=yes

This is the same as the file on my system, so nothing unusual here.

@RN716x wrote:

root@MaQBaLiNAS:~# mdadm --assemble --scan --force
mdadm: NOT forcing event count in /dev/sdc3(3) from 11031 up to 19055
mdadm: You can use --really-force to do that (DANGEROUS)
mdadm: /dev/md/data-0 assembled from 4 drives - not enough to start the array.
mdadm: No arrays found in config file or automatically

This is telling you that /dev/sdc3 is missing a lot of writes - so it is seriously behind the other drives in the array.

You could proceed with --really-force instead of force. You could end up with a lot of data loss/corruption if you do that.
Another other option is go back to SDA. Power down, remove SDC, insert the SDA. Power up, and try the same mdadm command again, and see if the event count gap is smaller. If you go this route, you should definitely mount the volume as read-only - as we know that SDA is failing, and writes will likely only accelerate that. Also they would increase the event gap between sdc and the rest of the array (making going back to SDC even more risky).

This isn't an obvious decision. SDA likely will have a smaller event gap, but we already know it has unreadable sectors. Plus it might fail when you try to offload data. Still, I think it is likely the best path (with read-only mounting).

A variant of (2) is to clone SDA, and insert the clone. The cloning process will skip over any unreadable sectors on the original. The benefit of the variant is that there will be no bad sectors detected on the clone (which is a mixed blessing, as bad sectors do give you some information on file corruption). The risk is that the original will completely fail during the cloning process.

Either way (1, 2, or 2 variant), you'd copy off as much data as you can, do a factory default with the two new drives installed in place of SDA and SDC. You'd reconfigure the NAS at that point, and then restore the files from the backup.

View solution in original post

StephenB · ‎2023-01-04

Do you have any log zip files - either after the first resync, or after the reboot? Maybe also download one now if you can.

Sandshark · ‎2023-01-04

Your old drive is out of sync with the others, so adding it back will cause more problems, not fix them. But since it didn't mount, probably no additional damage was done.

Your "two volumes" are actually two broken halves of your old volume. The NAS cannot properly mount them as one, presumably because of the failed drive. Since drive 1 is the failed one, that may be the reason you cannot enable SSH. I suggest you try booting without drive one, in read-only mode (using the reset button menu to do so) and see if that doesn't fix the issue (one accessible volume, though it will be "degraded) or at least let you enable SSH. I may be wrong, but I believe read only mode still allows writing to the OS partition, which is needed to enable SH. Maybe @StephenB can confirm.

RN716x · ‎2023-01-04

@StephenB

I've sent you a link to the logs.

@Sandshark
I'll try that and update.

Thanks

RN716x · ‎2023-01-04

So follwoing what @Sandshark has said, I am now able to see that the SSH service is enabled by default. I left it there for now, hoping to get some tips on how to proceed further.

StephenB · ‎2023-01-04

It's not enabled by default, but however it happened, it is enabled.

The log zip is for some reason missing most of the files. It looks like it is from 12/20.

The problem is that the resync of the new 8 TB disk failed due to disk errors on disk sda (in the first slot):

Dec 20 03:36:33 MaQBaLiNAS mdadm[3964]: Fail event detected on md device /dev/md127, component device /dev/sda3

Dec 20 03:36:02 MaQBaLiNAS kernel: blk_update_request: I/O error, dev sda, sector 11233252792
Dec 20 03:36:02 MaQBaLiNAS kernel: md/raid:md127: read error not correctable (sector 11223815544 on sda3).
Dec 20 03:36:02 MaQBaLiNAS kernel: md/raid:md127: read error not correctable (sector 11223815552 on sda3).
Dec 20 03:36:02 MaQBaLiNAS kernel: md/raid:md127: read error not correctable (sector 11223815560 on sda3).
Dec 20 03:36:02 MaQBaLiNAS kernel: md/raid:md127: read error not correctable (sector 11223815568 on sda3).
Dec 20 03:36:02 MaQBaLiNAS kernel: md/raid:md127: read error not correctable (sector 11223815576 on sda3).
Dec 20 03:36:02 MaQBaLiNAS kernel: md/raid:md127: read error not correctable (sector 11223815584 on sda3).
Dec 20 03:36:02 MaQBaLiNAS kernel: md/raid:md127: read error not correctable (sector 11223815592 on sda3).
Dec 20 03:36:02 MaQBaLiNAS kernel: md/raid:md127: read error not correctable (sector 11223815600 on sda3).
Dec 20 03:36:02 MaQBaLiNAS kernel: md/raid:md127: read error not correctable (sector 11223815608 on sda3).
Dec 20 03:36:02 MaQBaLiNAS kernel: md/raid:md127: read error not correctable (sector 11223815616 on sda3).

There are several more of the uncorrectable errors after this (look in systemd-journal.log).

One option is to try cloning either disk 1 (sda) or the original sdd to a new drive. Not sure which is better. Then you could forcibly assemble the array, and ideally offload data before preceeding. There likely would be some data loss (since there are errors on sda, and sdd also clearly had issues).

You could also try RAID recovery software - though you'd need to be able to connect the drives to a Windows PC (getting a suitable USB enclosure), and software like ReclaiMe is also pricey. Plus you'd need storage to offload the data that is recovered.

RN716x · ‎2023-01-04

If the original sdd (6TB) is still working in the pre-sync state. i.e. I removed it and replaced it with the new 8TB. Would I still need to clone it to a new drive or can I just connect it and try to assemble?

What difference would cloning make then?

For the sda (6TB), the plan was to replace it when the sdd is resynced successfully with a new 8TB that I've also got ready that I might use for cloning either now.

I'm trying to weigh my options here, would re-connecting the original sdd (6TB) help here while maybe isolating sda just to take a backup first? It should be in the pre-sync state.

Would you elaborate further on this part please, just to make sure I'm following correctly:
"Then you could forcibly assemble the array,"

StephenB · ‎2023-01-04

@RN716x wrote:
If the original sdd (6TB) is still working in the pre-sync state. i.e. I removed it and replaced it with the new 8TB. Would I still need to clone it to a new drive or can I just connect it and try to assemble?

The two paths are to start with the original sdd, or start with the original sda. Which is best depends on the amount of bad sectors on the two drives.

Cloning just eliminates the chance that the drive will fail during resync. SDA already did, so we know that one would be challenging. If the drive is cloned with errors, then there will probably be corrupt files/folders, but the RAID won't be able to detect that (since the sectors would all be readable).

SDD isn't so clear - you had issues with it before. But you could try putting the drive back into the NAS (with SDA removed), force the array to assemble, and then try to sync with a new 8 TB drive in slot 1. Assuming it does assemble, you could try making a backup first.

@RN716x wrote:

Would you elaborate further on this part please, just to make sure I'm following correctly:
"Then you could forcibly assemble the array,"

Basically there are write counters on each drive in the RAID array. Those are used by the RAID software to make sure all writes to all drives completed. When you reinsert a drive, in almost all cases the write counters will not be the same (even if you don't think you wrote anything to the array). If the counts aren't very close, the array won't mount - that is one cause of the "inactive volume" problem.

In your case, the count for sda3 is already off, and it was kicked out of the array. Likely that is partly due to the burst of errors at the end of the sync. Since it is off, the original sdd3 is almost certainly also off.

Overall, the process would be to power down the NAS, remove sda, and put back the original sdd. Then power up (perhaps read-only for safety). If that works, then you should back up the data if you can, and hot-insert one of the 8 TB drives.

If it's the one that you already tried, then remove the partitions with Windows Disk Manager before reinserting (or alternatively select it from the NAS web ui, and manually format it).

If that resync completes, then remove sdd, and insert the new 8 TB drive in it's place (NAS running).

However, I think the inactive volume error will persist when you power up the NAS w/o sda and with the original sdd. In that case, you'd need to use some linux commands to force the array to mount. Normally I've done this in tech support mode (not in the normal boot). But the commands should work in a normal boot as well.

Something like

mdadm --force --assemble /dev/md127
btrfs scan device
mount /dev/md127 /data

If you prefer mounting the array read-only then put a "-o ro" after "mount" in the last command.

If you try this, make sure you log in as root (using the NAS admin password) when accessing the NAS with ssh.

RN716x · ‎2023-01-05

After connecting to the readynas, and using the mdadm command:

@StephenB wrote:
Something like

mdadm --force --assemble /dev/md127
btrfs scan device
mount /dev/md127 /data

If you prefer mounting the array read-only then put a "-o ro" after "mount" in the last command.

If you try this, make sure you log in as root (using the NAS admin password) when accessing the NAS with ssh.

I get the below:

root@MaQBaLiNAS:~# mdadm /dev/md127
mdadm: cannot open /dev/md127: No such file or directory
root@MaQBaLiNAS:~# btrfs scan device
btrfs: unknown token 'scan'
    btrfs subvolume create [-i <qgroupid>] [<dest>/]<name>

then when trying to mount:

root@MaQBaLiNAS:~# mount /dev/md127 /data
mount: special device /dev/md127 does not exist

Not sure If I'm missing something here.

StephenB · ‎2023-01-05

@RN716x wrote:

root@MaQBaLiNAS:~# mdadm /dev/md127
mdadm: cannot open /dev/md127: No such file or directory

Not sure If I'm missing something here.

Well, you didn't try the mdadm command I posted.

One puzzle here is the missing files in your log zip. Can you post /etc/mdadm/mdadm.conf ???

I accidentally got the btrfs command backwards - it should have been

btrfs device scan

But you need to get the mdadm part to work before you try the btrfs command.

RN716x · ‎2023-01-06

@StephenB wrote:
Well, you didn't try the mdadm command I posted.

root@MaQBaLiNAS:~# mdadm --force --assemble /dev/md127
mdadm: --force does not set the mode, and so cannot be the first option.

@StephenB wrote:

One puzzle here is the missing files in your log zip. Can you post /etc/mdadm/mdadm.conf ???

root@MaQBaLiNAS:~#  /etc/mdadm/mdadm.conf
-bash: /etc/mdadm/mdadm.conf: Permission denied

@StephenB wrote:

btrfs device scan

But you need to get the mdadm part to work before you try the btrfs command.

============

Still tried it even though,

root@MaQBaLiNAS:~# btrfs device scan
Scanning for Btrfs filesystems

Then tried this command

root@MaQBaLiNAS:~# mdadm --assemble --scan
mdadm: /dev/md/data-0 assembled from 4 drives - not enough to start the array.
mdadm: No arrays found in config file or automatically

5 drives are connected (inc. original 6TB sdd4). The only drive not connected is the sdda. Should I try connecting sdda which failed during sync but is still working, or perhaps the sdd4 8TB which apparently didn't successfully sync?

The 'mount' command throws this output:

root@MaQBaLiNAS:~# mount /dev/md127 /data
mount: special device /dev/md127 does not exist

StephenB · ‎2023-01-06

@RN716x wrote:

root@MaQBaLiNAS:~# /etc/mdadm/mdadm.conf
-bash: /etc/mdadm/mdadm.conf: Permission denied

One challenge here is sorting out how much linux people know.

You tried to execute the conf file - which can't be done. If you want to list it, you type

cat /etc/mdadm/mdadm.conf

But there is a hint that the RAID array isn't in the config file (the error message you got when you tried to do mdadm --assemble --scan).

@RN716x wrote:

root@MaQBaLiNAS:~# mdadm --assemble --scan
mdadm: /dev/md/data-0 assembled from 4 drives - not enough to start the array. mdadm: No arrays found in config file or automatically

Again, this is the step you HAVE to get resolved first. There is no point in continuing until you can force the mdadm array to assemble.

Try adding --force to this command.

mdadm --assemble --scan --force

FWIW, I am worried that even if you get past this you will end up needing to do a factory default - it's pretty clear from the log zip that you have more issues - not just an inactive volume. So if we can get it mounted, it will be important to offload the data to other storage.

RN716x · ‎2023-01-06

Is it asking for an input here?

root@MaQBaLiNAS:~# cat /etc/mdadm/mdadm.conf
CREATE owner=root group=disk mode=0660 auto=yes

Then

root@MaQBaLiNAS:~# mdadm --assemble --scan --force
mdadm: NOT forcing event count in /dev/sdc3(3) from 11031 up to 19055
mdadm: You can use --really-force to do that (DANGEROUS)
mdadm: /dev/md/data-0 assembled from 4 drives - not enough to start the array.
mdadm: No arrays found in config file or automatically

StephenB · ‎2023-01-06

@RN716x wrote:

Is it asking for an input here?

root@MaQBaLiNAS:~# cat /etc/mdadm/mdadm.conf

CREATE owner=root group=disk mode=0660 auto=yes

This is the same as the file on my system, so nothing unusual here.

@RN716x wrote:

root@MaQBaLiNAS:~# mdadm --assemble --scan --force
mdadm: NOT forcing event count in /dev/sdc3(3) from 11031 up to 19055
mdadm: You can use --really-force to do that (DANGEROUS)
mdadm: /dev/md/data-0 assembled from 4 drives - not enough to start the array.
mdadm: No arrays found in config file or automatically

This is telling you that /dev/sdc3 is missing a lot of writes - so it is seriously behind the other drives in the array.

You could proceed with --really-force instead of force. You could end up with a lot of data loss/corruption if you do that.
Another other option is go back to SDA. Power down, remove SDC, insert the SDA. Power up, and try the same mdadm command again, and see if the event count gap is smaller. If you go this route, you should definitely mount the volume as read-only - as we know that SDA is failing, and writes will likely only accelerate that. Also they would increase the event gap between sdc and the rest of the array (making going back to SDC even more risky).

This isn't an obvious decision. SDA likely will have a smaller event gap, but we already know it has unreadable sectors. Plus it might fail when you try to offload data. Still, I think it is likely the best path (with read-only mounting).

A variant of (2) is to clone SDA, and insert the clone. The cloning process will skip over any unreadable sectors on the original. The benefit of the variant is that there will be no bad sectors detected on the clone (which is a mixed blessing, as bad sectors do give you some information on file corruption). The risk is that the original will completely fail during the cloning process.

Either way (1, 2, or 2 variant), you'd copy off as much data as you can, do a factory default with the two new drives installed in place of SDA and SDC. You'd reconfigure the NAS at that point, and then restore the files from the backup.

RN716x · ‎2023-01-06

Following option 2, removed sdd, connected sda in read-only. The difference is clearly narrower.

root@MaQBaLiNAS:~# mdadm --assemble --scan --force
mdadm: NOT forcing event count in /dev/sda3(0) from 18988 up to 19055
mdadm: You can use --really-force to do that (DANGEROUS)
mdadm: /dev/md/data-0 assembled from 4 drives - not enough to start the array.
mdadm: No arrays found in config file or automatically
root@MaQBaLiNAS:~# mdadm --assemble --scan --really-force
mdadm: forcing event count in /dev/sda3(0) from 18988 upto 19055
mdadm: /dev/md/data-0 assembled from 5 drives - not enough to start the array.
mdadm: No arrays found in config file or automatically

RN716x · ‎2023-01-06

Quick update after restarting the Nas. Not sure if trying to reconnect sdd to resync would be a good option, hoping to have better redundancy till offload the data.

StephenB · ‎2023-01-06

@RN716x wrote:

Quick update after restarting the Nas. Not sure if trying to reconnect sdd to resync would be a good option, hoping to have better redundancy till offload the data.

I'm glad it finally mounted.

Don't add any more drives - that will start a resync process that will stress the failing one. If you push it over the edge trying to get redundancy, then you will lose all your data. Remember this whole mess started when disk errors on sda caused the resync to fail.

Just offload everything you can as quickly as you can - using what space you have now to get things started.

RN716x · ‎2023-01-08

Final update: I have finally managed to offload the data safely. Sometime soon, will take care of the Nas issue with the drives.

@StephenBThanks a lot man.

@SandsharkThanks for your input.

StephenB · ‎2023-01-09

I'm glad you got the data off. It's very nerve-wracking when multiple disks are having problems.