× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: ReadyNAS 626 Status Is "Degraded" After Successful Resync

nsne
Virtuoso

ReadyNAS 626 Status Is "Degraded" After Successful Resync

Just added a new HDD and am seeing this message after the resync:

 

Dec 13, 2022 06:33:28 AM
Volume: The resync operation finished on volume data. However, the volume is still degraded.

 

Any ideas what the issue could be?

 

I saw on some other threads that this could result from a drive that has a high reallocated sector count or current pending sector count, but I'm not sure how to determine that info. All six HDDs are appearing with a green indicator in the GUI.

 

There was at least one hiccup when adding the HDD: The NAS was initially populated with HDDs 1, 2, 3, 4 and 6. Bay 5 was empty. I removed the existing HDD 4 and (perhaps stupidly?) inserted the new HDD in its place. The NAS seemed to think that new disk was the original and then immediately began to resync, which resulted in an error. So I returned the original HDD 4, waited for the resync, then added the "new" HDD to the empty bay 5. It had data-0 and data-1 volumes on it, which I had to destroy.

 

I'm running a disk test now to make sure everything checks out SMART-wise. All my shares and data appear to be still intact.

Message 1 of 15
StephenB
Guru

Re: ReadyNAS 626 Status Is "Degraded" After Successful Resync


@nsne wrote:

 

There was at least one hiccup when adding the HDD: The NAS was initially populated with HDDs 1, 2, 3, 4 and 6. Bay 5 was empty. I removed the existing HDD 4 and (perhaps stupidly?) inserted the new HDD in its place. The NAS seemed to think that new disk was the original and then immediately began to resync, which resulted in an error. So I returned the original HDD 4, waited for the resync, then added the "new" HDD to the empty bay 5. It had data-0 and data-1 volumes on it, which I had to destroy.

 


Can you download the log zip file, and post the contents of mdstat.log?  (copy/paste it into a reply).

Message 2 of 15
nsne
Virtuoso

Re: ReadyNAS 626 Status Is "Degraded" After Successful Resync

My posts with the log contents keep vanishing! I guess some automated spam system is in place?

 

The log is attached as a PDF.

 

Message 3 of 15
StephenB
Guru

Re: ReadyNAS 626 Status Is "Degraded" After Successful Resync


@nsne wrote:

I guess some automated spam system is in place?


Yes.  The quarantine is manually reviewed by mods, so at some point the missing posts will probably be released.

 

This is what you posted:

 

md123 : active raid1 sdb7[0] sdf7[1]
 1953364992 blocks super 1.2 [2/2] [UU]

md124 : active raid5 sdb6[0] sdf6[3] sdc6[1]
 7811627008 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3] [UUU]

md125 : active raid5 sdd5[6] sde5[7](S) sdf5[5] sdc5[4] sdb5[2] sda5[1]
 7809112576 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]

md126 : active raid5 sdb4[4] sdf4[7] sda4[3] sdd4[5] sdc4[6]
 9766874560 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/5] [UUUUU_]

md127 : active raid5 sda3[5] sdf3[8] sdc3[7] sdd3[9] sdb3[6]
 29278353920 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/5] [UUUUU_]

 

 

Because you have unequal size disks, your volume has multiple RAID groups concatenated together.  If you'd always had the mix of disks in your screen shot, you'd have four RAID groups.  You actually have five, which means that at one point in the past you had smaller disks in the the array (6 and 8 TB).  More importantly you should actually have six RAID groups.

 

What you should have is

  • md127 - 6x6TB RAID-5
  • md126 - 6x2TB RAID-5 
  • md125 - 6x2TB RAID-5 
  • md124 - 4x4TB RAID-5 - should include the 14, 16, and the two 18 TB disks
  • md123 - 3x2TB RAID-5 - should include the 16 TB and the two 18 TB disks
  • md122 - 2x2TB RAID-1 - should include the two 18 TB disks.

 

What you do have is 

  • md127 - 6x6TB RAID-5 (degraded) -includes all disks, but sde detected as missing
  • md126 - 6x2TB RAID-5 (degraded) - includes all disks, but sde detected as missing
  • md125 - 5x2TB RAID-5 - includes all disks, but sde shown as a spare
  • md124 - 3x4TB RAID-5 - missing sde together
  • md123 - 2x2TB RAID-1 - missing sde altogether, and should be RAID-5

 

The problem is clearly the 18 TB disk in slot 5 (sde) - the one you just added.

 

The first thing to figure out is whether sde is healthy. It might not be tested in the volume disk test, since the NAS is confused about its status.  Can you connect it to a Windows PC (either with SATA or a USB adapter/dock)?

 

 

 

Message 4 of 15
nsne
Virtuoso

Re: ReadyNAS 626 Status Is "Degraded" After Successful Resync

You're right. At one point I did have smaller disks. I've had this system a while.

 

The sde is the fifth HDD in the array, correct?

 

I'd rather let the disk test complete before I go noodling around and potentially cause more issues. Once that completes (and I'm assuming it could take days), I'll pull the HDD, mount it in a spare PC and run SeaTools. Or is there another diagnostic check you'd recommend?

Message 5 of 15
StephenB
Guru

Re: ReadyNAS 626 Status Is "Degraded" After Successful Resync


@nsne wrote:

 

The sde is the fifth HDD in the array, correct?

 


Yes.  Though you can (and should) confirm this by looking at disk_info.log.  That will give you the serial number of the disk - which you can use to make sure you test the right one.

 


@nsne wrote:

 

I'd rather let the disk test complete before I go noodling around and potentially cause more issues. 


Yes, that makes sense.  If you have a backup plan in place, it would be good to make sure your backup is up to date.

 


@nsne wrote:

Once that completes (and I'm assuming it could take days), I'll pull the HDD, mount it in a spare PC and run SeaTools. Or is there another diagnostic check you'd recommend?


Seatools would be good.

 

I'd power down the NAS, pull the disk and double-check the serial number.  You can power up the NAS at that point (the volume should be ok, but of course still degraded).

 

Personally I'd run both the long generic test, and the "erase disk" advanced test.  The erase disk test will remove all partitions from the disk.  If you don't run that, then use the Windows Disk Manager to delete all the partitions on the disk. 

 

Then put the disk back - hot-insert is fine.  The NAS will sync everything again.  When it finishes, the volume will expand - it should end up at 68 TB (61.8 TiB)

Message 6 of 15
nsne
Virtuoso

Re: ReadyNAS 626 Status Is "Degraded" After Successful Resync

Thanks, Stephen. Disk 5 just failed the disk test, so that might be why the system was finicky about it. I'll have to arrange an RMA with Seagate.

 

How do I remove the disk and have the system repair itself?

 

 

Message 7 of 15
StephenB
Guru

Re: ReadyNAS 626 Status Is "Degraded" After Successful Resync


@nsne wrote:

Disk 5 just failed the disk test, so that might be why the system was finicky about it. I'll have to arrange an RMA with Seagate.

 


If you are still in the return window, exchange it with the seller instead.  The seller will give you a new disk with a full warranty.  Seagate generally will give you a recertified disk with a one-year warranty.

 


@nsne wrote:

 

How do I remove the disk and have the system repair itself?

 


If you just boot up the NAS again w/o disk 5, the data should still be available (and the volume still degraded).  But the data is more at risk than usual - another disk failure would result in all your data being lost.  

 

When the replacement arrives, I suggest testing it with Seatools first. I generally use both the full long test, followed by the full erase test.  I have sometimes purchased disks that pass one of these tests, but fail the other.

 

Then just hot-insert it into the NAS.

 

 

Message 8 of 15
nsne
Virtuoso

Re: ReadyNAS 626 Status Is "Degraded" After Successful Resync

Is there any way to remove the HDD altogether and have the volume self-heal to redundancy?

 

I ask because it's going to take a while—two weeks, I'd say, at least—to get a replacement, and I don't want to live with degraded data for a second longer than I have to.

Message 9 of 15
StephenB
Guru

Re: ReadyNAS 626 Status Is "Degraded" After Successful Resync


@nsne wrote:

Is there any way to remove the HDD altogether and have the volume self-heal to redundancy?

 


No. You can remove the disk, but the volume will remain degraded until you replace it.

Message 10 of 15
Sandshark
Sensei

Re: ReadyNAS 626 Status Is "Degraded" After Successful Resync

You can do it manually via SSH: Reducing-RAID-size-removing-drives-WITHOUT-DATA-LOSS-is-possible .  You'd need to "remove" the missing drive.  But given your multiple-layer volume, it's very tricky and there will be a lot of re-syncs, which put the data more at risk.  You are probably better off waiting for the replacement drive.

Message 11 of 15
nsne
Virtuoso

Re: ReadyNAS 626 Status Is "Degraded" After Successful Resync

Thanks, @Sandshark and @StephenB for all your input so far, and thanks too for your patience. I feel like some of this is basic ReadyNAS 101 info, but I haven't ever had to resort to it before and am super afraid of data loss.

 

I have a spare ReadyNAS 314 that I've been using for a long time as a share-based backup. Each of the eight shares on my 626 has its own daily backup job. The 314 initiates those jobs via rsync.

 

For peace of mind, what I'd like to do is fully clone my 626 to the 314 in order to preserve every byte of data as is, including snapshots. (Long story short, I've noticed some music files that are preserved in the snapshots but missing from the active share. At some point when I have several hours to spare, I'll have to compare the share vs. the snapshot and restore the missing files.)

 

Cloning would save me the worry of losing any data while the volume is degraded, and it also might allow me (I think?!) to wipe the 626 completely and start fresh to avoid the fragmentation that has occurred as I've expanded my NAS array with larger HDDs over the years.

 

A) Is the cloning strategy that I've proposed a good idea and B) if so, what method would you recommend to carry it out?

Message 12 of 15
StephenB
Guru

Re: ReadyNAS 626 Status Is "Degraded" After Successful Resync


@nsne wrote:

 

A) Is the cloning strategy that I've proposed a good idea and B) if so, what method would you recommend to carry it out?


Cloning would be one way to catch up the RN314, but it's not very sustainable.

 

One aspect is that you would also be cloning the OS - so the RN314 would be exactly the same as the RN628.  It would have the same NAS name for example.

 

Another aspect is that updating the RN314 would require copying everything on every RN628 disk to the RN314.  This would take some days, and both NAS would be out of service while you were doing this.  There would be a lot of disk handling involved, so it would be easy to do damage with a mis-step (losing track of what you've already copied, or accidentally dropping a disk).

 


@nsne wrote:

 

I have a spare ReadyNAS 314 that I've been using for a long time as a share-based backup. Each of the eight shares on my 626 has its own daily backup job. The 314 initiates those jobs via rsync.

 


This should be copying everything in the main share (so the main shares on the two NAS should be identical after the backups are complete).

 

FWIW, this is what I am doing myself.  I have daily snapshots on the backup NAS, which are updated shortly before the backups are scheduled.  So I do have some backup versioning via the snapshots, though the snapshots themselves are a bit different than the ones on the main NAS.

 

 

Message 13 of 15
nsne
Virtuoso

Re: ReadyNAS 626 Status Is "Degraded" After Successful Resync

Just an update to all this.

 

I RMA'd the Seagate EXOS 18TB X18 drive that the 626 had flagged as faulty. (During a pre-return wipe of the data, the HDD failed completely.) The seller sent a replacement HDD and I added that to the unit.

 

After re-syncing, the 626 is now showing healthy status and data redundancy. So things have stabilized and I have more peace of mind.

 

HOWEVER...

 

When I inserted the new HDD, the NAS immediately began re-syncing. I wasn't prompted to format the disk. This immediate re-sync has never happened before.

 

The NAS re-synced, then hung on apparent completion of that operation. No shares or data were accessible via FTP, GUI or SMB. This error has happened repeatedly with every new HDD I've inserted since 6.10.8. To restore access, the only solution is to power off the NAS physically.

 

After the NAS rebooted, it was showing a superfluous "data 4" volume. I had to delete that (using the DESTROY command) that to get the NAS to re-sync once again. During this time, the 626 said it was "reshaping" data (as opposed to re-syncing).

 

After the reshape, the 626 re-synced yet again. It was only after this that the volume showed as "Healthy."

 

This whole re-sync/reshape/re-sync process took about 4 or 5 days, if I recall.

 

So even though the volume is now reporting that it's "healthy," I have very little faith in the integrity of my data.

Message 14 of 15
Sandshark
Sensei

Re: ReadyNAS 626 Status Is "Degraded" After Successful Resync

If you insert an unformatted drive that is a replacement for another missing ot dead one, then the NAS will automatically start the re-sync.  The bad drive must have caused the NAS to at least start the process of adding the drive, so it considered the replacement drive just that -- a replacement.

 

I suspect that the "new volume" was data-4, not data 4.  That's the name it gives the 4th MDADM RAID in a NAS with multiple groups, like yours.  And when BTRFS can't figure out where it belongs, it shows up as a separate volume.  But, usually, it is supposed to be part of the main volume and may contain data from it.  Destroying it may not have been a good thing, you'll just have to see if any of your data is now corrupt.

 

If your RAID1 layer is now expanding to RAID5, that's what's reshaping.

Message 15 of 15
Top Contributors
Discussion stats
  • 14 replies
  • 1587 views
  • 0 kudos
  • 3 in conversation
Announcements