ReadyNAS 104 very slow to recover data after upgrading disk size

[Unsound]

Hi Guys,

Hopefully someone can help - I have a ReadyNas 104, which up until recently had 4x4TB drives - I have been running out of space, so I have sequentially added 2 x 8TB drives - it resynced and was showing 25% capacity spare after the second - I have added a 3rd 8Tb drive which it has acceted, but the data resync is ridiculously sloooow - 0.05% per hour.

This did not occur with the first 2x 8Tb drives, but is occuring with the 3rd - any ideas?

All appears OK on the NAS web page - just says performance degraded, volume resyning - no errors or warnings.

Thanks,

Malcolm

[Sandshark]

It's usually a sign that one of the drives (probably the newest) is encountering read/write problems.

[Unsound]

Thanks Sandshark,

Appreciate the response - it is a brand new drive - does this mean the drive is faulty??

Cheers,

M

[StephenB]

---

@Unsound wrote:

it is a brand new drive - does this mean the drive is faulty??

---

It might be.  It could also be that a different drive is failing, or there might be another explanation. 

Download the log zip file, and look for disk errors in system.log and kernel.log.  Maybe also the smart stats in disk_info.log.

[Unsound]

Thanks @StephenB 

I am not an expert - I have downloaded the logs, but can't make head nor tail of them.

I attach them - would you be able to assist - any guideance would be greatly appreciated.

Thanks,

Malcolm

[Unsound]

@StephenB 

[Unsound]

@StephenB 

[StephenB]

Unfortunately this stuff in kernel.log isn't good.

Jan 09 08:54:05 NAS kernel: md/raid:md127: read error not correctable (sector 24311632 on sdb3).
Jan 09 08:54:05 NAS kernel: md/raid:md127: read error not correctable (sector 24311640 on sdb3).
Jan 09 08:54:05 NAS kernel: md/raid:md127: read error not correctable (sector 24311648 on sdb3).
Jan 09 08:54:05 NAS kernel: md/raid:md127: read error not correctable (sector 24311656 on sdb3).
Jan 09 08:54:05 NAS kernel: md/raid:md127: read error not correctable (sector 24311664 on sdb3).
...
Jan 09 11:19:53 NAS kernel: md/raid:md127: read error not correctable (sector 43087368 on sdc3).
Jan 09 11:19:53 NAS kernel: md/raid:md127: read error not correctable (sector 43087376 on sdc3).
Jan 09 11:19:53 NAS kernel: md/raid:md127: read error not correctable (sector 43087384 on sdc3).
Jan 09 11:19:53 NAS kernel: md/raid:md127: read error not correctable (sector 43087392 on sdc3).
Jan 09 11:19:53 NAS kernel: md/raid:md127: read error not correctable (sector 43087400 on sdc3).
Jan 09 11:19:53 NAS kernel: md/raid:md127: read error not correctable (sector 43087408 on sdc3).
Jan 09 11:19:53 NAS kernel: md/raid:md127: read error not correctable (sector 43087416 on sdc3).
Jan 09 11:19:53 NAS kernel: md/raid:md127: read error not correctable (sector 43087424 on sdc3).
Jan 09 11:19:53 NAS kernel: md/raid:md127: read error not correctable (sector 43087432 on sdc3).
...
Jan 09 11:19:53 NAS kernel: sd 2:0:0:0: rejecting I/O to offline device
Jan 09 11:19:53 NAS kernel: sd 2:0:0:0: rejecting I/O to offline device
Jan 09 11:19:53 NAS kernel: ata3.00: detaching (SCSI 2:0:0:0)
Jan 09 11:19:53 NAS kernel: sd 2:0:0:0: [sdc] Synchronizing SCSI cache
Jan 09 11:19:53 NAS kernel: sd 2:0:0:0: [sdc] Synchronize Cache(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Jan 09 11:19:53 NAS kernel: sd 2:0:0:0: [sdc] Stopping disk

It looks like you have two failing disks.  Do you have a backup of the data?

[Unsound]

@StephenB 

Unfortunately, no backup - I have all the data spread across a number of hard drives in computers, etc as original copies.

Can you tell which discs ar failing?

The ReadyNas web page shows all drives are green/healthy - 66 hours remaining?

Thanks again,

Malcolm

[Sandshark]

From the log, it looks like drives 2 & 3.  But sdb isn't always drive 2 and sdc drive 3.  If you hot pull and replace drives, the order can get mixed up till the next re-boot.

At this point, I think you need to hope the sync completes.  then, before you rplace any more drives, make a backup.  With two drives giving you read issues, your volume is in jeopardy of total loss.

[Unsound]

@Sandshark  Thanks,

Drives in bays 2 & 3 are the new drives - they are compatible according to the ReadyNAS compatability table - I would be stunned if they both failed?? But I take your point that the bay numbers may be mixed up.

I have 3 new (over the past few months - 2 this week) 8Tb drives and one old 4Tb drive - I can only hope it is the 4 Tb drive that is failing.

Thanks again,

Malcolm

[StephenB]

---

@Unsound wrote:

Drives in bays 2 & 3 are the new drives  - I would be stunned if they both failed??

---

It can happen; I've had drives that failed out of the box.  Generally I test them before I put them in the NAS (in a Windows PC with vendor tools - Seatools in your case).

However, the mapping I am seeing in the logs isn't stable, because the system is resetting the SATA interface due to the errors.  Looking more closely at the mapping, disk 1 (bay 2, since the disk numbering starts from 0) is sometimes sdb, and sometimes sdc.  So it actually looks like only one disk is having issues.

Jan 07 22:26:51 NAS kernel: disk 0, wo:0, o:1, dev:sde1
Jan 07 22:26:51 NAS kernel: disk 1, wo:0, o:1, dev:sdb1
Jan 07 22:26:51 NAS kernel: disk 2, wo:0, o:1, dev:sdd1
Jan 07 22:26:51 NAS kernel: disk 3, wo:0, o:1, dev:sda1
...
Jan 09 08:54:05 NAS kernel: md/raid:md127: read error not correctable (sector 24311632 on sdb3). 
...
Jan 09 08:54:56 NAS kernel: disk 0, wo:0, o:1, dev:sde1 
Jan 09 08:54:56 NAS kernel: disk 1, wo:1, o:1, dev:sdc1 
Jan 09 08:54:56 NAS kernel: disk 2, wo:0, o:1, dev:sdd1 
Jan 09 08:54:56 NAS kernel: disk 3, wo:0, o:1, dev:sda1
...
Jan 09 11:19:53 NAS kernel: md/raid:md127: read error not correctable (sector 43087368 on sdc3).
...
Jan 09 11:23:43 NAS kernel: disk 0, o:1, dev:sde3
Jan 09 11:23:43 NAS kernel: disk 1, o:1, dev:sdb3
Jan 09 11:23:43 NAS kernel: disk 2, o:1, dev:sdd3
Jan 09 11:23:43 NAS kernel: disk 3, o:1, dev:sda3

[Unsound]

@StephenB Thanks again,

I think I'll wait for the resync to complete and then replace the drive.

29 hours to go!

Cheers,

M

[Unsound]

@StephenB  - well the Resync has finished, but now it is Reshaping the data @50.88%, with 181 hours to go!

I was planning on replacing the potentially faulty disk - can I remove it whilst it is "Reshaping" the data?

I can still access my data via the NAS web page, but not via my Finder (Mac).

Thanks,

Malcolm

[StephenB]

---

@Unsound wrote:

I was planning on replacing the potentially faulty disk - can I remove it whilst it is "Reshaping" the data?

There are two steps, because you have two different disk sizes.  The way this works is that you have two different RAID groups in the data volume.  One is the original 4x4TB RAID-5 group.  The new group uses the extra capacity in the new 8 TB disks.  Both are concatenated into the single data volume.

The resync was rebuilding the original group.  The Reshaping is converting the second 2x4TB group (currently RAID-1) to 3x4TB RAID-5.  That step is what expands the array.

There is risk that removing the disk now will cause the new group to fail.  Of course there is also risk that the disk will completely fail...

I'm don't know which path is safest.  I'd be inclined to let the process complete, but that is really just a feeling - not something with any data behind it.