What do I need to know before swapping in a new drive?

[fearless_fool]

As reported in http://www.readynas.com/forum/viewtopic.php?f=24&t=74918, one of my SAMSUNG HD103UJ drives in my ReadyNAS NV is on the decline: the system gets stuck on FSCK at 92%, Raw Read Error Rate, Read Soft Error Rate and Current Pending Sector errors are climbing.

I just received a new HD103UJ (thank you Amazon). I'm running RAIDiator 4.1.13 on my ReadyNAS NV in X-RAID configuration, so I should be able to swap out the failing drive and put in a fresh one, right? But I've read a few horror stories on systems that have failed during the refresh process, so I want to proceed carefully. Here are my questions:

• Hot or cold swap: which is safer? Note that my system gets hung during FSCK on startup (so I have to do the 'hold the power button for 5 seconds on powerup to skip FSCK' to get past that), so that *might* argue for hot swapping. Comments?


• The SMART report says that Channel 3 is dying. Just to make perfectly sure: are the disks labelled 1-2-3-4 left to right? I don't want to yank the wrong one.


• StephenB and others recommend backing up the ReadyNAS before attempting a disk swap. Ignoring the fact that this *is* my backup system, how and where would you back up 2TB of data? I"m concerned that by the time I procure any necessary gear and to a backup, the drive will have failed completely. My gut says that I should install the new disk sooner than later. What would you do?

That's it for now. I'm sure other will have similar questions.

[vandermerwe]

Ideally you should have a backup, do you have a copy of the data on the nas somewhere else? If its a backup then presumably there is a primary copy of all the data somewhere else.
USB backup is probably the easiest, with an NV you'd only be able to use a 2tb usb drive so all the data would need to fit into about 1.86tb, of course you could split the backup over 2 usb disks.
As for replacing the disk, you should do this ASAP.
Hot swap it - remove disk 3 (yes they are numbered from left to right, 1-4.
Then insert the new disk.
Don't format the new disk, the nas will do it.
It is wise to check the new disk for errors using a pc before inserting it, using vendor tools - for samsung drives you will probably need seatools from seagate or a generic hard drive testing tool like hard disk sentinel, which costs money.
The disk you are using is not on the compatibility list, but given that you already have one and it works, you should be OK.

[StephenB]

fearless_fool wrote:
Hot or cold swap: which is safer? Note that my system gets hung during FSCK on startup (so I have to do the 'hold the power button for 5 seconds on powerup to skip FSCK' to get past that), so that *might* argue for hot swapping. Comments?

The benefit of hot-swap is that the NAS detects the new disk insertion directly, and starts the resync process.

Of course there are times when you don't want the disk wiped and rebuilt - for instance if you are removing it so you can run a diagnostic on it. In those cases you should always cold-swap.

If you are inserting an unformatted disk, both should work (as the NAS should discover it is unformatted when it starts up). Though still, I think hot-swap is generally the best way to go when adding/replacing disks in the array.

fearless_fool wrote:
The SMART report says that Channel 3 is dying. Just to make perfectly sure: are the disks labelled 1-2-3-4 left to right? I don't want to yank the wrong one.

It would be nice if there was a label. One thing to add to vandermerwe's reply: If for some reason you accidentally remove the wrong disk, the best thing to do is immediately power down the NAS, and cold-insert it. You don't want the system to rebuild the array. Also, if you do just stick it back in (hot-insert) be sure you wait until the resync is complete before proceeding. The NAS can only rebuild one disk at a time. Doing two at once will destroy the volume.

fearless_fool wrote:
StephenB and others recommend backing up the ReadyNAS before attempting a disk swap. Ignoring the fact that this *is* my backup system, how and where would you back up 2TB of data? I"m concerned that by the time I procure any necessary gear and to a backup, the drive will have failed completely. My gut says that I should install the new disk sooner than later. What would you do?

A couple of comments here:
(a) Anyone who cares about their data should develop a backup strategy that protects it against the various failure scenarios. There is of course a cost, so you might chose not to protect it against every possibility. Loss of data on one device is reasonably common. Sometimes failure of the backup device isn't detected until you attempt to recover the data from it. So two failures are more likely than you might think. Then there is a disaster recovery - lightening, fire, theft, etc - something that destroys all your devices.

My strategy is set up to protect against two failures (primary device and the primary backup), and also the disaster scenario. Basically I keep a primary and a secondary backup on site, and use crashplan for the disaster recovery. These are always recent, so when something does fail, I can act immediately. Possibly overkill, but I prefer a "belt and suspenders" approach (and learned the hard way about needing a secondary backup some years ago).

(b) You are correct in thinking that delay increases the possibility of data loss in the NAS. You have 4 disks in the NAS, probably all are about the same age. Another one could be start failing (or might even be already silently failing). The volume is more vulnerable because you have lost redundancy.

So if your NAS is only used for backup, and is not used for primary storage, then I would probably risk doing the disk insertion immediately w/o making a secondary backup. That assumes your primary storage (pc, whatever) is running normally and is not at risk of failure - something I would double-check before proceeding. If the NAS is used for some primary storage, then I would back that stuff up first, and then do the new disk insertion.

Then perhaps after you are up and running again, you can revisit the overall backup plan, and make any changes you think are needed.

EDIT: I just wanted to add that 2 TB of data can easily be backed up on one USB drive. It gets more complicated when you have > 4 TB, as then the backup won't fit on a single disk. One approach to that problem (which you don't have at this point) is back up to another NAS.

[fearless_fool]

Thank you everybody for informative and detailed responses.

The new disk has been inserted and is currently re-syncing -- RAIDar reports 66 hours to complete (and climbing?!?).

The log looks reasonable, but I have a question about it:

Severity 	Date 	Message 
	Thu Jan 23 14:35:04 PST 2014	A SATA reset has been performed on one or more of your disks that may have affected the RAID parity integrity. It is recommended that you perform a RAID volume resync from the RAID Settings tab ( accessible in the Volumes page => Volume tab in FrontView ). The resync process will run in the background, and you can continue to use the ReadyNAS in the meantime.
	Thu Jan 23 14:33:25 PST 2014	RAID sync started on volume C.
	Thu Jan 23 14:33:24 PST 2014	Disk initialization successfully finished.
	Thu Jan 23 14:33:20 PST 2014	Disk add event occurred on SATA channel 3.

Should I start a RAID volume resync now (as the log file requests)? Or is it actually underway? Or do I do a RAID volume resync after the device resync finishes (now reported as 88 hours from now?!!!??)? (Will it converge?)

[StephenB]

Just let it finish resyncing. If the volume status then says redundant, you are fine.

Also, you can't trust the time estimates - they vary a lot from reality. I'd be very surprised if it isn't finished much sooner.

[fearless_fool]

@StephenB: Many thanks. You were right about the time estimates -- the resync finished this morning, 62 hours ahead of previously predicted schedule.