Re: Expansion Difficulties - XRAID2 - Case #17781064

alcester · ‎2012-01-30

I'd be very grateful for any suggestions to help solve a problem I have encountered with expansion. I have beed looking at the forums for a while and can't find an exact situation to match.

I bought my Ultra 4 Plus with no disks 6 weeks ago and initially installed 2 x 2TB Seagate drives (brand new) from the HCL. As expected, I ended up with a single volume (redundant) of just under 2TB (1.8TB) capacity. I have been using this without any problems since then (streaming music/video and backups of family PC/Macs).

This weekend, I took delivery of a further 2 x 2TB drives (identical to the first 2). To install, I first powered down the system and put both disks in together. The system rebooted and started a restripe process. During this, one of the original drives was reported as "Dead", but only in Frontview. Raidar showed it as green with a red cross (I understand this means 'Inactive Spare'). Due to the email I received warning me about the "dead" disk, I ran a full USB backup job just in case (completed successfully), but restriping and then resyncing continued to 100%.

Now, Frontview reports volume status as X-RAID2/3 disks/Redundant with a capacity of 1.8TB. The disk reported by Frontview as Dead (and Raidar as Inactive), is now showing in Frontview as availble but not part of the X-RAID2 array, and Raidar simply shows it with a green dot (Normal).

I have hot-pulled the drive in question and hot-plugged back in, but aside from notification about disk removal and install, no further expansion activity is apparent. I'd rather not do a factory reset as I bought this device on the understanding Expansion would be 'easy', and what I am trying to do should be a regular use case.

In summary my questions are: -

1. Why did the expansion process originally incorporate 3 rather than 4 disks. And if it ignored a drive becasue it was "dead", why isn't my capacity now approx 4TB with 3 x 2TB disks in the X-RAID2 array rather than the reported 1.8TB?
2. Is there a way to 'force' the expansion process to recognise all disks and give me my expected (approx 6TB) capacity? I have already tried to reboot with the quota check option as suggested elsewhere in the forum.

Kind Regards,

Jason.

mdgm-ntgr · ‎2012-01-30

If a disk is dead redundancy cannot be rebuilt or expansion occur till it is replaced.

Could you hook the drive up to an internal SATA port in your PC and test it using SeaTools?

Please open a tech support (http://www.readynas.com/support) case and edit the thread title (i.e. subject of first post in the thread) to include it.

alcester · ‎2012-01-30

Thanks for prompt response. I have opened a ticket under case number above.

I do not have access to a PC to test drive, but I am not convinced it is dead. Aside from being <6 weeks old with relatively light use, reading elsewhere in the forums it seems other users have had reports of dead drives when there is nothing wrong with them. The drive is currently plugged in to the ReadyNAS in slot 1, shows Green in Raidar and Green/Available in Frontview.

Smart+ report from Frontview shows the following: -

Model: ST2000DL003-9VT166
Serial: **REDACTED**
Firmware: CC3C

Spin Up Time 0
Start Stop Count 209
Reallocated Sector Count 0
Power On Hours 2852
Spin Retry Count 0
Power Cycle Count 58
Reported Uncorrect 0
High Fly Writes 0
Airflow Temperature Cel 35
G-Sense Error Rate 0
Power-Off Retract Count 9
Load Cycle Count 211
Temperature Celsius 35
Current Pending Sector 0
Offline Uncorrectable 0
UDMA CRC Error Count 0
Head Flying Hours 137615047131583

...Although I think Flying Head Hours must be wrong!!

StephenB · ‎2012-01-30

Hopefully support can complete the expansion.

Some SMART parameters are used by vendors in a non-standard way (either using unusual data format, or sometimes to carry something else).

mdgm-ntgr · ‎2012-01-30

If an online case, attach the logs zip file (Status > logs > download all logs) you to your case.

alcester · ‎2012-01-30

Logs now attached to case...

mdgm-ntgr · ‎2012-01-30

While you wait for a response you may wish to try the "Disk Test" boot option: http://www.readynas.com/kb/faq/boot/how_do_i_use_the_boot_menu

Do also note that disks can and do fail at any time. You could well have a bad disk.

alcester · ‎2012-01-30

Thanks, I will try that.

However, I am still troubled by the fact that even if this one drive is faulty, I apparently have 3 other normal drives that are not correctly delivering their capacity in the XRAID2 configuration. To me this suggests that something else is wrong?

mdgm-ntgr · ‎2012-01-30

alcester wrote:

However, I am still troubled by the fact that even if this one drive is faulty, I apparently have 3 other normal drives that are not correctly delivering their capacity in the XRAID2 configuration. To me this suggests that something else is wrong?

Yes if this is the case there is something wrong. The dead disk needs replacing. Once it's replaced redundancy can be restored and then expansion can take place.

alcester · ‎2012-01-30

Things are getting more interesting...

While trying to get to boot menu (shutdown NAS and restarted holding in the Reset Button), I did not see any 'Boot Menu' message in the display, but I now see "Resizing xx%".

I will see what this does and report back when it is complete...

mdgm-ntgr · ‎2012-01-30

Yes, wait and see what happens.

To access boot menu you would shutdown NAS, then press and hold reset button, turn on NAS, keeping holding in reset button etc.

alcester · ‎2012-01-30

Resizing complete and now I have 3692 GB across 3 disks. My fourth disk (in slot 1) is still showing as Available in Volumes > Volume Settings, and OK in Status > Health, but still is not part of the XRAID2 array.

I will now try and kick off the Disk Test in the Boot Menu as previously advised.

PapaBear1 · ‎2012-01-30

alcester - are you by any chance using a Mac on your network?

alcester · ‎2012-01-31

MDGM - Managed to run the disk test from the boot menu last night. No errors seem to have been reported (I haven't found any log file generated?), and RAIDar and Frontview all show disks in 'Green' condition. Status is still 3 disk RAID with 1 disk 'Available'. I have updated my support ticket with this info. Support are currently looking at logs generated yesterday.

PapaBear - My wife has a Macbook Pro with Time Machine enabled on the ReadyNAS for backups. Aside from Time Machine, she only occasionally mounts shares for media etc.

StephenB · ‎2012-01-31

BTW, the green light in frontview does not mean the disk is healthy, you need to look at the SMART stats for that.

alcester · ‎2012-01-31

Hi StephenB,

Without access to a SATA PC to run any diagnostics, I have to rely on what the device reports. Do you know if there is any user readable output from the the Boot Menu Disk Test? That test ran last night for what looks like approx 5 hours, but without a log or some other output I have nothing to show for it other than Green lights in RAIDar and 'OK' in Frontview!

Update from the Support Case: -

Support requested I shut down device, remove disk, startup device (waiting for any Restripe/Resync to complete) and reinstall disk. I have done that and I end up with no change. There is no hint of a Restriping/Resyncing/Expansion taking place at any stage.

StephenB · ‎2012-01-31

I suggest you go into the "health" menu and pull up the SMART stats for each drive. Also, if you have not configured email alerts, you should.

BTW, my Pro 6 got into a similar state (provoked initially by a disk failure during expansion). I didn't call support, but in the end needed a reset to factory default. Hopefully support will be able to avoid that.

alcester · ‎2012-01-31

Hi Stephen,

I have looked SMART stats (earlier post contained output from Disk 1), but nothing in there suggests a failure to me. All 'Error Counts' are zero across all disks but disk 1 is the one not part of the X-RAID2 array at the moment. I also have email alerts set up and have received nothing relating to the Disk Test run last night, or any other disk failure. All other alerts about backup jobs, disk additions etc etc I am receiving as normal.

mdgm-ntgr · ‎2012-01-31

Then the disks probably passed long SMART tests. You could check the relevant log in the logs zip file downloadable via Status > Logs > Download all logs

StephenB · ‎2012-01-31

Since the SMART parameters are saved in the drives, any failures in the boot menu diag would degrade them. So zero error counts is good news.

I'd suggest letting the support process play out, if you do too much on your own you could interfere with them. Grabbing logs is a good thought.

What is your backup situation? Since a factory reset is possible, you should probably save your config to your PC and make sure you have backups of your critical data.

alcester · ‎2012-01-31

Hi mdgm,

The only log that seems relevant is disk_smart.log (is this the right one?). All 4 drives show substantially the same as disk 1 as follows (header info only): -

***** smartctl output for sda *****

smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     ST2000DL003-9VT166
Serial Number:    **REDACTED**
Firmware Version: CC3C
User Capacity:    2,000,398,934,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Tue Jan 31 08:54:38 2012 WET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

Is there anywhere else I can look for possible errors?

alcester · ‎2012-01-31

Hi StephenB,

I have a USB backup of my data updated at the weekend when I first got the disk fail message, and I have downloaded the Config. I should be able to restore OK (albeit taking 2 days), but surely that is not the point of a ReadyNAS with "automatic horizontal expansion via X-RAID2"!

Support have said that the logs they've seen show my ReadyNAS working 'perfectly', so It is looking to me like the next stop in their troubleshoot will be to blame disks and request a Factory reset... Hope I'm wrong.

alcester · ‎2012-02-01

Support suggested reformatting the errant drive in any PC. They suggested that the existing partitions were somehow preventing the disk being included in Expansion. As the partitions were created originally by the ReadyNAS, this seems like a bug to me.

Anyhow, as it stands I do not have access to a suitable PC to reformat and therefore cannot retify this solution. I have instead initiated a Factory Reset and lo and behold, all drives are now included in the X-RAID2 array providing me with the expected <6TB capacity. I just hope my USB restore works OK!

Thanks for all your help.

StephenB · ‎2012-02-01

alcester wrote:
Support suggested reformatting the errant drive in any PC. They suggested that the existing partitions were somehow preventing the disk being included in Expansion. As the partitions were created originally by the ReadyNAS, this seems like a bug to me.

Anyhow, as it stands I do not have access to a suitable PC to reformat and therefore cannot retify this solution. I have instead initiated a Factory Reset and lo and behold, all drives are now included in the X-RAID2 array providing me with the expected <6TB capacity. I just hope my USB restore works OK!

Thanks for all your help.

You might pickup up a SATA/USB adapter for future use. These have their limitations, but they certainly would have allowed you to delete the partitions.

Good luck on the restore.

alcester · ‎2012-02-02

Restore completed successfully (config+data). Phew!

I'm happy I've got to where I wanted to, but not with the way I got there. If there are common problems with the Expansion process (e.g. disks being erroneously reported as dead during process), I would expect the product documentation to provide workarounds or mitigations. As it stands, I will not trust any further automatic expansion and instead factor in a backup/factory reset/restore cycle.

Hopefully any further expansion is a long way off with the capacity I now have.

Thanks again for all comments and advice - gratefully received.

Re: Expansion Difficulties - XRAID2 - Case #17781064

Expansion Difficulties - XRAID2 - Case #17781064

Re: Expansion Difficulties - XRAID2 Ultra 4 Plus