NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Solved
Volume scan failed to run properly #26188803
I have a ReadyNAS Ultra 4 with four disks in X-RAID 2. One disk had several ATA errors and I replaced the old disk 1.5 TB with a new 4 TB disc. After reboot the NAS started with sync, and after about...
Hallelujah!
I have now managed to get my data volume back up! Unfortunately, Netgear L2 tech did not manage and I had to do this by myself.
As promised I will give you the results. Note that I am not a trained IT-engineer, so I got this from Google search and dedication. If you decide to do this it is on your own risk - I take no responsibility that it will work on your system.
I had only 4 sata-ports on my PC, and had to install Linux on a USB in order to get all 4 disks connected. I wanted to use Knoppix Linux, but I am sure most Linux versions would do.
- Get Linux installed on the computer:
I installed Knoppix Linux to a USB-stick. This was not trivial as I had a new Z170 motherboard and a regular USB-boot would not work, using Universial-USB-installer-1.9.6.3 or unetbootin-windows-613. I ended up attaching a sata DVD and burned a Knoppix DVD, booted from the DVD and installed Knoppix on the USB.
- Removed the DVD-RW drive and attached all the 4 NAS-drives to the PC. Booted up in Knoppix (USB) and started to see if I could access the drives. I noticed that Knoppix displayed one mounted volume and a few volumes that was not mounted (these volumes turned out to be LVM physical volumes). I could access the files on the mounted RAID volume which turned out to be the NAS OS.
- I performed several different commands after some Google search, being careful not to run anything that could make changes to the drive, in case I needed to perform some data recovery.
First I checked the partition tables on all four drives using gdisk:
knoppix@Microknoppix:~$ sudo gdisk /dev/sda
They were identical, not shown as I did this in four different windows.
Then I wanted to checked the raid setup and ran:
knoppix@Microknoppix:~$ sudo mdadm --detail --scan ARRAY /dev/md/4 metadata=1.2 name=A021B7C18D0C:4 UUID=d6301b60:0ce2f767:558c574f:db007ccb ARRAY /dev/md/1 metadata=1.2 name=A021B7C18D0C:1 UUID=d2791ec8:5adda84e:c7463c2e:c0f2016b ARRAY /dev/md/0 metadata=1.2 name=A021B7C18D0C:0 UUID=a218f0a3:1b607e2e:953b087b:04ed9c99 INACTIVE-ARRAY /dev/md3 metadata=1.2 name=A021B7C18D0C:3 UUID=5aa62eb3:fa4e39b8:213486da:d587542d ARRAY /dev/md/2 metadata=1.2 name=A021B7C18D0C:2 UUID=829ccffc:55683ba6:36bb7959:6eed3523
From this I figured out there was an inactive array md3.
Then I used e2fsck to check the partition:
knoppix@Microknoppix:~$ e2fsck /dev/md3 e2fsck 1.42.13 (17-May-2015) e2fsck: Invalid argument while trying to open /dev/md3 The superblock could not be read or does not describe a valid ext2/ext3/ext4 filesystem. If the device is valid and it really contains an ext2/ext3/ext4 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 8193 <device> or e2fsck -b 32768 <device>
This made me think there was a problem with the superblocks on the partitions, that turned out not to be important. Searching and looking for answers I decided to stop the array and start it again:
knoppix@Microknoppix:~$ sudo mdadm --stop --scan knoppix@Microknoppix:~$ sudo mdadm --assemble --scan mdadm: /dev/md/4 has been started with 2 drives (out of 3). mdadm: restoring critical section mdadm: /dev/md/3 has been started with 4 drives. mdadm: /dev/md/2 has been started with 4 drives. mdadm: /dev/md/1 has been started with 4 drives. mdadm: /dev/md/0 has been started with 4 drives. mdadm: Found some drive for an array that is already active: /dev/md/4 mdadm: giving up.
Then used lvmdiskscan to see if I could see the volumes and if there was a problem with any of them :
knoppix@Microknoppix:~$ sudo lvmdiskscan /run/lvm/lvmetad.socket: connect failed: No such file or directory WARNING: Failed to connect to lvmetad. Falling back to internal scanning. /dev/ram0 [ 4.00 MiB] /dev/md0 [ 4.00 GiB] /dev/ram1 [ 4.00 MiB] /dev/md1 [ 1023.88 MiB] /dev/ram2 [ 4.00 MiB] /dev/md2 [ 4.08 TiB] LVM physical volume /dev/ram3 [ 4.00 MiB] /dev/md3 [ 931.50 GiB] LVM physical volume /dev/ram4 [ 4.00 MiB] /dev/md4 [ 3.64 TiB] LVM physical volume /dev/ram5 [ 4.00 MiB] /dev/ram6 [ 4.00 MiB] /dev/ram7 [ 4.00 MiB] /dev/ram8 [ 4.00 MiB] /dev/ram9 [ 4.00 MiB] /dev/ram10 [ 4.00 MiB] /dev/ram11 [ 4.00 MiB] /dev/ram12 [ 4.00 MiB] /dev/ram13 [ 4.00 MiB] /dev/ram14 [ 4.00 MiB] /dev/ram15 [ 4.00 MiB] /dev/sde1 [ 4.46 GiB] /dev/sde2 [ 24.82 GiB] /dev/sdf1 [ 4.46 GiB] 0 disks 21 partitions 0 LVM physical volume whole disks 3 LVM physical volumes
There was 3 volumes listed. Followed up with lvdisplay to see the logical volume:
knoppix@Microknoppix:~$ sudo lvdisplay /run/lvm/lvmetad.socket: connect failed: No such file or directory WARNING: Failed to connect to lvmetad. Falling back to internal scanning. --- Logical volume --- LV Path /dev/c/c LV Name c VG Name c LV UUID DHaiSO-OE5j-wbTe-rW1L-Zh1L-DNFP-vbPjvA LV Write Access read/write LV Creation host, time , LV Status NOT available LV Size 6.80 TiB Current LE 111404 Segments 3 Allocation inherit Read ahead sectors auto
From here I assumed the volume c was not available. Followed up with lvscan:
knoppix@Microknoppix:~$ sudo lvscan /run/lvm/lvmetad.socket: connect failed: No such file or directory WARNING: Failed to connect to lvmetad. Falling back to internal scanning. inactive '/dev/c/c' [6.80 TiB] inherit
Hmm. The data volume (c) was inactive. Now, I had previously tried to activate the array using mdadm --detail --scan. I searched the web further and came across this site/post that solved the case: http://pissedoffadmins.com/os/mount-unknown-filesystem-type-lvm2_member.html
knoppix@Microknoppix:~$ modprobe dm-mod knoppix@Microknoppix:~$ sudo vgchange -ay /run/lvm/lvmetad.socket: connect failed: No such file or directory WARNING: Failed to connect to lvmetad. Falling back to internal scanning. 1 logical volume(s) in volume group "c" now active
Voila! The volume came up and I then managed to mount it! I put all the disks back in the Netgear NAS and it booted normally. I am now transferring files to the other backup Netgear NAS as we speak. I guess this will take a bit. Also the 4th disk is now resyncing.
Sat Jan 30 17:04:37 CET 2016 System is up. Sat Jan 30 17:04:37 CET 2016 Volume C is approaching capacity: 88% used 878G available Sun Jan 17 12:15:59 CET 2016 System is up. Sun Jan 17 12:15:59 CET 2016 The paths for the shares listed below could not be found. Typically, this occurs when the ReadyNAS is unable to access the data volume. Squeezeboxserver Documents Video media Photos Music Sun Jan 17 12:15:41 CET 2016 Volume scan failed to run properly.
I hope this can be useful for others, including the L2 Netgear support, which in my opinion should have been able to address this issue in the first place. Not letting me go searching around the web for possible solutions. If I am able to figure this out (though I have a PhD in genetics, and have been around computers for 25 years) an engineer at Netgear definitely should have fixed this easily. This in my point qualify for a refund! Also, that Netgear does not log their service to provide proof/documentation of their work is surprising.
I am happy I figured it out, and hope this can be useful for someone else in a similar situation.
Did you try the boot menu option to "skip volume check"?
Thank you StephenB for your suggestion. Unfortunately, I still have no volume on the NAS. I have tried the following with no solution to the problem: Replace the old (dead) drive and restart the NAS. Remove the faulty drive and run with only 3 drives (drive 4 removed) Replaced the new updated firmware with the previous one (that was working before the "crash") Rebooted with the option of "reinstall OS" from the boot menu as suggested by Netgear Tech Support. Rebooted with the option of "skip volume check" from boot menu. I assume I have to pay Netgear using their "Data recovery contract", a 155 Euro no guarantee attempt to fix the problem using Telnet. I will contact them tomorrow. If this solves the problem I am not sure I have access to what the Netgear Tech performed on the unit. Hence, I will not be able to provide any help for other users. Anyway, I will post the outcome here.
You can potentially start with per-incident support (the data might be intact if they can mount the volume). That is cheaper than data recovery.
Ok, time for an update.
Tech support has come back to me and asked for the log files, which I have attached to the case.
After looking at it I have been asked to remove the drive in slot 4 and the volume should be available. Unfortunately, I have tried that already, and this is not solving the problem. I hope that my 1 hour support is not spent without someone logging into the NAS.
I have also removed the three functional drives from the NAS, attached them to my computer, and run "ReclaiMe" software. All files/folders seems to have been "identified" in the software. I can only recover the files if I purchase the software ($200). :smileysad:
Hypothesis:
During the sync and resizing of the volume, the NAS was rebooted and FW was automatically updated, possibly causing the start and end blocks of each volumes on the drives being alterd (LVM header?) due to volume expansion, and these changes not being updated with mdadm (this might be speculative as I am not fully aware of the details here).
Would it be possible to use a software to scan the drives, identify the start-end blocks, verify these with LVM header/mdadm to see if they are the same? If not, make changes so mdadm, or somewhere else, so that the volume will be identified again in my NAS?
Alternative solutions:
Are there any Linux software that can be used to scan three out of four disks in an X-RAID 2 to obtain the files? If not, am I only stuck with Level 3 Netgear Tech Support (~€150) or a commercial Windows software e.g. ReclaiMe at $200?
I will keep updating this thread until I have resolved the case. Again, thanks for any suggestions you might have.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
