"No volume exists" & altered disk partitions #23786292
Hello,
I have a ReadyNAS Pro with 6x2TB drives. Other than a failed drive in 2012, everything has run well since purchase in 2011.
In late August, I tried accessing the NAS and discovered that /dev/md2 was gone and contacted support (case #23786292) and was told a data recovery contract would be required. Before proceeding, I did some additional investigation and found that each disk's partition map had been altered from having 3 partitions (from an earlier expansion.log):
nas:/# sgdisk -p /dev/sda Disk /dev/sda: 3907029168 sectors, 1.8 TiB Logical sector size: 512 bytes Disk identifier (GUID): 145A7DAB-C8DD-4E5A-A1C9-7BF265FA1C6A Partition table holds up to 128 entries First usable sector is 34, last usable sector is 3907029134 Partitions will be aligned on 64-sector boundaries Total free space is 3897591917 sectors (1.8 TiB)
Number Start (sector) End (sector) Size Code Name 1 64 8388671 4.0 GiB FD00 2 8388672 9437247 512.0 MiB FD00
No factory reset was performed, and from a user or admin perspective, the only activity was accessing data on the NAS. What would cause partition maps to be changed, especially without any user/admin intervention?
Looking back through the expansion log, it doesn't seem that backups of the partition maps were successfully written due to an OS error:
get disk /dev/sde format is (GPT=2,MBR=1,MX=3,MISC=-1): 2 [2012/12/29 13:55:15 28678] LINE 7839: exec command: sgdisk -p /dev/sde | grep '[0-9] ' > /var/log/frontview/.known_cfgdir/MN3220FA060J0E [2012/12/29 13:55:16 28678] Sgdisk /dev/sde partition to file Error. rc = -1 [2012/12/29 13:55:16 28678] /var/log/frontview/.known_cfgdir/JK11A8B9H7VE9F
and all files in /var/log/frontview/.known_cfgdir/* are empty. In a properly working ReadyNAS Pro, should this be the case?
Frontview status reports all disks as "OK" with green lights. If this were a matter only of the md2 volume missing, I think the next step would be trying to reassemble the RAID, but I'm concerned that the disk partitions were overwritten and am not sure if rewriting/restoring the partitions to their previous state will allow md2 to be reassembled or not.
No backup. I'm considering dd copy of each disk to a backup disk before proceeding.
Any advice on next steps would be greatly appreciated.
Re: "No volume exists" & altered disk partitions #23786292
We do recommend making regular backups. No important data should be trusted to just a single device.
Yes, a data recovery contract is required in a situation like this.
The Pro Business Edition has a 5 year limited hardware warranty.
If you try to fix this yourself you could only succeed in making it harder still to try to recover your data.
We overwrite the partition table on RAIDiator 4.2.x e.g. when expanding a volume. Backups are kept of the partition table.
Cloning disks can be useful, so that if something goes badly wrong with a command used to try to recover data all hope is not lost (of course you need to be careful to get the cloning right).
Re: "No volume exists" & altered disk partitions #23786292
Of course, backups rule! But alas, no current backup here.
Yes, I completely understand the official ReadyNAS answer is that a data recovery contract is required. I'm also seeking explanations from those more experienced for what triggered a blind rewrite of each disk's partition table, where partition tables are backed up to, where these types of activities are logged, etc.
Very true, there's risk with everything. In the interest of due diligence, I'm seeking ideas and advice from Jedis/experts who frequent these forums and who have seen similar problems before taking any further action. I've seen people get through difficult situations on these forums because those more experienced and knowledgable were willing to lend a patient hand.
That may be true, but the volume wasn't being changed or expanded, no hardware was changed/failed. Each disk's partition table changed from having 3 partitions to only 2, effectively deleting the /dev/md2 volume.
After combing through the remaining root filesystem, I can't locate a backup of the partition tables anywhere. Where are partition tables backed up to?
Better safe than sorry with a block or bit-level copy, especially since there's no guarantee data recovery will work.
Re: "No volume exists" & altered disk partitions #23786292
It's been a while since this was posted but I experienced the same problem and managed to resolve it. Posting my experience just in case someone else turns up here with the same problem.
After my ReadNAS 6 Ultra became unresponsive I was stupid enough to give it a hard reboot. After the reboot I received six nice little 'New Disk Found' e-mails from the system, one for each of the six disks that had been in the system for over 18 months already. Feeling these mails couldn't mean anything good I logged into Frontview which reported that no volume was found. Upon closer inspection I found only two partitions on each of the six disks where there used to be three.
As no new partitions were created I felt confident that the data was still intact inside the 'free space'. I figured simply finding some way to restore the partition table was going to bring the /dev/md2 array back.
Initially I used TestDisk to try and restore the partitions but I think it was not aligning the partitions correctly when restoring them as it seemed the resulting partitions were smaller than before (and mdadm was not having it!). In the end I recreated the partitions using sgdisk using a sector alignment value of 8. The command I used was:
sgdisk -a 8 -n 3:9437256:3907029134 /dev/sd_
Where 9437256 was the first available start sector in 8 sector alignment after the second partition. 3907029134 was the last usable sector as reported by sgdisk -p
After this I rebooted the system (it seemed necessary to inform the kernel about the new partition table) and assembled the partition using
mdadm --assemble --scan
I mounted the volume and it looks like everything is working just fine again. I'm currently copying everything off the NAS as I feel I can't really trust it after this manual tinkering.
As for the cause: during my initial investigation I found that /dev/md0 had run out of inodes due to a faulty cronjob creating a new file every 5 minutes. All the partition backups in /var/log/frontview/ were 0 bytes long... I can't be 100% sure what chain of events followed that caused one out of three partitions per disk to go missing but I am just going to assume inode exhaustion was the root cause.
Re: "No volume exists" & altered disk partitions #23786292
Yes, you've found a way to fix your system.
However note that it can be more complicated than this. If expansion has taken place there will be more than 3 partitions. There's also factors of on which firmware the disk was added. If it was added on very old firmware the steps will be different to the disk being added on fairly recent firmware as we have made changes over time to how we do things. So I would strongly recommend that users encountering a problem like this clone disks before trying to fix it themselves if they really want to try. What I would recommend is that users in this kind of situation contact support and consider the paid recovery attempt options.
Usually the partition table backups are a big help. It would be advisable to download the logs zip file after each time you e.g. replace a disk. That way if the OS partition does fill up for some reason you have what the partition table should look like in the logs you have downloaded.
Did you create the faulty cronjob? Filling up the OS partition certainly sounds like the root cause.
Re: "No volume exists" & altered disk partitions #23786292
I agree. I had backups of any truly important data on the NAS so I did go into it with a win-all or lose-all attitude.
The disks were added over time so they were added using different versions of the firmware but I do not remember which. They were all of equal size so no expansion was done. I'm planning to do a factory reset of the device before putting any data back on it.
It was a small cronjob I created myself to do a wget send a small update to a dyndns service. I had forgotten the --spider parameter so every 5 minutes the result of the request was written to disk. There was plenty of 'space' left on the disk but with 50k free inodes those took little under half a year to all get used up.
Even though I created that situation myself I still feel that space/inode exhaustion of the OS partition could have been handled a little more gracefully by the netgear software. I assume some script in the 'disk management process' relies on to ability to write and read a file to disk. It should check if that actually worked at all before doing something as destructive as writing a brand new partition table to disk...
