Volume scan failed to run properly #26188803
Accepted Solutions
All Replies
Ok, time for an update.
Tech support has come back to me and asked for the log files, which I have attached to the case.
After looking at it I have been asked to remove the drive in slot 4 and the volume should be available. Unfortunately, I have tried that already, and this is not solving the problem. I hope that my 1 hour support is not spent without someone logging into the NAS.
I have also removed the three functional drives from the NAS, attached them to my computer, and run "ReclaiMe" software. All files/folders seems to have been "identified" in the software. I can only recover the files if I purchase the software ($200). 
Hypothesis:
During the sync and resizing of the volume, the NAS was rebooted and FW was automatically updated, possibly causing the start and end blocks of each volumes on the drives being alterd (LVM header?) due to volume expansion, and these changes not being updated with mdadm (this might be speculative as I am not fully aware of the details here).
Would it be possible to use a software to scan the drives, identify the start-end blocks, verify these with LVM header/mdadm to see if they are the same? If not, make changes so mdadm, or somewhere else, so that the volume will be identified again in my NAS?
Alternative solutions:
Are there any Linux software that can be used to scan three out of four disks in an X-RAID 2 to obtain the files? If not, am I only stuck with Level 3 Netgear Tech Support (~€150) or a commercial Windows software e.g. ReclaiMe at $200?
I will keep updating this thread until I have resolved the case. Again, thanks for any suggestions you might have.
Update: 29.12.2015
Tech support has asked me to boot the NAS into "tech support mode" this afternoon. I assume this opens up the NAS to the standard "tech support mode" root login? I feel a little bit worried to leave the unit with this open access for too long. I hope support will be using the "tech support mode" during the day tomorrow. Does anyone know how long I will have to leave the NAS in this mode? I have been waiting since the 18th of December after I paid for Tech Level 2 support, and we are only a few days away from New Year. I understand that Tech Support also has Christmas Holidays so I am not complaining here - just have had no access to my files for more than 12 days now.
I will ask tech support to provide me with their log file after they have accessed the NAS. I assume I will be given all the logs for the session they have while it is in "tech support mode"? Anyone gotten log files after a remote session?
Or, will the NAS store all communications during the remote session for me to use for later troubleshooting?
I think the log is useful for me because:
1. I have documentation of what has been done to the unit.
2. What did work or what did not work.
3. What will not be needed to try out in the future if Level 2 tech support does not work.
Well, I will keep the forum updated with the development.
@StephenB wrote:tech support mode would allow telnet access on your local LAN, but that is normally blocked by your router over the internet.
It also allows remote access by Netgear (though they need an access code that you provide them in order to do that)
Ok, but I am sure someone can exploit this, and that is why I am reluctant in keeping it in this mode for too long.
I have not yet been told to give them a code, so I guess they will not access it this year ...
I promised to keep this post updated, but I am sorry to inform that I do not have much more information to update with yet.
Tech Support has not asked for an "access code" but have accessed the NAS without.
The NAS seems to still be in sync process after 9 days, at least that is the last response I got from Tech Support.
The sync process seems to be very very slow process, but I will keep this post updated when I get more information. Tech Support is not sure if the data is ok yet, I guess they will have to wait for the sync process to finish. Myself, I am puzzled that the sync process can take this long, but I have no possibility to monitor the status myself as the NAS is still in "debug mode" and has been so for the last 15-16 days. 
UPDATE!
I got feedback from Tech Support the 15. of January:
Tech Support wrote 15. january 2016
Hello Jim,
I spoke to my colleague who was working on the NAS.
He explains the issue seems to either be caused at the time the one disk failed, or during the sync when the larger disk was used as a replacement.
It may have tried to reshape the RAID and this has caused the issue. At present it is not something we can fix at level 2.
To escalate this issue to level 3 you would need to purchase a data recovery contract.
The contract is around 155euro.
But it is not clear if the data can be recovered. It is hard to estimate but my colleague who looked at the RAID said he thinks there is about a 50/50% chance.
Please understand its very hard to estimate, even with the data recovery and level 3 involvement there is a chance no data can be recovered.
Due to the nature of disks, raid and storage in general it is not something to which a guarantee can be given.
Let me know if you are interested in the data recovery contract or if you have any questions.
The NAS is still missing the volume. How could L2 tech support start a sync process if the volume was not fixed?
I have connected the drives back to my computer and run the recovery software today. ALL FILES are GONE! The drives can not be assembled in ReclaiMe. I have also tried "NAS data recovery" from Runtime to no awail. I am so utterly upset now that I can hardly sit still. Almost 7 TB of data seems to be gone after Netgear Support L2 initiated ("forced"?) sync on the NAS. I should have recovered the data while I had the chance using the recovery software, but I put the trust in tech support. Now, tech support suggest to pay 155 euro for L3 to have a look at it.
I have requested all log files to be handed over to me as agreed upon with L2 support when I accepted the terms, all SSH commands/script used during the 15+ days it was in "debug mode". Maybe there is something that can be fixed after I get this data.
Anyone with suggestions of what to do next?
Thank you mdgm for making time to comment this thread. I honestly appreciate what you and others do by participating in the discussion forum. Further, I respect your opinion and can live with the fact that we might not see things the same way. I have shared my experience, the way I experienced it. You might criticize me, which you of course are allowed to do. Given this is the first comment to this thread something like “I am happy to see that you managed to get you data back” from you would warm my heart a bit before you start a rather long criticism of me and my wrong doing. No hard feelings ... 
@mdgm wrote:You purchased a per incident support contract and support advised they would need a data recovery contract in place to escalate this to L3 the day after the case was opened. When you declined to purchase it they continued to provide what support they could.
I honestly did not know what type of support existed when I called Netgear. It was suggested by a forum member to check with Support if a “pay-per incident” would be appropriate – which I did over the phone. Netgear Tech Support approved over the phone that this would be a good first go at the problem before they charged my credit card. Again I appreciate that you take time to go through the correspondence, but maybe you do not know the story as well as I do. Remember that I have been part of this experience since mid-December. Besides, even if we indeed were equally involved in the case we might end up with a different experience – which is a normal human response. However, I do not accept that I “declined” L3 support as Tech Support agreed that a “pay-per incident” would be a good start. If not, I would not pay 60 euro for nothing.
Data recovery attempts inherently may be unsuccessful. It's important to backup your data if you value it. No important data should be stored on just the one device, no matter which device that is. See Preventing Catastrophic Data Loss
Absolutely, and I cannot agree more. That is why I had a dedicated NAS (identical), and still has, to perform backup using rsync. If you read carefully, you would see that I indeed had that in place. Also, some of my “very” important data had a backup on the cloud (not mentioned as I did not see it very relevant at the time).
L3 support handles data recovery cases. It is not something which L2 is trained to perform. If you don't know what you're doing when attempting data recovery you can make things worse. I can see some strange commands in your list of commands I see you attempted to run a filesystem check on one of the raid layers even though there is a layer between the raid layers and the filesystem. In this case that command shouldn't have done any damage as it recognised that their wasn't an EXT filesystem directly on the raid layer, but with other commands you can do damage.
Again, thank you for bringing this up. I believe you are right. I have had the intention while posting on the board to be transparent – my intention is to get feedback to help me and others. And, you have made me aware that using the e2fsck command is not trivial and should be avoided in this case.
What still puzzles me, and maybe you can clear that up for me is what are L2 engineers trained to do? I was told that they could remotly log in make changes to see if they could get the volume up. Is this wrong?
As you know, since you have gone through the correspondence, L2 engineer has synced the partitions from one disk in the array to a new disk. The new disk was a part of an expanding array. In your opinion that is a job that L2 is qualified to do, right? However, it is not expected that they should be able to run lvdisplay and lvscan to investigate a problem with the volume? Followed up with running modprobe and the vgchange? You see, I do not know these things – and I believe my disappointment with Netgear Support, and don’t take this personally, comes down to communication. From my communication with Tech Support over the phone and in writing I was under the impression that the above diagnostic could be expected from L2 – hence I paid and now I am disappointed. Do you think it is wrong to express this disappointment on the forum?
Now you may get fortunate and find that blindly entering commands you've found on the web works fine, but then again you may not.
Now, I am sure you do not suggest that I blindly entered commands and by chance managed to get the volume back up? If so what do you think would be the odds for that to have a happy ending?
Your logs attached to the case show that one of your disks failed very badly.
If not in April, certainly by the alert in September it should have been clear that the disk needed replacing or at the very least that it was advisable to update your backup. A common cause of getting into data recovery situations is leaving a disk in the NAS that needs replacing for a long time after it should be replaced.
It is ok to look back and be wise – sometimes I do that. However, I did not know of the problems with the disks in April and not in September. I should, but i did not. When I did find out, that’s when the real trouble started, right? Again, this is something I learned from. I did not have email warning, which I should. I learned from this incident, which is a philosophy of mine. I make mistakes, and I learn from them. I gladly share my mistakes for others to learn from them, even though it might make me look silly.
Finishing your comment with what I failed to do to put myself in this situation in the first place - is hardly constructive. Remember, I managed to get my volume up and save my 6.8 TB of data something L2 Tech did not. TATA - a celebration for me! Even if you feel that everything is my fault with the outcome in this case it still does not change the fact that I did indeed have a bad experience with Tech Support. I believe that this comes down to communication and my expectations as a paying customer. My expectations, which I got from my dialog with Tech Support, were not met. I have learned from this, hopefully so did Tech Support. And maybe also some of the many forum members, if they visit this thread. 
