Lost Volumes following stuck Reboot

Didn't help. Pl. see attached

The logs should be different than before, since the RAID volume should be created now that the event counters have been forcibly synced.

Are you seeing the same superblock issue in the ReadyNAS logs, or you seeing something else?

There are a few issues:

1. Your root volume is full. There's no point running a balance if the space is actually used up. You need to free up space on the root volume first.

2. The data volume won't mount

Jul 20 11:31:28 ReadyNAS64 kernel: BTRFS: device label ffffffffa8c00f00:data devid 1 transid 548436 /dev/md127
Jul 20 11:31:39 ReadyNAS64 kernel: BTRFS info (device md127): has skinny extents
Jul 20 11:31:49 ReadyNAS64 kernel: BTRFS error (device md127): qgroup generation mismatch, marked as inconsistent
Jul 20 11:31:49 ReadyNAS64 kernel: BTRFS error (device md127): parent transid verify failed on 6765785743360 wanted 548438 found 548430
Jul 20 11:31:49 ReadyNAS64 kernel: BTRFS error (device md127): parent transid verify failed on 6765785743360 wanted 548438 found 548430
Jul 20 11:31:49 ReadyNAS64 kernel: BTRFS warning (device md127): failed to read log tree
Jul 20 11:31:49 ReadyNAS64 kernel: BTRFS error (device md127): open_ctree failed
Jul 20 11:31:49 ReadyNAS64 start_raids[2072]: Scanned Btrfs device /dev/md/data-0

Unfortunately the volume is not mounted. After you suggested to reboot I checked and the volume did not come up. If I go to NAS web page I get a message no volume exists.

Thanks for your guidance mdgm. Hope you got the detailed logs that were sent by email. I guess I am still confused how the Root FS got full and is not clearing up even after lots of deleted files. If I run the below I do not see any suspicious files (log attached)

root@ReadyNAS64:/# find / -type f -atime -90 -size +500k -ls

I could only comprehend the Amazon Drive backup that started to write to /data/... directory which by virtue of un-mounted /data FS volume wrote to /. All that has been deleted already still I cannot see the original volume mounting automatically

root@ReadyNAS64:/# cat /etc/fstab
LABEL=0a433d38:data /data btrfs defaults 0 0

Please see attached log

---

mdgm-ntgr wrote:

 

1. Your root volume is full. There's no point running a balance if the space is actually used up. You need to free up space on the root volume first.

 

---

This info suggests that it has about a gigabyte of free space.  What am I missing?

root@ReadyNAS64:~# btrfs fi df -h /
Data, single: total=2.84GiB, used=2.60GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=128.00MiB, used=23.09MiB

The challenge I am facing is to decide what more to delete. I agree the OS FS should ideally have nothing other than the OS & apps. The fact that Amazon Drive backup landed on root FS because of un-mounted /data (still a mystery) has been addressed. Files written in the /data (local) under Amazon Drive folder created have also been deleted. So my struggle now is in deciding what else I can delete and why am I not getting teh space back. Afterall I really don't have a lot to play with. From ~4gb size of root I have already cleared ~1gb. Shouldn't that be sufficient to enable the mounting of /data volume as specified in /etc/fstab.

Why does an attempt to manually mount the /data volume that has all of my data not getting mounted. I could accept to forego all of the NAS configurations and start fresh (as I have most of the backup) except for some files that haven't been replicated to the secondary NAS before the failure occured.

Your time and guidance is very much appreciated.

Also, from the log of files from last 90 days with size over 500K that I attached, if anything looks like it can be deleted please advise so

Trying to dig through the directories to find files that are big, I see a lot of space being consumed by this file. At this time I am running out of options, OK to delete the file and gracefully reboot the NAS? See attached PDF as the message was truncated.

Attached is the current view from web page and consumption below:

root@ReadyNAS64:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             10M  4.0K   10M   1% /dev
/dev/md0        4.0G  2.7G  1.1G  72% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G  3.2M  1.9G   1% /run
tmpfs           961M  1.4M  960M   1% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
root@ReadyNAS64:~# btrfs fi df -h /
Data, single: total=2.84GiB, used=2.63GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=128.00MiB, used=22.81MiB
GlobalReserve, single: total=16.00MiB, used=0.00B

I think it's pretty clear that you need to do some repair on the data volume's BTRFS file system.

That's difficult to do manually.  You could use Netgear's data recovery service, or if you can connect the disks to a Windows PC you can use ReclaiMe software.

Or you can try some of the btrfs filesystem repair features - that's not something I've tried myself, so I don't have any advice on the safest path there.  If you do continue to try and fix it on your own (and do the wrong things) then you can make it impossible for Netgear (or recovery software) to repair the filesystem.

Getting support from Netgear past 90 days is out of question. Most of our efforts until now have been in balancing the Root FS. The NAS however does come up now cleanly (except the /data volume) and the WebPage says no volume is configured although I know it was. Is it possible to fsck the /dev/md127 volume? Assuming we can mount the volume once, I can extract the data and reset the NAS to factory default and restore everything from backup?

root@ReadyNAS64:~# mount -a
mount: /dev/md127: can't read superblock

root@ReadyNAS64:~# cat /etc/fstab
LABEL=0a433d38:data /data btrfs defaults 0 0

root@ReadyNAS64:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             10M  4.0K   10M   1% /dev
/dev/md0        4.0G  2.8G  1.1G  74% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G  3.2M  1.9G   1% /run
tmpfs           961M  1.5M  960M   1% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup

root@ReadyNAS64:~# btrfs fi df -h /
Data, single: total=3.25GiB, used=2.67GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=128.00MiB, used=22.77MiB
GlobalReserve, single: total=16.00MiB, used=0.00B

It can't read the superblock which is a serious problem.

Unless you know what you're doing you're going to need to try contacting support and asking them if they can make a data recovery attempt (there are costs involved with this) or 3rd party data recovery software or a 3rd party data recovery service. Data recovery attempts can be completely unsuccessful.

Requesting to close this thread. Last week with support from Netgear, the FS was mounted and successfully retrieved all data. Netgear was very professional and worth every penny I spent. Gracefully no data recovery options was needed. I requested and wanted to post here but Netgear would not share the actions leading to recovery. 

Upon asking if I caused the problem by force rebooting the NAS I was advised the following.

"File system errors usually occur due to improper shutdown, full root OS, full Volume usage and other disputes on the volume due to too much fragmentations and unbalanced volume. When its on booting sequence, if possible do not force shutdown, you can just boot it into read only"

---

pgovil wrote:

When its on booting sequence, if possible do not force shutdown, you can just boot it into read only"

---

That of course is correct advice (when it is possible).