Missing volume after hard reboot

I'm taking a look at the logs now. But you should remove them for Google Drive. There is your serial number in there.

You have 5 good drives (WD30EFRX), the 6th is a Seagate Desktop drive (ST3000DM001), that's a shame. 

It looks like this drive has shown a failure rate higher than normal:

https://www.backblaze.com/blog/3tb-hard-drive-failure/

https://superuser.com/questions/1037228/if-someone-purchased-a-seagate-st3000dm001-3tb-hard-drive-and-it-failed-sooner-t

https://www.extremetech.com/extreme/222267-seagate-faces-lawsuit-over-3tb-hard-drive-failure-rates

If it was my NAS, I would replace it immediately. It doesn't show any error, but has 25,000 hours and a terrible reputation.

Based on the logs:

You have daily snapshots on 8 shares (applications, backup, Documents, istat, Music, Pictures, Transmission, Videos).

No balance nor defrag were run since the creation of the volume, 2017/01/04.

I cannot see the metadata allocation because the data volume isn't mounted.

The logs don't give any details about the configuration of the shares in regards to Bit Rot Protection (implies Copy-on-Write) and Compression, can you tell us?

My guess is that the metadata allocation is high and the data fragmented which may lead to crashes. But the issue can be completely unrelated. It's not something I can explain from these logs. Unfortunately, after a reboot, the dmesg logs are flushed. It's possible to setup something like netconsole to push the kernel logs with a similar process to syslog in order to capture them on another machine, so you have the ones just before the crash.

Not sure why, but you have a cron job to capture the last status of processes and load every 5 minutes. Maybe in an attempt to debug the lock-ups?

The LAN interface seems to be connected to a FastEthernet network (100Mbps), that's weird.

It would be interesting to know to read the loadavg.log file, but I don't know how. (@mdgm maybe?)

The NAS was power-cycled twice recently:

- It seems that the NAS was power-cycled on Jun 18 15:01 (the logs start there so can't tell when it hung).

- It seems that the NAS crashed shortly after Jun 18 18:25 and was power-cycled on Jun 19 20:28. There didn't seem to be much going on before the crash, in terms of process anyway.

Power-cycles clearly don't help, but I understand you may not have had a choice.

The NAS remained then shutdown for nearly a month. When it booted last, the volume was unmountable.

That's when it failed to mount the data volume.

Jul 12 20:27:10 readynas01 kernel: BTRFS critical (device md127): corrupt leaf, slot offset bad: block=2291416563712, root=1, slot=77
Jul 12 20:27:10 readynas01 kernel: BTRFS error (device md127): failed to read block groups: -5
Jul 12 20:27:10 readynas01 kernel: BTRFS critical (device md127): corrupt leaf, slot offset bad: block=2291416563712, root=1, slot=77
Jul 12 20:27:10 readynas01 kernel: BTRFS error (device md127): failed to read block groups: -5
Jul 12 20:27:10 readynas01 kernel: BTRFS critical (device md127): corrupt leaf, slot offset bad: block=2291416563712, root=1, slot=77
Jul 12 20:27:10 readynas01 kernel: BTRFS error (device md127): failed to read block groups: -5
Jul 12 20:27:10 readynas01 kernel: BTRFS critical (device md127): corrupt leaf, slot offset bad: block=2291416563712, root=1, slot=77
Jul 12 20:27:10 readynas01 kernel: BTRFS error (device md127): failed to read block groups: -5
Jul 12 20:27:10 readynas01 kernel: BTRFS critical (device md127): corrupt leaf, slot offset bad: block=2291416563712, root=1, slot=77
Jul 12 20:27:10 readynas01 kernel: BTRFS error (device md127): failed to read block groups: -5
Jul 12 20:27:10 readynas01 kernel: BTRFS error (device md127): open_ctree failed
Jul 12 20:27:10 readynas01 mount[1465]: mount: wrong fs type, bad option, bad superblock on /dev/md127,
Jul 12 20:27:10 readynas01 mount[1465]:        missing codepage or helper program, or other error
Jul 12 20:27:10 readynas01 mount[1465]:        In some cases useful info is found in syslog - try
Jul 12 20:27:10 readynas01 mount[1465]:        dmesg | tail or so.
Jul 12 20:27:10 readynas01 systemd[1]: data.mount: Mount process exited, code=exited status=32
Jul 12 20:27:10 readynas01 systemd[1]: Failed to mount /data.

That's when you usually see many reboot attempts in the log as panic grows, but it doesn't seem you tried to reboot the NAS at all.

I would change the fstab from:

LABEL=33ea999f:data /data btrfs defaults 0 0

to:

LABEL=33ea999f:data /data btrfs defaults,ro,recovery 0 0

and try to reboot the NAS gracefully from the GUI or rn_shutdown -r

After reboot, if the data volume mounts OK, update your backups immediately.

Please then give the ouput of this command:

btrfs fi us /data

After your backups are complete (AND inspected!!!), you'll have to recreate the volume and reimport the data.