Missing volume after power failure

F_L_ · ‎2017-03-14

Thank you for your answer.

I must admit that my knowledge about raids is limited and I thought it was unlikely that one disc could cause something like this. If this now is the case..

I know the importance of backup and I have some.

For now I just want to see what my options are.

If/how I can retreive some of the data, without sending discs to Netgear, or if I should consider the data gone.

If the data is lost, should I execute a factory reset and reinstall the NAS?

mdgm didn´t say which disc that was faulty but I´m assuming the oldest one (the 2tb)

jak0lantash · ‎2017-03-14

@F_L_ wrote:
I thought it was unlikely that one disc could cause something like this

It is unlikely, but it's possible.

StephenB · ‎2017-03-14

@F_L_ wrote:

mdgm didn´t say which disc that was faulty but I´m assuming the oldest one (the 2tb)

PM him and ask. It might not be the oldest. If you have a Windows PC you can also test the disks yourself (seatools for seagate, lifeguard for western digital). Label the disks by slot number, so you will be clear on the order.

@F_L_ wrote:

For now I just want to see what my options are.

Netgear's recovery service is generally done remotely, though it is possible that they might need the NAS or the disks shipped. It is expensive (these services generally are): https://kb.netgear.com/69/ReadyNAS-Data-Recovery-Diagnostics-Scope-of-Service?cid=wmt_netgear_organi...

Some users have had good luck with ReclaiMe data recovery software. The benefit there is that you can see what it can recover before you pay for it.

If you do want to try recovery yourself, you could try to "clone" the disks to new ones (sector by sector copy). That preserves the original state, minimizing any risk of further damage to the arrary.

F_L_ · ‎2017-03-14

This is insane....

Today we had another power failure in the house and when the NAS went back it was all good!?

So except from the obvious, take as much backup as I can.

What should I do?

No disk are reporting faulty when looking at the drive info.

StephenB · ‎2017-03-14

@F_L_ wrote:

take as much backup as I can.

Absolutely, and don't delay.

I also suggest getting a UPS for the ReadyNAS.

F_L_ · ‎2017-03-14

@StephenB wrote:
@F_L_ wrote:

take as much backup as I can.

Absolutely, and don't delay.

I also suggest getting a UPS for the ReadyNAS.

Running backups now to Amazon Cloud.

I have a UPS which should turn down the NAS gracefully after 15 min.

But can someone explain what just happened? I don't believe in magic

F_L_ · ‎2017-03-15

A follow-up question that I think is related to the previous system crash.

I can now see that my OS-partition is full

udev             10M  4.0K   10M   1% /dev
/dev/md0        4.0G  3.5G  112M  97% /
tmpfs           994M  4.0K  994M   1% /dev/shm
tmpfs           994M  6.0M  988M   1% /run
tmpfs           497M  9.4M  488M   2% /run/lock
tmpfs           994M     0  994M   0% /sys/fs/cgroup
/dev/md127      7.3T  5.5T  1.9T  75% /data
/dev/md127      7.3T  5.5T  1.9T  75% /apps
/dev/md127      7.3T  5.5T  1.9T  75% /home
/dev/md127      7.3T  5.5T  1.9T  75% /run/nfs4/data/Music
/dev/md127      7.3T  5.5T  1.9T  75% /run/nfs4/data/Videos

Running a folder check I see that I have some large folders such as usr, apps, run etc.

7.7M    ./bin
0       ./boot
8.0K    ./dev
13M     ./etc
112M    ./home
37M     ./lib
4.0K    ./lib64
0       ./media
0       ./mnt
22M     ./opt
0       ./proc
150M    ./root
1.8T    ./run
11M     ./sbin
0       ./selinux
0       ./srv
0       ./sys
0       ./tmp
1.1G    ./usr
269M    ./var
4.5T    ./data
849M    ./apps
8.5M    ./c
44K     ./~
44K     ./.sabnzbd
0       ./logs
29M     ./frontview
6.3T    .

Since I had about ~50% free space before the crash, could it be that while the raid was down .apps was rewritten to the OS-partition instead of /dev/md127?

Or what can be the cause for this?

I´ve check /var/log and it´s quite small.

/usr/:

root@Ultra:/usr# du -h --max-depth=1  | sort -rh | head -10
1.1G    .
561M    ./lib
273M    ./share
175M    ./bin
68M     ./local
27M     ./include
18M     ./sbin
0       ./src
0       ./games

/var/:

root@Ultra:/var# du -h --max-depth=1  | sort -rh | head -10
269M    .
91M     ./lib
70M     ./backups
69M     ./log
19M     ./couchpotato
14M     ./cache
6.5M    ./cores
2.1M    ./readynasd
8.0K    ./spool
8.0K    ./netatalk

I´m a little bit afraid to reboot the NAS right now to see if that free up some space

@mdgm-ntgr: which hdd was reporting high count when you checked? I will check that one with lifeguard when backup is done

StephenB · ‎2017-03-15

Some of those folders (apps for instance) are mounted, and are actually on the data volume. Sometimes files can be stored "underneath" the mount points (e.g., the folder that the mount is mounted on).

You can see the full mount point list (rather long) with

mount

You should take a closer look w/o the mount points by

mount --bind / /mnt

When done,

umount /mnt

I suspect your apps might have have installed onto the OS partition. A common trick is to copy them to the data folder, and put a soft link in the OS folder.

jak0lantash · ‎2017-03-15

Can you use -x option with du please?

This only account for data within the current filesystem, so data from the data volume(s) won't be accounted for. It makes it easier to read in this situation.

mdgm-ntgr · ‎2017-03-15

Should be obvious from my description which disk is bad. Only one has both a lot of reallocated sectors and a number of current pending sectors.

I would free up some space on the 4GB root volume and backup what you can first.

F_L_ · ‎2017-03-15

@StephenB

Here is the full mount

root@Ultra:~# mount
udev on /dev type devtmpfs (rw,noatime,nodiratime,size=10240k,nr_inodes=253708,mode=755)
devpts on /dev/pts type devpts (rw,noatime,nodiratime,mode=600,ptmxmode=000)
/dev/md0 on / type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=5,subvol=/)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime,nodiratime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime,nodiratime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,noatime,nodiratime,size=508652k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,noatime,nodiratime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,noatime,nodiratime,devices)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,noatime,nodiratime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,noatime,nodiratime,blkio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,noatime,nodiratime,cpuset)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,noatime,nodiratime,freezer)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,noatime,nodiratime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,noatime,nodiratime)
sunrpc on /proc/fs/nfsd type nfsd (rw,noatime,nodiratime)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,noatime,nodiratime)
mqueue on /dev/mqueue type mqueue (rw,noatime,nodiratime)
configfs on /sys/kernel/config type configfs (rw,noatime,nodiratime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,noatime,nodiratime)
/dev/md127 on /data type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=5,subvol=/)
/dev/md127 on /apps type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=258,subvol=/.apps)
/dev/md127 on /home type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=256,subvol=/home)
/dev/md127 on /run/nfs4/data/Music type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=262,subvol=/Music)
/dev/md127 on /run/nfs4/data/Videos type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=261,subvol=/Videos)
/dev/md127 on /run/nfs4/home type btrfs (rw,noatime,nodiratime,nospace_cache,subvolid=256,subvol=/home)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noatime,nodiratime)

You should take a closer look w/o the mount points by
mount --bind / /mnt

When done,
umount /mnt

Can you please give me some more info, what should I look for after mount --bind / /mnt ?

I mounted to /mnt and saw that /mnt/apps/ was empty

@jak0lantash du with X, some looks the same:

/

root@Ultra:/# du -hx --max-depth=1  | sort -rh | head -10
1.6G    .
1.1G    ./usr
278M    ./var
118M    ./root
37M     ./lib
29M     ./frontview
22M     ./opt
13M     ./etc
11M     ./sbin
8.7M    ./c

/usr/

root@Ultra:/usr# du -hx --max-depth=1  | sort -rh | head -10
1.1G    .
561M    ./lib
273M    ./share
175M    ./bin
68M     ./local
27M     ./include
18M     ./sbin
0       ./src
0       ./games

/var/

root@Ultra:/var# du -hx --max-depth=1  | sort -rh | head -10
278M    .
70M     ./log
70M     ./backups
59M     ./lib
53M     ./cache
19M     ./couchpotato
6.5M    ./cores
2.1M    ./readynasd
8.0K    ./spool
8.0K    ./netatalk

@mdgm-ntgr found it thanks, too bad it was one of the 3TB drives...

jak0lantash · ‎2017-03-15

@F_L_ wrote:

root@Ultra:/# du -hx --max-depth=1  | sort -rh | head -10
1.6G    .

The OS volume doesn't look full here, which points towards data in a folder underneath a mount.

What if you do:

mount --bind / /mnt
du -h --max-depth=1 /mnt

F_L_ · ‎2017-03-15

Then I get:

root@Ultra:/# mount --bind / /mnt
root@Ultra:/# du -h --max-depth=1 /mnt
7.7M    /mnt/bin
0       /mnt/boot
24K     /mnt/dev
13M     /mnt/etc
0       /mnt/home
37M     /mnt/lib
4.0K    /mnt/lib64
0       /mnt/media
0       /mnt/mnt
22M     /mnt/opt
0       /mnt/proc
117M    /mnt/root
0       /mnt/run
11M     /mnt/sbin
0       /mnt/selinux
0       /mnt/srv
0       /mnt/sys
0       /mnt/tmp
1.1G    /mnt/usr
278M    /mnt/var
1.8G    /mnt/data
0       /mnt/apps
8.8M    /mnt/c
44K     /mnt/~
44K     /mnt/.sabnzbd
0       /mnt/logs
29M     /mnt/frontview
3.4G    /mnt
root@Ultra:/#

jak0lantash · ‎2017-03-15

@F_L_ wrote:

Then I get:

root@Ultra:/# mount --bind / /mnt
root@Ultra:/# du -h --max-depth=1 /mnt
1.8G    /mnt/data

There's your issue... /mnt/data should be empty.

Do you have replicate jobs running? It's common that it pushes data to /data even though the data volume is not mounted, filling up the OS volume.

Then, once the data volume is mounted on /data, you can no longer see what was in the folder itself but the OS volume is still full.

As you have temporarily mounted the OS volume back to /mnt, you should check what's in /mnt/data and clear it to free up space on your OS volume. WARNING: Be careful what you're doing, you want to clear /mnt/data, NOT /data (where your actual data volume is mounted)!

F_L_ · ‎2017-03-15

@jak0lantash wrote:
@F_L_ wrote:
Then I get:
root@Ultra:/# mount --bind / /mnt
root@Ultra:/# du -h --max-depth=1 /mnt
1.8G    /mnt/data
There's your issue... /mnt/data should be empty.
Do you have replicate jobs running? It's common that it pushes data to /data even though the data volume is not mounted, filling up the OS volume.
Then, once the data volume is mounted on /data, you can no longer see what was in the folder itself but the OS volume is still full.
As you have temporarily mounted the OS volume back to /mnt, you should check what's in /mnt/data and clear it to free up space on your OS volume. WARNING: Be careful what you're doing, you want to clear /mnt/data, NOT /data (where your actual data volume is mounted)!

Thanks!

All good now, I deleted the files in /mnt/data and unmounted it again.

OS is now back to 49% full.

It turned out to be a download from usenet that was triggered when the raid was down.

Next step will be to have a lifeguard check on disk3

Device:             sdc
Controller:         0
Channel:            2
Model:              WDC WD30EFRX-68AX9N0
Serial:             WD-WMC1T1298310
Firmware:           80.00A80
Class:              SATA
Sectors:            5860533168
Pool:               data
PoolType:           RAID 5
PoolState:          5
PoolHostId:         5e26b5f2
Health data 
  ATA Error Count:                0
  Reallocated Sectors:            514
  Reallocation Events:            337
  Spin Retry Count:               0
  Current Pending Sector Count:   74
  Uncorrectable Sector Count:     0
  Temperature:                    17
  Start/Stop Count:               46
  Power-On Hours:                 34909
  Power Cycle Count:              46
  Load Cycle Count:               22

jak0lantash · ‎2017-03-15

@F_L_ wrote:

Next step will be to have a lifeguard check on disk3

Device:             sdc
Controller:         0
Channel:            2
Model:              WDC WD30EFRX-68AX9N0
Serial:             WD-WMC1T1298310
Firmware:           80.00A80
Class:              SATA
Sectors:            5860533168
Pool:               data
PoolType:           RAID 5
PoolState:          5
PoolHostId:         5e26b5f2
Health data 
  ATA Error Count:                0
  Reallocated Sectors:            514
  Reallocation Events:            337
  Spin Retry Count:               0
  Current Pending Sector Count:   74
  Uncorrectable Sector Count:     0
  Temperature:                    17
  Start/Stop Count:               46
  Power-On Hours:                 34909
  Power Cycle Count:              46
  Load Cycle Count:               22

"Lifeguard check"?

35 thousand hours, 514 reallocated sectors, 74 pending sectors. No need to check it, it needs to be replaced. It's unfortunate I know, and it's $100, but it's time ^^

F_L_ · ‎2017-03-15

True true, will buy a new one tomorrow 🙂

F_L_ · ‎2017-03-21

Someone marked this as a solved which I guess is fine.

However, I'm still a bit concerned about the original error message and the increase of posts in the forum with this message.

It seems to me that this error has been posted a lot more since after FW 6.6.0.

Is this just a coincidence or has anything been found by Netgear in the FW related to this?

Missing volume after power failure

Re: Missing volume after power failure

Re: Missing volume after power failure

Re: Missing volume after power failure

Re: Missing volume after power failure

Re: Missing volume after power failure

Re: Missing volume after power failure

Re: Missing volume after power failure

Re: Missing volume after power failure

Re: Missing volume after power failure

Re: Missing volume after power failure

Re: Missing volume after power failure

Re: Missing volume after power failure

Re: Missing volume after power failure

Re: Missing volume after power failure

Re: Missing volume after power failure

Re: Missing volume after power failure

Re: Missing volume after power failure

Re: Missing volume after power failure