× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

ReadyNAS Pioneer Pro OS6 volume missing after hard reset

vasyam
Tutor

ReadyNAS Pioneer Pro OS6 volume missing after hard reset

Hello,

I am running OS6.10.3 on my legacy RNDP6000-100 Pioneer Pro. It works generally fine but freezes someties and the only way to unfreeze it was to perform a hard reset which also worked generally fine until this morning. After a usual resync I got the volume missing error message with all discs being shown as working fine.

It seems that this issue does occur regularly so hope there is a ready solution.

TIA

 

Model: RNDP6000|ReadyNAS Pro 6 Chassis only
Message 1 of 23

Accepted Solutions
vasyam
Tutor

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset

Got it, thank you

View solution in original post

Message 21 of 23

All Replies
StephenB
Guru

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset

There must have been pending disk writes when you did the hard reset - resulting in an out-of-sync RAID array.

 

The puzzle here is to understand why the system hangs.  One possibility is that it has insufficient memory.  Are you using the stock 1 GB, or have you upgraded the RAM?

Message 2 of 23
vasyam
Tutor

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset

I have added a 4Gb stick and upgraded the processor

Message 3 of 23
StephenB
Guru

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset


@vasyam wrote:

I have added a 4Gb stick and upgraded the processor


So 5 GB total?  Whether 4 or 5 GB, it should certainly have enough RAM. 

 

Have you tried accessing the NAS with ssh when it appears to hang?

Message 4 of 23
vasyam
Tutor

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset

I did not as I am just a user and need to educate myself for this kind of steps when absolutely needed)

Message 5 of 23
vasyam
Tutor

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset

And yes, 5Gb total RAM

Message 6 of 23
StephenB
Guru

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset


@vasyam wrote:

I did not 


Then I suggest downloading the log zip file, and looking for any clues about the hang in there.

 

Are you running apps?  If so, which ones are installed.

Message 7 of 23
vasyam
Tutor

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset

I have downloaded the log files, but am not sure where to look. I do not believe I am running any apps besides Transmission

Message 8 of 23
StephenB
Guru

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset


@vasyam wrote:

I have downloaded the log files, but am not sure where to look. 


Usually I start with disk_info.log - looking for disk health issues in the smart stats.  Looking at the tail end of volume.log is also useful - makeing sure you have adequate free space in the OS partition ( /dev/md0 ).  Normally the OS partition runs about 25% full, but Transmission might raise that some.

 

You can then look through readynasd.log and status.log - paying particular attention to anything logged there around the time of the hang.  

 

Similarly, system.log, kernel.log, and dmesg.log might have useful status around the time of the hang, but those are loaded with info that is often hard to interpret.

 

Another approach is to just turn off transmission for a while, and see if the hangs stop.

Message 9 of 23
vasyam
Tutor

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset

Got it, thank you, but right now there is no volume and no apps are shown as installed, as in the attached screenshot. Judging by other threads, the data should still be there, but I do not know how to repair the RAID to be able to access it. Any help will be greatly appreciated.

Message 10 of 23
vasyam
Tutor

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset

disk_info.log - no issue whatsoever with any disk

/dev/md0 4.0G 1.1G 2.6G 29% /

readynasd.log - nothing logged for the hang

status.log - again nothing for the hang, but shows that after rebooting the volume data was resynced

I am indeed lost with the other logs you have mentioned, do not know how to interpret the info there

Message 11 of 23
vasyam
Tutor

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset

I noticed errors in dmesg.log which may be telling smth:

 

[Sat Sep 26 17:53:22 2020] md/raid:md127: device sda3 operational as raid disk 0
[Sat Sep 26 17:53:22 2020] md/raid:md127: device sdf3 operational as raid disk 5
[Sat Sep 26 17:53:22 2020] md/raid:md127: device sde3 operational as raid disk 4
[Sat Sep 26 17:53:22 2020] md/raid:md127: device sdd3 operational as raid disk 3
[Sat Sep 26 17:53:22 2020] md/raid:md127: device sdc3 operational as raid disk 2
[Sat Sep 26 17:53:22 2020] md/raid:md127: device sdb3 operational as raid disk 1
[Sat Sep 26 17:53:22 2020] md/raid:md127: allocated 6474kB
[Sat Sep 26 17:53:22 2020] md/raid:md127: raid level 5 active with 6 out of 6 devices, algorithm 2
[Sat Sep 26 17:53:22 2020] RAID conf printout:
[Sat Sep 26 17:53:22 2020] --- level:5 rd:6 wd:6
[Sat Sep 26 17:53:22 2020] disk 0, o:1, dev:sda3
[Sat Sep 26 17:53:22 2020] disk 1, o:1, dev:sdb3
[Sat Sep 26 17:53:22 2020] disk 2, o:1, dev:sdc3
[Sat Sep 26 17:53:22 2020] disk 3, o:1, dev:sdd3
[Sat Sep 26 17:53:22 2020] disk 4, o:1, dev:sde3
[Sat Sep 26 17:53:22 2020] disk 5, o:1, dev:sdf3
[Sat Sep 26 17:53:22 2020] md127: detected capacity change from 0 to 9977163939840
[Sat Sep 26 17:53:23 2020] Adding 2094844k swap on /dev/md1. Priority:-1 extents:1 across:2094844k
[Sat Sep 26 17:53:23 2020] BTRFS: device label 33ea29df:data devid 1 transid 886902 /dev/md127
[Sat Sep 26 17:53:23 2020] BTRFS info (device md127): has skinny extents
[Sat Sep 26 17:53:35 2020] BTRFS critical (device md127): corrupt leaf, invalid file extent type: block=720666624, root=1, slot=124
[Sat Sep 26 17:53:35 2020] BTRFS critical (device md127): corrupt leaf, invalid file extent type: block=720666624, root=1, slot=124
[Sat Sep 26 17:53:35 2020] BTRFS: error (device md127) in btrfs_recover_log_trees:5713: errno=-5 IO failure (Couldn't read target root for tree log recovery.)
[Sat Sep 26 17:53:35 2020] BTRFS: error (device md127) in btrfs_replay_log:2436: errno=-5 IO failure (Failed to recover log tree)
[Sat Sep 26 17:53:35 2020] BTRFS info (device md127): delayed_refs has NO entry
[Sat Sep 26 17:53:35 2020] BTRFS error (device md127): cleaner transaction attach returned -30
[Sat Sep 26 17:53:35 2020] BTRFS error (device md127): open_ctree failed

Message 12 of 23
StephenB
Guru

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset

[Sat Sep 26 17:53:23 2020] BTRFS: device label 33ea29df:data devid 1 transid 886902 /dev/md127
[Sat Sep 26 17:53:23 2020] BTRFS info (device md127): has skinny extents
[Sat Sep 26 17:53:35 2020] BTRFS critical (device md127): corrupt leaf, invalid file extent type: block=720666624, root=1, slot=124
[Sat Sep 26 17:53:35 2020] BTRFS critical (device md127): corrupt leaf, invalid file extent type: block=720666624, root=1, slot=124
[Sat Sep 26 17:53:35 2020] BTRFS: error (device md127) in btrfs_recover_log_trees:5713: errno=-5 IO failure (Couldn't read target root for tree log recovery.)
[Sat Sep 26 17:53:35 2020] BTRFS: error (device md127) in btrfs_replay_log:2436: errno=-5 IO failure (Failed to recover log tree)
[Sat Sep 26 17:53:35 2020] BTRFS info (device md127): delayed_refs has NO entry
[Sat Sep 26 17:53:35 2020] BTRFS error (device md127): cleaner transaction attach returned -30
[Sat Sep 26 17:53:35 2020] BTRFS error (device md127): open_ctree failed

This doesn't explain the hang, but it is showing the problem with the missing volume.  md127 is your data volume, and the btrfs file system on it has gotten corrupted.

 

Do you have a backup of the files?

Message 13 of 23
vasyam
Tutor

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset

Unfortunately, I do not having counted on RAID5 redundancy

Message 14 of 23
vasyam
Tutor

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset

Is there anything that could be done to attempt to repair the FS?

Message 15 of 23
StephenB
Guru

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset


@vasyam wrote:

Unfortunately, I do not having counted on RAID5 redundancy


That's not a good strategy, as you've just learned.  RAID is convenient, but it isn't enough to keep data safe.

 


@vasyam wrote:

Is there anything that could be done to attempt to repair the FS?


If you are skilled with ssh, you could try to recover the data volume.  But if you don't know what you are doing, you can easily do more damage (making recovery impossible). 

 

Netgear does offer a data recovery service ( https://kb.netgear.com/69/ReadyNAS-Data-Recovery-Diagnostics-Scope-of-Service ) - which they do remotely.  But since you have a converted legacy NAS, I don't think they will provide it in your case. Perhaps @JohnCM_S or @Marc_V can clarify that.  There are other services out there, I think most would require you to mail them your disks.

 

If you can connect all the disks to a Windows PC, you can try RAID recovery software that supports BTRFS.  There aren't many options here, one that folks here have used is ReclaiMe ( https://www.reclaime.com/ ).  One challenge here is that in addition to paying for the software, you likely will either need several USB adapter/docks or a USB enclosure that you can connect to the PC.

 

Also, ReclaiMe doesn't repair the file system - it just lets you off-load the data to other storage.

 

 

 

 

Message 16 of 23
vasyam
Tutor

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset

I am not skilled with ssh, but if the solution is documented somewhere, I could perhaps attempt to replicate it. All the critical data is backed up, I could be OK with losing the rest but rather not as it would take a while to repopulate.

Message 17 of 23
StephenB
Guru

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset


@vasyam wrote:

I am not skilled with ssh, 


Have you ever used it?

Message 18 of 23
vasyam
Tutor

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset

I used it once a while ago to make changes to a media player setup and it worked since I had a step-by-step instructions on how to do it. I realize that it may not be straightforward in this case, but am willing to try to recover the volume if the main downside is to have to wipe out the RAID clean. 

Message 19 of 23
StephenB
Guru

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset


@vasyam wrote:

I used it once a while ago to make changes to a media player setup and it worked since I had a step-by-step instructions on how to do it. I realize that it may not be straightforward in this case, but am willing to try to recover the volume if the main downside is to have to wipe out the RAID clean. 


There is no guide (since there can be more than one cause).  Plus this is a procedure I've never needed to do myself - so I am certainly not an expert on this.  So this would certainly be at your own risk.

 

But (assuming you were running XRAID), you could try

# btrfs device scan
# btrfs fi show
# btrfs-zero-log /dev/md127 # mount -t btrfs -r ro,recovery /dev/md127 /data

That would attempt to recover the file system, and mount the result as read-only.  If that looks ok, you can just reboot, and it should mount normally.

 

Remember that you do need to log in as root (using the NAS admin password) when you use ssh.

 

Message 20 of 23
vasyam
Tutor

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset

Got it, thank you

Message 21 of 23
vasyam
Tutor

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset

This is what I am getting

 

root@NASHNAS:~# btrfs-zero-log /dev/md127

WARNING: this utility is deprecated, please use 'btrfs rescue zero-log'

 

Clearing log on /dev/md127, previous log_root 747175936, level 0

root@NASHNAS:~# mount -t btrfs -r ro,recovery /dev/md127 /data

 

Usage:

 mount [-lhV]

 mount -a [options]

 mount [options] [--source] <source> | [--target] <directory>

 mount [options] <source> <directory>

 mount <operation> <mountpoint> [<target>]

 

Options:

 -a, --all               mount all filesystems mentioned in fstab

 -c, --no-canonicalize   don't canonicalize paths

 -f, --fake              dry run; skip the mount(2) syscall

 -F, --fork              fork off for each device (use with -a)

 -T, --fstab <path>      alternative file to /etc/fstab

 -h, --help              display this help text and exit

 -i, --internal-only     don't call the mount.<type> helpers

 -l, --show-labels       lists all mounts with LABELs

 -n, --no-mtab           don't write to /etc/mtab

 -o, --options <list>    comma-separated list of mount options

 -O, --test-opts <list>  limit the set of filesystems (use with -a)

 -r, --read-only         mount the filesystem read-only (same as -o ro)

 -t, --types <list>      limit the set of filesystem types

     --source <src>      explicitly specifies source (path, label, uuid)

     --target <target>   explicitly specifies mountpoint

 -v, --verbose           say what is being done

 -V, --version           display version information and exit

 -w, --rw, --read-write  mount the filesystem read-write (default)

 

 -h, --help     display this help and exit

 -V, --version  output version information and exit

 

Source:

 -L, --label <label>     synonym for LABEL=<label>

 -U, --uuid <uuid>       synonym for UUID=<uuid>

 LABEL=<label>           specifies device by filesystem label

 UUID=<uuid>             specifies device by filesystem UUID

 PARTLABEL=<label>       specifies device by partition label

 PARTUUID=<uuid>         specifies device by partition UUID

 <device>                specifies device by path

 <directory>             mountpoint for bind mounts (see --bind/rbind)

 <file>                  regular file for loopdev setup

 

Operations:

 -B, --bind              mount a subtree somewhere else (same as -o bind)

 -M, --move              move a subtree to some other place

 -R, --rbind             mount a subtree and all submounts somewhere else

 --make-shared           mark a subtree as shared

 --make-slave            mark a subtree as slave

 --make-private          mark a subtree as private

 --make-unbindable       mark a subtree as unbindable

 --make-rshared          recursively mark a whole subtree as shared

 --make-rslave           recursively mark a whole subtree as slave

 --make-rprivate         recursively mark a whole subtree as private

 --make-runbindable      recursively mark a whole subtree as unbindable

 

For more details see mount(8).

Message 22 of 23
StephenB
Guru

Re: ReadyNAS Pioneer Pro OS6 volume missing after hard reset


@vasyam wrote:

 

# mount -t btrfs -r ro,recovery /dev/md127 /data

Sorry, it should have been 

# mount -t btrfs -o ro,recovery /dev/md127 /data

 

Message 23 of 23
Top Contributors
Discussion stats
  • 22 replies
  • 4109 views
  • 0 kudos
  • 2 in conversation
Announcements