Re: Readynas 204 - v6.9.2 - Read only

davor242 · ‎2018-02-20

It seems my RN204 decided to put shares into Ready Only mode. How do I find out why, and how do I fix it?

Found this in dmesg.log:

[Tue Feb 20 16:58:20 2018] BTRFS error (device md127): parent transid verify failed on 4204384321536 wanted 135472 found 138739
[Tue Feb 20 16:58:20 2018] BTRFS error (device md127): parent transid verify failed on 4204384321536 wanted 135472 found 138739
[Tue Feb 20 16:58:20 2018] BTRFS warning (device md127): Skipping commit of aborted transaction.
[Tue Feb 20 16:58:20 2018] BTRFS: error (device md127) in cleanup_transaction:1856: errno=-5 IO failure
[Tue Feb 20 16:58:20 2018] BTRFS info (device md127): forced readonly
[Tue Feb 20 16:58:20 2018] BTRFS info (device md127): delayed_refs has NO entry

StephenB · ‎2018-02-20

Generally this is happening when the system decides that allowing writes to the volume will put data at risk. That suggests the the best fix is to destroy and recreate the volume, and then reload the data from backup.

You can download the log zip file, and look for clues.

You could attempt to repair the volume using btrfs shell commands (at your own risk). I strongly recommend ensuring you have a full backup before attempting that.

davor242 · ‎2018-02-20

Thank you. Yeah, I'll make sure to have a backup before proceeding. I don't really understand from logs what has happened and why it puts mounts in read-only.

StephenB · ‎2018-02-20

Usually it's because the system is seeing btrfs errors.

davor242 · ‎2018-02-23

Update: I got an external drive and did a backup (via PC).

* There is some data corruption unfortunatelly.

* Also the snapshots on one of the two shares, unfortunatelly the important one, are borked. They are sort of listed but when I open one it just opens the list of snapshots again.

* There are inconsistencies in the web UI. It says system is healthy, it says 0 snapshots in one place but a lot of snapshots in other place.

* I suspect that things went bad when I deleted a iSCSI LUN, possibly specifically when volume was rebalanced. iSCSI in itself has caused me lot of pain and tears with ReadyNAS. 😕 Not touching iSCSI on ReadyNAS again.

* Currently scrubbing to see if any luck with the corrupt data.

* I have no idea how to remove the read-only stuff and try btrfs repairs.

atz6975 · ‎2018-02-28

Hi same status on 6.9.2 aswell. My btrfs mounts are read only.

I ssh into the system and the data is readable through SCP/SFTP/SSH. Volume capacity is 100% FREE in Readynas OS Web ui.

I've run a btrfs check --init-extent-tree on /dev/md127.

I can deal with backuping the data and losing some before Factory Default...But I wanted some adivce first.

Thanks.

Dmesg shows transid failed...

[   18.807227] md: md127 stopped.
[   18.809384] md: bind<sdb3>
[   18.809719] md: bind<sdc3>
[   18.810061] md: bind<sdd3>
[   18.810519] md: bind<sda3>
[   18.813000] md/raid:md127: device sda3 operational as raid disk 0
[   18.813038] md/raid:md127: device sdd3 operational as raid disk 3
[   18.813045] md/raid:md127: device sdc3 operational as raid disk 2
[   18.813052] md/raid:md127: device sdb3 operational as raid disk 1
[   18.813705] md/raid:md127: allocated 4362kB
[   18.813845] md/raid:md127: raid level 5 active with 4 out of 4 devices, algorithm 2
[   18.813853] RAID conf printout:
[   18.813856]  --- level:5 rd:4 wd:4
[   18.813862]  disk 0, o:1, dev:sda3
[   18.813866]  disk 1, o:1, dev:sdb3
[   18.813871]  disk 2, o:1, dev:sdc3
[   18.813875]  disk 3, o:1, dev:sdd3
[   18.815156] md127: detected capacity change from 0 to 8986874216448
[   19.073929] Adding 1046524k swap on /dev/md1.  Priority:-1 extents:1 across:1046524k
[   19.170923] BTRFS: device label 758b6566:data devid 1 transid 473649 /dev/md127
[   20.317800] BTRFS info (device md127): checking UUID tree
[   20.398973] BTRFS error (device md127): parent transid verify failed on 29458432 wanted 473646 found 473649
[   20.435184] BTRFS error (device md127): parent transid verify failed on 29458432 wanted 473646 found 473649
[   20.635667] BTRFS error (device md127): parent transid verify failed on 29458432 wanted 473646 found 473649
[   20.651949] BTRFS error (device md127): parent transid verify failed on 29458432 wanted 473646 found 473649
[   20.653226] BTRFS error (device md127): parent transid verify failed on 29458432 wanted 473646 found 473649
[   20.654264] BTRFS error (device md127): parent transid verify failed on 29458432 wanted 473646 found 473649
[   20.883842] BTRFS error (device md127): parent transid verify failed on 29458432 wanted 473646 found 473649
[   20.885570] BTRFS error (device md127): parent transid verify failed on 29458432 wanted 473646 found 473649
[   20.885603] BTRFS warning (device md127): iterating uuid_tree failed -5
...

[   43.472283] ------------[ cut here ]------------
[   43.472311] WARNING: CPU: 3 PID: 1884 at fs/btrfs/extent-tree.c:941 btrfs_lookup_extent_info+0x433/0x460()
[   43.472317] Modules linked in: vpd(PO)
[   43.472333] CPU: 3 PID: 1884 Comm: btrfs-cleaner Tainted: P           O    4.4.111.x86_64.1 #1
[   43.472340] Hardware name: NETGEAR  ReadyNAS/ReadyNAS , BIOS 080016  09/20/2010
[   43.472346]  0000000000000000 ffff88002ffefce0 ffffffff883ec0e8 0000000000000000
[   43.472358]  ffffffff88d1a2fe ffff88002ffefd18 ffffffff8806f38c ffff88003d3720a0
[   43.472368]  0000000000000002 ffff88003a01ba10 ffff88002f7221b8 0000000000000000
[   43.472379] Call Trace:
[   43.472394]  [<ffffffff883ec0e8>] dump_stack+0x4d/0x65
[   43.472406]  [<ffffffff8806f38c>] warn_slowpath_common+0x7c/0xb0
[   43.472415]  [<ffffffff8806f475>] warn_slowpath_null+0x15/0x20
[   43.472424]  [<ffffffff882e63c3>] btrfs_lookup_extent_info+0x433/0x460
[   43.472434]  [<ffffffff882ea7ae>] btrfs_drop_snapshot+0x1fe/0x860
[   43.472446]  [<ffffffff88301200>] btrfs_clean_one_deleted_snapshot+0xb0/0x100
[   43.472457]  [<ffffffff882f8602>] cleaner_kthread+0x192/0x210
[   43.472469]  [<ffffffff882f8470>] ? btrfs_destroy_pinned_extent+0xc0/0xc0
[   43.472481]  [<ffffffff88089336>] kthread+0xe6/0x100
[   43.472492]  [<ffffffff88089250>] ? kthread_create_on_node+0x170/0x170
[   43.472504]  [<ffffffff88a726df>] ret_from_fork+0x3f/0x70
[   43.472515]  [<ffffffff88089250>] ? kthread_create_on_node+0x170/0x170
[   43.472523] ---[ end trace 58c64bad13bfb6d3 ]---
[   43.579794] ------------[ cut here ]------------
[   43.581016] kernel BUG at fs/btrfs/extent-tree.c:9102!

mdgm-ntgr · ‎2018-02-28

@atz6975 wrote:

I've run a btrfs check --init-extent-tree on /dev/md127.

Why? If there is corruption that can bake it in.

atz6975 · ‎2018-02-28

Indeed...I got lucky it didn't do much.

Any thoughts on next step (beside wiping device)?

mdgm-ntgr · ‎2018-02-28

Have you tried using btrfs restore to see if you can restore a copy of the data e.g. onto a USB disk?

atz6975 · ‎2018-02-28

Thx for you concern Mdgm.

Yes btrfs restore works, and the volume is actually mounted (read-only) by readynas OS anyway.

So accessing the data is not looking too hard.

I just hoped to save the snapshots and avoid rebuilding and back and forth of data (6TB out and back in).

As I type this, the NAS has become unresponsive remotly, so I'll go on-site have a look and probably cold reboot. It seems Frontview is bringing it down trying to access some info (in my case Backup jobs definition).

Re: Readynas 204 - v6.9.2 - Read only

Readynas 204 - v6.9.2 - Read only