NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Readynas 3312 BTRFS error (device md127) in __btrfs_free_extent:6998: errno=-28 No space left
Hi,
I have another issue with BTRFS.
All the sudden, I couldn't write to the share.
It gave the exception 'Permission denied'.
I also could reset the shares permissions.
So i tried to reset the server but it didnt come back up.
After a power cycle, it was back.
It all started with exception in the journal:
Jan 23 00:28:15 SKNAS05 kernel: ------------[ cut here ]------------
Jan 23 00:28:19 SKNAS05 kernel: WARNING: CPU: 3 PID: 3751 at fs/btrfs/extent-tree.c:6998 __btrfs_free_extent.isra.71+0x28b/0xd70()
Jan 23 00:28:27 SKNAS05 kernel: BTRFS: Transaction aborted (error -28)
Jan 23 00:28:27 SKNAS05 kernel: Modules linked in: vpd(PO)
Jan 23 00:28:27 SKNAS05 kernel: CPU: 3 PID: 3751 Comm: btrfs-transacti Tainted: P O 4.4.116.x86_64.1 #1
Jan 23 00:28:27 SKNAS05 kernel: Hardware name: NETGEAR ReadyNAS 3312/ReadyNAS RR3312, BIOS RR3312v112 07/25/2016
Jan 23 00:28:27 SKNAS05 kernel: 0000000000000000 ffff88024bb03ae8 ffffffff883ec5a8 ffff88024bb03b30
Jan 23 00:28:27 SKNAS05 kernel: ffffffff88d1aab1 ffff88024bb03b20 ffffffff8806f3ac 00000000ffffffe4
Jan 23 00:28:27 SKNAS05 kernel: 0000294acccc8000 ffff8802664fea00 ffff8800182a5f50 0000000000000002
Jan 23 00:28:27 SKNAS05 kernel: Call Trace:
Jan 23 00:28:27 SKNAS05 kernel: [<ffffffff883ec5a8>] dump_stack+0x4d/0x65
Jan 23 00:28:27 SKNAS05 kernel: [<ffffffff8806f3ac>] warn_slowpath_common+0x7c/0xb0
Jan 23 00:28:27 SKNAS05 kernel: [<ffffffff8806f427>] warn_slowpath_fmt+0x47/0x50
Jan 23 00:28:27 SKNAS05 kernel: [<ffffffff882e520b>] __btrfs_free_extent.isra.71+0x28b/0xd70
Jan 23 00:28:27 SKNAS05 kernel: [<ffffffff882e8d26>] __btrfs_run_delayed_refs+0x626/0x10f0
Jan 23 00:28:27 SKNAS05 kernel: [<ffffffff882ec75d>] btrfs_run_delayed_refs+0x7d/0x2e0
Jan 23 00:28:27 SKNAS05 kernel: [<ffffffff8830177f>] btrfs_commit_transaction.part.25+0x5f/0xa80
Jan 23 00:28:27 SKNAS05 kernel: [<ffffffff882ec8e9>] ? btrfs_run_delayed_refs+0x209/0x2e0
Jan 23 00:28:27 SKNAS05 kernel: [<ffffffff883021d5>] btrfs_commit_transaction+0x35/0x60
Jan 23 00:28:27 SKNAS05 kernel: [<ffffffff882fcd36>] transaction_kthread+0x216/0x230
Jan 23 00:28:27 SKNAS05 kernel: [<ffffffff882fcb20>] ? btrfs_cleanup_transaction+0x4a0/0x4a0
Jan 23 00:28:27 SKNAS05 kernel: [<ffffffff88089356>] kthread+0xe6/0x100
Jan 23 00:28:27 SKNAS05 kernel: [<ffffffff88a6f972>] ? __schedule+0x1b2/0x600
Jan 23 00:28:27 SKNAS05 kernel: [<ffffffff88089270>] ? kthread_create_on_node+0x170/0x170
Jan 23 00:28:27 SKNAS05 kernel: [<ffffffff88a731c5>] ret_from_fork+0x55/0x80
Jan 23 00:28:27 SKNAS05 kernel: [<ffffffff88089270>] ? kthread_create_on_node+0x170/0x170
Jan 23 00:28:27 SKNAS05 kernel: ---[ end trace 7029bcdeed995522 ]---
Jan 23 00:28:27 SKNAS05 kernel: BTRFS: error (device md127) in __btrfs_free_extent:6998: errno=-28 No space left
Jan 23 00:28:27 SKNAS05 kernel: BTRFS info (device md127): forced readonly
Jan 23 00:28:32 SKNAS05 kernel: BTRFS: error (device md127) in btrfs_run_delayed_refs:2993: errno=-28 No space left
Some more info:
root@SKNAS05:~# btrfs filesystem show
Label: '2fe72d70:root' uuid: 8444e252-7714-4ce8-a75c-daff364d1c2c
Total devices 1 FS bytes used 644.34MiB
devid 1 size 4.00GiB used 1.64GiB path /dev/md0
Label: '2fe72d70:data' uuid: 7638dba2-0e72-4c52-b071-987b1395d91e
Total devices 1 FS bytes used 42.54TiB
devid 1 size 54.55TiB used 44.25TiB path /dev/md127Enough space free.
Running on firmware 6.9.3
Any idea what this is and how to prevent it?
8 Replies
Replies have been turned off for this discussion
some more information:
root@SKNAS05:~# btrfsck /dev/md127 --force WARNING: filesystem mounted, continuing because of --force parent transid verify failed on 45403858534400 wanted 1820270 found 1820360 parent transid verify failed on 45403858534400 wanted 1820270 found 1820360 parent transid verify failed on 45403858534400 wanted 1820270 found 1820360 leaf parent key incorrect 45403858534400 ERROR: cannot open file system
Yes, when there is an unrecoverable BTRFS error, one thing the OS tries is to make the volume read-only so you can make sure your backup is in order before something more happens (which likely would if writes were allowed). The only solution to this is normally a factory default and restoration of data from backup. It's a good idea to first check the SMART data for your drives just to be sure a hardware issue isn't the driving factor.
Thanks for the information.
Shouldn't a balance fix / prevent this?
smartctl doesn't show anything weird.
Anything else I can check?
For now, i'll copy the whole 80TB somewhere else and reset the NAS.
But this shouldn't be happening in a NAS, we buy them to use the RAID5, to prevent downtime.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
