516 /dev/md127 suddenly read only- although space available

[aequus]

All my shares became suddenly read only although there is still 4TB free . After a reboot it was writeable for an hour but now it is again RO. OS version is latest 6.2.3 balance is done weekly.

What can I do?

root@nas1:/data/daten# btrfs filesystem show
Label: '7c6e49a2:root' uuid: 72ce1783-d430-4507-b346-48dae62b47f9
Total devices 1 FS bytes used 1.09GiB
devid 1 size 4.00GiB used 4.00GiB path /dev/md0

Label: '7c6e49a2:data' uuid: 4f716840-2b0d-4723-bbda-2d0ef4cc249e
Total devices 1 FS bytes used 13.28TiB
devid 1 size 18.17TiB used 13.81TiB path /dev/md127

root@nas1:~# btrfs filesystem df -h /data
Data, single: total=13.78TiB, used=13.27TiB
System, DUP: total=32.00MiB, used=1.91MiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=17.00GiB, used=15.81GiB
root@nas1:~#

Mount
/dev/md127 on /data type btrfs (ro,noatime,nodiratime,nospace_cache)
/dev/md127 on /home type btrfs (ro,noatime,nodiratime,nospace_cache)
/dev/md127 on /apps type btrfs (ro,noatime,nodiratime,nospace_cache)
/dev/md127 on /var/ftp/home type btrfs (ro,noatime,nodiratime,nospace_cache)
/dev/md127 on /var/ftp/ccu type btrfs (ro,noatime,nodiratime,nospace_cache)
/dev/md127 on /var/ftp/media type btrfs (ro,noatime,nodiratime,nospace_cache)
/dev/md127 on /run/nfs4/data/ccu type btrfs (ro,noatime,nodiratime,nospace_cache)
/dev/md127 on /run/nfs4/data/ccu-historian type btrfs (ro,noatime,nodiratime,nospace_cache)
/dev/md127 on /run/nfs4/data/filme type btrfs (ro,noatime,nodiratime,nospace_cache)
/dev/md127 on /run/nfs4/data/media type btrfs (ro,noatime,nodiratime,nospace_cache)
/dev/md127 on /run/nfs4/data/sabnzbd type btrfs (ro,noatime,nodiratime,nospace_cache)
/dev/md127 on /run/nfs4/home type btrfs (ro,noatime,nodiratime,nospace_cache)
tmpfs on /data/daten/snapshot type tmpfs (rw,noatime,nodiratime,size=4k)

[EskenderNG]

Hello,

Depending on if you are still under warranty it would be advisable to contact support. Otherwise you could try deleting some files or disabling snapshots and run balance again.

Bye,
Eskender

[EskenderNG]

It is strange that with the data and metadata partitions being almost full the remaining free space has not been allocated.

[aequus]

Will disabling Snapshot cause the lost of all old file Version? How can I make the volume RW to delete some file? Is doing this a solution new files will be added and I can rerun in the Problem again?

[aequus]

I would like to delete some files, but therefore I have remount /data, or make it writeable. How can I do this?

/dev/md127 on /data type btrfs (ro,noatime,nodiratime,nospace_cache)

[mdgm-ntgr]

Hi aequus,

Can you please send me your logs (see the Sending Logs link in my sig)?

[aequus]

you have mail

BR Gerd

[mdgm-ntgr]

Do you have a backup?

I think running a balance weekly is very often. I wouldn't run it that regularly.

[StephenB]

mdgm wrote:
...I think running a balance weekly is very often. I wouldn't run it that regularly.

I run a balance every 3 months myself (that's the cadence I use for defrag, scrub, and disk test as well)

[aequus]

Hi mdgm,

I have daily backups from the important data but I want to keep also the non important one. What is the problem with my machine?

[aequus]

My 516 is not usable since two weeks now. I googled a little bit and found a known problems in a btfrs mailing list it sound very simular to my situation.

Can anyboby of the admin team help to get my 516 working again soon?


root@nas2:~# btrfs --version used by OS6
Btrfs v3.17.1


https://forums.opensuse.org/showthread. ... o-readonly

http://www.spinics.net/lists/linux-btrfs/msg38372.html

Juan Orti Alcaine posted on Wed, 15 Oct 2014 09:08:14 +0200 as excerpted:

> I've also experienced Btrfs corruptions with 3.17.0

> I do readonly snapshots every hour of all the subvolumes, so I have
> hundreds of snapshots.

That's a known issue with read-only snapshots in 3.17.0. There's quite a
thread on the list about it.

So I'd suggest either turning off read-only snapshots on 3.17 (which I'm
running here without snapshots, no problem), possibly switching to
writable snapshots as they don't seem to trigger the problem, or as you
mentioned doing already, going back to 3.16.x (x>2 due to another bug,
latest should be good), until the read-only snapshots issue with 3.17.0
is traced down and fixed.

Given the approximately two kernel cycles it took for the widely
reproduced but rather difficult to trace compression-related bug in 3.15
to be reported in 3.15 and traced and fixed in 3.17-rc2 and 3.16.2, I'd
guess a fix for this similarly widely reproduced read-only-snapshot-
related bug should be no later than 3.19-rc3 and 3.18.3, possibly rather
earlier if it proves easier to trace, especially since this one seems to
have been reported and recognized as widely occurring a bit faster than
the compression-related bug. But with testing, etc, it's still likely to
be late in the 3.18-rc cycle before mainline commit, so it'll probably be
rather late in the 3.17.x stable cycle, if it makes it at all. Unless it
gets picked as a long-term support kernel, the full 3.17 stable cycle
might in fact be blacklisted for btrfs due to this bug, much like the
full 3.15 stable cycle ended up being blacklisted due to the compression-
related bug.

So either switch your snapshots to writable if it's not going to
interfere with your use-case, or stay on the 3.16.x, x>2, stable series
until the problem is fixed, hopefully with 3.18.0, tho it might be 3.18.2
or so. I seriously doubt it'll be longer than that, because it's a well
reproduced bug which makes it both high priority and easy to test fixes
for.

[mdgm-ntgr]

BTRFS version 3.17.1 indicates you are using 6.2.0 not 6.2.3. Can you confirm what version of OS6 you are running?

[aequus]

These are my versions:

root@nas1:~# btrfs --version
Btrfs v3.17.3
root@nas1:~# cat /proc/version
Linux version 3.0.101.RNx86_64.3 (jenkins@blocks) (gcc version 4.7.2 (Debian 4.7.2-5) ) #1 SMP Wed Apr 1 16:02:14 PDT 2015
root@nas1:~# cat /var/readynasd/.flash_update_info
6.2.3
57862144
ReadyNASOS Update

[hankburton]

Did you ever get a solution to this?  I'm having the same problem.

I deleted an iSCSI LUN via the web admin.  The system doesn't report the space freed from deleting that file.  I SSH into the unit and see that the /dev/md127 device is mounting as read only.  I try to make folder in /data and cannot.  

thanks