NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
RN314 - Every morning: volume error read only.
Hi there, every morning while my backups are running the system says: on volume data an error appears - and puts it into read-only mode. I checked all HDDs and they show no ATA error. I did...
ThiloS wrote:
If i reboot the nas i can easily finish every backup-job without a problem... : - (
So i can't see a problem with a specific backup-job
If you can't figure out which subvolume is damaged, then you likely will need to do a full reset and restore everything from backup.
If you have ssh enabled, you could try doing a btrfs send for every subvolume to /dev/null, and look for errors after each send.
Are these backups writing to the NAS, or are they reading from the NAS?
Another thought:
My NAS starts at 7:00 in the morning.
The first Backup-Jobs are set between 7:00 and 8:00.
The second "wave" from 8:00 til 9:00.
[All backup-jobs are Rsync via SSH from my webservers to the NAS]
Since all backup jobs running well in the middle of the day if i start them manually - i tried now to set them from 8:00 to 9:00
and from 9:00 til 10:00.
(The idea is - that the anti-virus is scanning on the system start-up and at the same time the first backup-jobs are running. Maybe those two services don't like each other ?)
At least the result is for today: No errors. Backup-jobs are running well. The volume is in normal state.
What do you think? Is it possible that this is the error?
Should i deactivate AntiVirus at all ?
ThiloS wrote:
Another thought:
My NAS starts at 7:00 in the morning.
The first Backup-Jobs are set between 7:00 and 8:00. ...
Since all backup jobs running well in the middle of the day if i start them manually - i tried now to set them from 8:00 to 9:00
and from 9:00 til 10:00.
At least the result is for today: No errors. Backup-jobs are running well. The volume is in normal state.That is a bit odd. I'd leave it set up this way for now, and see if the problem comes back. FWIW, the problem could have been initiated by an error when the NAS started up, and not necessarily the backup jobs.
If the problem doesn't recur over the next couple of days, then I'd suggest doing a scrub from the volume settings wheel - followed by a balance. You'd need to leave the NAS on 24x7 until those complete.
ThiloS wrote:
(The idea is - that the anti-virus is scanning on the system start-up and at the same time the first backup-jobs are running. Maybe those two services don't like each other ?)
...Should i deactivate AntiVirus at all ?
I don't think you have enough information to isolate this to AntiVirus, and there are other things that might be happening at startup (snapshot pruning for example). Plus AntiVirus should also be scanning the files that are being updated by the backup jobs - no matter when they run.
Normally the NAS will go into read-only mode when it detects a problem with the file system itself - and you are seeing those errors in your log zip file. No matter what the cause, it's important to sort out if there is corruption somewhere in the file system on-disk structures, and also to check disk health. The scrub and the balance I suggested above will help with that (and if you've never run them, it would be good to run them anyway). If the volume is healthy, then we can take a closer look at the cause.
BTW, if the backups are initiated by the web server, then you might want to create snapshots when the NAS powers up - that would ensure that your snapshots are coherent. Since the snapshots can only be created on hourly boundaries, that would require you to start up the NAS an hour before the first backup (and also schedule the the first backup wave a bit after the hour). I wouldn't rely on the NAS making a 7 am snapshot if it is booting up at 7 am. FWIW, The NAS backup jobs start at 5 minutes past the hour.
As an aside, I suggest custom snapshots, and not smart ones. Custom snapshots allow you to set retention, which makes the snapshot space easier to manage.
ok - thank you !!!
I'll have a look at it. check for errors in the next days and try scrubbing and balancing.
i make custom snapshots once a week. (but this since 5 years now...)
If things turn out to be bad i'll be back again : -)
hmmm - today i got the read-only warning again... ok - so there is a problem:
Kernel log says:
Nov 25 09:33:52 NAS kernel: Call Trace:
Nov 25 09:33:52 NAS kernel: [<ffffffff88363e61>] dump_stack+0x4d/0x63
Nov 25 09:33:52 NAS kernel: [<ffffffff880dd14a>] warn_slowpath_common+0x8f/0xa8
Nov 25 09:33:52 NAS kernel: [<ffffffff88282c3d>] ? __btrfs_free_extent+0xaf4/0xb31
Nov 25 09:33:52 NAS kernel: [<ffffffff88063f94>] warn_slowpath_fmt+0x47/0x49
Nov 25 09:33:52 NAS kernel: [<ffffffff8828222f>] ? __btrfs_free_extent+0xe6/0xb31
Nov 25 09:33:52 NAS kernel: [<ffffffff88282c3d>] __btrfs_free_extent+0xaf4/0xb31
Nov 25 09:33:52 NAS kernel: [<ffffffff882dff48>] ? btrfs_merge_delayed_refs+0x60/0x43c
Nov 25 09:33:52 NAS kernel: [<ffffffff88286769>] __btrfs_run_delayed_refs+0xa78/0xcbf
Nov 25 09:33:52 NAS kernel: [<ffffffff880e5eda>] ? set_page_dirty+0x3c/0x5b
Nov 25 09:33:52 NAS kernel: [<ffffffff88288537>] btrfs_run_delayed_refs+0x66/0x24f
Nov 25 09:33:52 NAS kernel: [<ffffffff8828bea7>] btrfs_start_dirty_block_groups+0x32c/0x3eb
Nov 25 09:33:52 NAS kernel: [<ffffffff8829af9f>] btrfs_commit_transaction+0xe5/0x9a2
Nov 25 09:33:52 NAS kernel: [<ffffffff8829bb1b>] ? start_transaction+0x2bf/0x3bc
Nov 25 09:33:52 NAS kernel: [<ffffffff882966b3>] ? btrfs_cleanup_transaction+0x458/0x458
Nov 25 09:33:52 NAS kernel: [<ffffffff88296784>] transaction_kthread+0xd1/0x1b3
Nov 25 09:33:52 NAS kernel: [<ffffffff88079cb2>] kthread+0xdc/0xe4
Nov 25 09:33:52 NAS kernel: [<ffffffff88079bd6>] ? kthread_worker_fn+0x129/0x129
Nov 25 09:33:52 NAS kernel: [<ffffffff888e6c6f>] ret_from_fork+0x3f/0x80
Nov 25 09:33:52 NAS kernel: [<ffffffff88079bd6>] ? kthread_worker_fn+0x129/0x129
Nov 25 09:33:52 NAS kernel: ---[ end trace 822d266df2293e05 ]---
Nov 25 09:33:52 NAS kernel: BTRFS: error (device md127) in __btrfs_free_extent:7004: errno=-28 No space left
Nov 25 09:33:52 NAS kernel: BTRFS info (device md127): forced readonly
Nov 25 09:33:54 NAS kernel: BTRFS: error (device md127) in btrfs_run_delayed_refs:2995: errno=-28 No space leftSystem log says:
Nov 25 09:01:14 NAS clamd[4699]: SelfCheck: Database status OK. Nov 25 09:01:37 NAS connmand[2782]: ntp: adjust (slew): +0.011105 sec Nov 25 09:03:07 NAS dbus[2765]: [system] Activating service name='org.opensuse.Snapper' (using servicehelper) Nov 25 09:03:07 NAS dbus[2765]: [system] Successfully activated service 'org.opensuse.Snapper' Nov 25 09:03:07 NAS snapperd[6114]: THROW: invalid config Nov 25 09:03:07 NAS snapperd[6114]: config '3' is invalid Nov 25 09:03:07 NAS snapperd[6114]: THROW: invalid config Nov 25 09:03:07 NAS snapperd[6114]: config '4' is invalid Nov 25 09:05:01 NAS CRON[6371]: pam_unix(cron:session): session opened for user root by (uid=0) Nov 25 09:05:01 NAS CRON[6370]: pam_unix(cron:session): session opened for user root by (uid=0) Nov 25 09:05:01 NAS CRON[6372]: (root) CMD (/frontview/bin/fvbackup 108 &> /dev/null) Nov 25 09:05:01 NAS CRON[6373]: (root) CMD (/frontview/bin/fvbackup 107 &> /dev/null) Nov 25 09:05:01 NAS CRON[6371]: pam_unix(cron:session): session closed for user root Nov 25 09:05:01 NAS CRON[6370]: pam_unix(cron:session): session closed for user root Nov 25 09:09:01 NAS CRON[6411]: pam_unix(cron:session): session opened for user root by (uid=0) Nov 25 09:09:01 NAS CRON[6412]: (root) CMD ( [ -x /usr/lib/php5/sessionclean ] && /usr/lib/php5/sessionclean) Nov 25 09:09:06 NAS CRON[6411]: pam_unix(cron:session): session closed for user root Nov 25 09:11:14 NAS clamd[4699]: SelfCheck: Database status OK. Nov 25 09:17:01 NAS CRON[6526]: pam_unix(cron:session): session opened for user root by (uid=0) Nov 25 09:17:01 NAS CRON[6527]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Nov 25 09:17:01 NAS CRON[6526]: pam_unix(cron:session): session closed for user root Nov 25 09:18:41 NAS connmand[2782]: ntp: adjust (slew): -0.014350 sec Nov 25 09:20:29 NAS smbd[6564]: [2019/11/25 09:20:29.251041, 0] ../lib/param/loadparm.c:1016(lpcfg_service_ok) Nov 25 09:20:29 NAS smbd[6564]: WARNING: No path in service RNXconfig - making it unavailable! Nov 25 09:20:30 NAS smbd[6564]: [2019/11/25 09:20:30.937883, 0] ../lib/param/loadparm.c:1016(lpcfg_service_ok) Nov 25 09:20:30 NAS smbd[6564]: WARNING: No path in service RNXconfig - making it unavailable! Nov 25 09:21:15 NAS clamd[4699]: SelfCheck: Database status OK. Nov 25 09:31:15 NAS clamd[4699]: SelfCheck: Database status OK. Nov 25 09:34:27 NAS msmtpq[6739]: mail for [ -C /etc/msmtprc info@inside-layout.de --timeout=60 ] : send was successful Nov 25 09:35:18 NAS apache_access[3652]: 000.000.000.00 "GET /admin/ HTTP/1.1" 401 Nov 25 09:35:21 NAS apache_access[3652]: Admin session started from 192.168.178.44 Nov 25 09:35:24 NAS apache_access[3652]: 000.000.000.00 "POST /dbbroker HTTP/1.1" 401 Nov 25 09:35:30 NAS msmtpq[6849]: mail for [ -C /etc/msmtprc here-stays@mymail.de --timeout=60 ] : send was successful Nov 25 09:35:45 NAS connmand[2782]: ntp: adjust (slew): +0.002666 sec Nov 25 09:36:15 NAS apache_access[3653]: ::1 "OPTIONS * HTTP/1.0" 200 Nov 25 09:36:45 NAS dbus[2765]: [system] Activating service name='org.opensuse.Snapper' (using servicehelper) Nov 25 09:36:45 NAS dbus[2765]: [system] Successfully activated service 'org.opensuse.Snapper' Nov 25 09:36:45 NAS snapperd[7122]: THROW: invalid config Nov 25 09:36:45 NAS snapperd[7122]: config '3' is invalid Nov 25 09:36:45 NAS snapperd[7122]: THROW: invalid config Nov 25 09:36:45 NAS snapperd[7122]: config '4' is invalid
(Ip Adress, mail and Nas-Name are changed for web-reasons :-) - greetings from a paranoid user :- )
So - that means i have to set up the whole thing from the scratch?
Or is there a nother possibility.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
