ReadyNAS 516 stuck at booting on systemd-journald.service #28252379

[cmassey]

I noticed some MySQL errors (unable to write data) in some logs and thought it had been a while since a restart so I went to restart it.

After 30 or so minutes of being stuck on NAS Rebooting now I chose to power cycle the unit.

It appeared to boot but I was unable to access shares or the admin web page.  The front touch panel was also unresponsive.

So again I power cycled the unit and now it is freezing at start up at systemd-journald.service

Boot sequence gets to about 92% then I see a failed errors for a second (can't catch what it is), the sticks on booting...

When pushing the power button it reports being stuck at the error above (systemd-journald.service)

Power cycling and booting in to read only mode doesn't work either, any suggestions before calling support tomorrow?

[jak0lantash · Accepted solution]

MySQL... mmm, maybe a full root (OS volume being 100% full). Depending on how you installed MySQL, it often installs itself on the OS volume, with has only 4GB capacity. When MySQL gets big enough, then the OS volume gets 100% full and "break" the OS. Sometimes, there is no significant damage to the OS and moving the MySQL data to the data volume (+ symlink back) is enough, sometimes it's necessary to install a fresh OS volume. In both cases, your data volume remains untouched (so data is OK).

Can you download the logs? In btrfs.log and volume.log, you should see info about md0.

Now, it could be something completely different, but without more info, that's the first thing I can think of.

[jak0lantash · Accepted solution]

MySQL... mmm, maybe a full root (OS volume being 100% full). Depending on how you installed MySQL, it often installs itself on the OS volume, with has only 4GB capacity. When MySQL gets big enough, then the OS volume gets 100% full and "break" the OS. Sometimes, there is no significant damage to the OS and moving the MySQL data to the data volume (+ symlink back) is enough, sometimes it's necessary to install a fresh OS volume. In both cases, your data volume remains untouched (so data is OK).

Can you download the logs? In btrfs.log and volume.log, you should see info about md0.

Now, it could be something completely different, but without more info, that's the first thing I can think of.

[cmassey]

Update:

1. OS Reinstall hasn't fixed the issue.

2. Disk check appears to finish with no response to say it has finished and the disks look like a Christmas tree flashing Red and Blue

3. I can telnet to the unit in tech support mode but obviously I don't have the access.

Logging a support request now

[cmassey]

Thanks for the reply.

It was the MySQL app by SuperPoutin (I think that is the spelling)

The Database stores info for SABNZDB & KODI(XBMC)

I cannot recall where the database is stored.  But I wouldn't think it would be 4GB but you never know...

I have no way of getting the logs at this stage.

Talking to support now.

[jak0lantash]

OK, let us know how it goes.

[mdgm-ntgr]

Looking at the case notes it turns out the 4GB root volume was actually filled by huge syslog log files. You must have an app installed that put rsyslog on.

[jak0lantash]

So I was right while being wrong 

[cmassey]

Thanks guys.

All good now.  Just doing some cleaning up now and checking some other apps to hopefully make sure it doesn't happen again.

I've found out the root cause of the issue and it was virtual box that was creating a heap of syslog and kern.log errors.

VBoxNetFlt: Failed to allocate packet buffer, dropping the packet.  https://www.virtualbox.org/ticket/15569

So just about to do OS upgrade from 6.5 to 6.6.1

Then I'll upgrade VBox from 5.0x to 5.1.8

Can anyone confirm if the GPL file from 6.6.0 works with 6.6.1?

ReadyNASOS_V6.6.0_WW_src

PS.  Big thanks to the Netgear employee from Ireland 🙂

[mdgm-ntgr]

6.7.0 is coming soon so at this point you may wish to wait for that release and update directly to that.

[cmassey]

It's all good.  I've now upgraded to 6.6.1

VirtualBox is updated too and now the numerous errors have stopped.

I will wait to see feedback on 6.7.0 before I upgrade unless there are some really good reasons to upgrade sooner.

And yes for anyone still reading this thread the GPL for 6.6.0 works with 6.6.1.

Thanks again mdgm