Readynas 424 SSH service not starting (no error)

[Killerjerick]

Hi all, I updated from 6.10.3 -> 6.10.4 today, against my better judgement (I've had issues with updating firmware in the past) and again, I'm having issues, the firmware update screwed with mono, so I had to SSH in to fix it, I got it working and rebooted my NAS, unfortunately more problems arose, at which point I had no SSH access, whenever I try to enable it under "services" it just doesn't do anything, it doesn't turn on, there's no message box, no logs, nothing. 

I tried an OS reinstall using the boot option, to no avail, I have a feeling it's something to do with OS storage being full but I've no idea how to clear it, I could telnet in using the Netgear tech support boot option but didn't really know where to go from there

Any help would be greatly appreciated, thank you.

[StephenB]

Are you able to download the log zip file from the web ui?

[Killerjerick]

Hi, yeah, I am, I managed to download the log file, is there somewhere I should upload it? From what I can tell it thinks it's starting fine, and when I try to enable something else it realises it's actually disabled and says so in the log, and as I found out last night, it looks like all services have this issue, not just SSH.

[StephenB]

---

@Killerjerick wrote:

Hi, yeah, I am, I managed to download the log file, is there somewhere I should upload it?

You don't want to post the full log zip publicly.  You can request that the mods review it for you though.  You upload it to a cloud service (dropbox, google drive, etc), and then send them ( @JohnCM_S or @Marc_V ) a private message (PM) asking them to check it. Includes a download link to the log zip, and a link to this forum thread.  

You send a PM using the envelope icon in the upper right of the forum page.

---

@Killerjerick wrote:

I have a feeling it's something to do with OS storage being full

---

Look in volume.log (in the log zip).  Scroll down, and you will find a section like this:

=== df -h ===
Filesystem      Size  Used Avail Use% Mounted on
udev             10M  4.0K   10M   1% /dev
/dev/md0        4.0G  703M  3.0G  19% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
...

The /dev/md0 line tells you how full the OS partition is.  Generally it is less than 25% full.

[Killerjerick]

Hi Stephen, I attempted to contact JohnCM, but haven't heard back, can only assume he's busy. This is what I see in the volume.log

=== df -h ===
Filesystem      Size  Used Avail Use% Mounted on
udev             10M  4.0K   10M   1% /dev
/dev/md0        4.0G  2.6G  1.2G  70% /
tmpfs           980M   20K  980M   1% /dev/shm
tmpfs           980M  3.8M  976M   1% /run
tmpfs           490M  1.4M  489M   1% /run/lock
tmpfs           980M     0  980M   0% /sys/fs/cgroup

I'm guessing that I was incorrect, doesn't look like the root dir is full, although it is more full than it should be. Is there anywhere else I can poke around to see if I can find something that doesn't add up?

Thank you 🙂

[StephenB]

You could look in system.log and kernel.log for errors around the time you tried starting ssh from system->settings.

Also, systemctl-status.log should show that ssh.service is running (or not).  Systemctl-journal.log is probably also worth a look.

BTW, if you use the @ (e.g., @JohnCM_S ), then forum user (including mods) should get a notification of some kind.  That might help get attention.

[Killerjerick]

Hey @StephenB It's been over a month and no response from either mod, I assume they are both too busy right now, what's my next course of action? Ideally I'd love to not shell out the $$ for netgear support, but is there anything else I can do from my end? I've checked both of the logs you suggested and nothing jumps out at me, but my experience in linux in particular is lacking, I've checked and I seem to be able to install apps from the netgear "upload" option, but if my problem stems from an incorrect install from a particular app, I'm not able to uninstall it, the only thing that I did around the time of the issues was attempt to install a full version of FFMPEG, which is why in my case I assumed I had accidentally installed it to the OS partition, but that seems to not be the case, judging from log files.

I'm hesitant to attempt to install ffmpeg again using the upload function as I'm positive that'll mess something up.

Thanks in advance.

[rn_enthusiast]

I am thinking browser cache to be honest. Did you try and clear all browser history and cache and then try to enable SSH from the Web UI again after that? The browser cache often messes with me after updates to newer firmwares.

This issue sounds like something I have seen a bunch of times, TBH.

You can go in over Tech-Support mode and re-enable SSH if all comes to all but start with attacking the browser cache and let us know (even try different browsers as well). If you hit a dead-end let me know I will throw you some commands for Tech Support mode.

[Killerjerick]

Hey @rn_enthusiast I indded did try that, including using Edge, Chrome, Firefox and my phone with no luck, thanks anyway.

Those commands would be wonderful, I successfully telnetted into the nas with the netgear tech support mode, but I didn't know where to go from there, and stopped myself short of causing irreversible damage. My limited knowledge has gotten me into a pickle once before but I managed to solve that myself (just linux permission based stuff).

[rn_enthusiast]

Hi again

The below is at your own risk. Making mistakes can have consequences, as you know yourself 🙂 If you see errors or unexpected behaviour then stop and think before proceeding. This should be the steps to enable SSH, if my memory serves me well.

Once you are in over tech support mode, you can start the OS raid manually:

mdadm --assemble /dev/md0 /dev/sd[a-z]1

You should see md0 running now:

cat /proc/mdstat

Hereafter, mount the OS volume:

mount /dev/md0 /sysroot

You should see it mounted to /sysroot. You can check with:

mount

Now, use "vi" to change the default services file in order to enable SSH.

vi /sysroot/etc/default/services

Find these 5 lines in the file and change them to the same values as per below.

SSH=1
SSHPORT=22
SSH_UI_ENABLE=1
REMOTE_ACCESS_SSH_PORT=0
SSH_PASSWORD_AUTHENTICATION=1

Save the changes and exit the "vi" editor.

Un-mount the OS partition

umount /sysroot

Stop the OS raid:

mdadm --stop /dev/md*

Check that the raid is stopped:

cat /proc/mdstat

Flush memory to disk and reboot the NAS:

sync
reboot -fn

Cheers

[Sandshark]

---

@Killerjerick wrote:

the only thing that I did around the time of the issues was attempt to install a full version of FFMPEG, which is why in my case I assumed I had accidentally installed it to the OS partition, but that seems to not be the case, judging from log files.

Are you saying you didn't successfully install FFMPEG at all, or that you think you didn't install it to the OS partition.  Because if you did successfully install it via SSH and a .deb package, it most definitely would have installed to the OS partition.  While that alone shouldn't fill your OS partition, it could be pointing to it as workspace, which could become problematic if you've actually used it.

But another possibility is that this installation's dependancies installed an update to a package already in the NAS with one that's incompatible with the Netgear portion or over-wrote a configuration file that it needs.  While an OS re-install will, I believe, restore most Debian packages that are a part of the OS to their proper version, it doesn't over-write all configuration files because it doesn't want to mess with your properly installed apps.

So while you are in support mode, it would be good to still check that the OS partition isn't too full.

[mdgm]

As well as editing the default services file you may also need to create a symlink whilst the OS partition is mounted (if it doesn't already exist):

ln -s /lib/systemd/system/ssh.service /sysroot/etc/systemd/system/multi-user.target.wants/ssh.service

[StephenB]

You might also need to chroot in order to get the commands you want to use.

Instead of starting with the mdadm-assemble command suggested by @rn_enthusiast you could start with 

# rnutil chroot

That should start RAID, mount the OS partition, and chroot in one step.

[mdgm]

Going into the chroot is not needed for rn_enthusiast's commands or the one I mentioned. You can do it if you want, once you chroot into the sysroot you would remove references to sysroot in any of the commands until you exit the chroot again.

[rn_enthusiast]

You are changing a config file. No need for chroot to do that 🙂 It would just complicate matters as you need to exit out of it again, etc.

Also, the rnutil chroot command will start the data raid. There isn't really a need for that.

[Killerjerick]

Hi @rn_enthusiast thanks again for all your help, unfortunately it looks like my problem is worse than originally thought, your first command fails every time with the error

"mdadm: No super block found on /dev/sdc1 (expected magic a92b4efc, got 07020701)

mdadm: no RAID superblock on /dev/sdc1

mdadm: /dev/sdc1 has no superblock - assembly aborted" 

is there something else I should be doing? Sorry for the late reply, I can only really do this after college between 9-11pm AEST (Timezones suck)

[Killerjerick]

---

@Sandshark wrote:

---

@Killerjerick wrote:

the only thing that I did around the time of the issues was attempt to install a full version of FFMPEG, which is why in my case I assumed I had accidentally installed it to the OS partition, but that seems to not be the case, judging from log files.

Are you saying you didn't successfully install FFMPEG at all, or that you think you didn't install it to the OS partition.  Because if you did successfully install it via SSH and a .deb package, it most definitely would have installed to the OS partition.  While that alone shouldn't fill your OS partition, it could be pointing to it as workspace, which could become problematic if you've actually used it.

I didn't successfully install FFMPEG at all, although I can't exactly check that without SSH (I believe) I'll definitely check the disk space if I manage to successfully mount the drive.

[StephenB]

Have you made a backup of the NAS?  

It might be more direct to just do a factory default and start over.

[Killerjerick]

Unfortunately, there's no feasible way to backup 12tb of collected media, is there no way to just reinstall the os while keeping the data?

[rn_enthusiast]

From TS mode, what does this give you?

cat /proc/partitions

And what happens if you run:

rnutil chroot

[Killerjerick]

various partitions including

sda

sda1

sda2

sda3

sdb

sdb1

sdb2

sdb3

sdc

sdc1

md0

md1

md127

rnutil chroot seems to throw me into "root@<serialnumber>:/#"

[StephenB]

---

@Killerjerick wrote:

Unfortunately, there's no feasible way to backup 12tb of collected media...

---

There is, and I do back up my own NAS (which has a similar amount of data). 

I back up to other ReadyNAS, but you can purchase a 16 TB USB drive for ~$300 USD.  For instance, https://www.bestbuy.com/site/wd-easystore-16tb-external-usb-3-0-hard-drive-black/6427998.p?skuId=642...

[StephenB]

---

@Killerjerick wrote:

rnutil chroot seems to throw me into "root@<serialnumber>:/#"

---

I think @rn_enthusiast is wondering if that command mounted md0 (the OS partition).

You can easily check that by running mount.

If it is mounted, then cat /proc/mdstat would give you more information on what is going on with sdc.

[mdgm]

---

@Killerjerick wrote:

Hi @rn_enthusiast thanks again for all your help, unfortunately it looks like my problem is worse than originally thought, your first command fails every time

---

This is why givng instructions on how to deal with problems like this can be problematic. There are a bunch of other commands that we would nearly always run when checking a system and we get used to seeing problems like this.

No superblock is bad, but the 4GB root volume uses raid-1 so one disk isn't too bad. It does warrant further investigation to get an idea as to whether there's a problem with that disk. For starters

smartctl --all /dev/sdc

Would tell you the SMART stats from that disk.

You can also examine the raid device underneath the 4GB root volume using e.g.

for i in /dev/sd[a-z]1; do echo $i;  mdadm -E $i | egrep "Creat|Update|State|Raid"; done;

or

mdadm --examine /dev/sd[a-z]1 | egrep -i 'sd|event|uuid|recovery|super|role|update'

@Killerjerick wrote:
Sorry for the late reply, I can only really do this after college between 9-11pm AEST (Timezones suck)

---

Different time zones is expected with an international community.

@Killerjerick wrote:

rnutil chroot seems to throw me into "root@<serialnumber>:/#"

---

That's what you expect to see when you chroot in. If you are in the chroot instead of

vi /sysroot/etc/default/services

You would do e.g.

vi /etc/default/services

You should still look for the lines @rn_enthusiast mentioned, bearing in mind that as above some of the commands are slightly different when you are inside the chroot.

[Killerjerick]

smartctl --all /dev/sdc

throws

/dev/sdc: Unknown USB bridge [0x13fe:0x5200 (0x100)]
Please specify device type with the -d option.

Use smartctl -h to get a usage summary

while

for i in /dev/sd[a-z]1; do echo $i;  mdadm -E $i | egrep "Creat|Update|State|Raid"; done;

throws

  Creation Time : Mon Sep 23 18:19:13 2019
     Raid Level : raid1
   Raid Devices : 2
          State : clean
    Update Time : Tue Mar  2 20:56:40 2021
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb1
  Creation Time : Mon Sep 23 18:19:13 2019
     Raid Level : raid1
   Raid Devices : 2
          State : clean
    Update Time : Tue Mar  2 20:56:40 2021
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)

Thank you for bearing with me thus far 🙂