RN316 does no finish booting and OS install gives error "could not extracting files) last F

[StephenB]

---

@helio wrote:

3.9G var

---

This is where the problem is, so we need to delve deeper into that folder.

Now enter

cd /mnt/var
du -csh *

Mine looks like this:

104M backups
48M cache
0 cores
0 ftp
448M lib
0 local
4.0K lock
60M log
0 mail
8.0K netatalk
0 opt
22M readynasd
4.0K run
4.0K spool
0 tmp
4.0K www
682M total

[helio]

@StephenB , it looks to me like it is the Lib size:

149M backups
27M cache
388K cores
0 crash
0 ftp
3.6G lib
0 local
4.0K lock
77M log
0 mail
8.0K netatalk
0 opt
18M readynasd
4.0K run
4.0K spool
0 target
0 tmp
4.0K www
3.9G total

[helio]

@StephenB looked into the 3.6G lib folder
but cannot see what is taking the 3.6G :

root@3C62350D01347:/mnt/var# cd /mnt/lib
root@3C62350D01347:/mnt/lib# du -csh *
12K hdparm
40K init
4.0K libip4tc.so.0
32K libip4tc.so.0.1.0
4.0K libip6tc.so.0
32K libip6tc.so.0.1.0
4.0K libipq.so.0
12K libipq.so.0.0.0
4.0K libiptc.so.0
8.0K libiptc.so.0.0.0
4.0K libxtables.so.10
52K libxtables.so.10.0.0
20K lsb
1.3M modules
1.9M nut
48K security
44K startpar
5.9M systemd
164K terminfo
4.4M udev
15M x86_64-linux-gnu
3.4M xtables
33M total

[StephenB]

@helio wrote:

looked into the 3.6G lib folder
but cannot see what is taking the 3.6G :

root@3C62350D01347:/mnt/var# cd /mnt/lib

 You looked in the wrong lib folder.

Try 

cd /mnt/var/lib
du -csh *

I see

0       apache2
38M     apt
397M    clamav
12K     connman
0       dbus
11M     dpkg
0       ftp
0       initscripts
0       insserv
0       logrotate
4.0K    mdadm
0       misc
16K     nfs
0       nut
24K     pam
4.0K    readyNASVault.version
8.0K    replisync
2.6M    samba
0       snmp
128K    systemd
12K     ucf
0       update-rc.d
0       urandom
0       vim
448M    total

[helio]

@StephenB what is 3.0G glpi ?
root@3C62350D01347:/mnt/var# cd /mnt/var/lib
root@3C62350D01347:/mnt/var/lib# du -csh *
0 apache2
38M apt
524M clamav
52K connman
0 dbus
14M dpkg
0 ftp
3.0G glpi
0 initscripts
0 insserv
0 logrotate
4.0K mdadm
0 misc
30M mysql
0 mysql-files
0 nfs
0 nut
24K pam
0 php5
0 python
4.0K readyNASVault.version
8.0K replisync
2.6M samba
4.0K shellinabox
0 snmp
124K systemd
124K ucf
0 update-rc.d
0 urandom
0 vim
3.6G total

[StephenB]

---

@helio wrote:

3.0G glpi

This is the problem.  Did you ever try to install the GLPI app from the apps page?

[helio]

@StephenB I do not know what that is and I do not care at this point - I just want to be able to reinstall and recover my data.

I know that Plex cannot be uninstalled or updated from the store anymore.

Can we get rid of it?

[StephenB]

---

@helio wrote:

I do not know what that is and I do not care at this point - I just want to be able to reinstall and recover my data.

I know that Plex cannot be uninstalled or updated from the store anymore.

Can we get rid of it?

---

Not sure why you are talking about plex.

Somehow GPLI did get installed.  It's not an application I've ever used, and I don't know the method to properly uninstall it. It's possible that you uninstalled it already sometime in the past - and what we are seeing now are remnants left behind due to a poorly written uninstall script.  But I think it's more likely that it has been running in the background all along, and has been gradually increasing the size of /var/lib/gpli.

 but there is some risk that we'll leave some remnants behind that will do damage (possibly interfering with booting up).  

FWIW, GPLI uses PHP (and mySQL), so if this was ever running properly, then PHP must have been installed also.

This command will delete the big folder.  Do be very careful on the typing, as getting this bit wrong could remove folders that you need.  

rm -rf /var/lib/gpli

After you do this, type

cd //
umount /mnt
reboot

These commands will undo the temporary mounting of /mnt that we did before, and then reboot the NAS.  I believe that should work this time.  But there is some risk, since I don't know what GPLI will do if /var/lib/gpli is missing.

Once the NAS comes back up, you should immediately make a backup of all your files.  Then you should do a factory default, reconfigure the NAS, and then restore all the files from the backup.  That is because there are other folders (and potentially changes to NAS config files) that also need to be removed, and I am not certain how to safely remove them.

[helio]

@StephenB I saw it on the web, now I remember GLPI is an IT Department inventory application.

We will need to force the uninstall. here are the results:

rm: cannot remove '/var/lib/glpi/files/_sessions/sess_c1medjpofuh7i32bofmtd4mqu7': No space left on device
rm: cannot remove '/var/lib/glpi/files/_sessions/sess_3mrduebl7p61hnsu3qhtqdfmb3': No space left on device
rm: cannot remove '/var/lib/glpi/files/_sessions/sess_oav9i33v4f8caj32d5ot1dq9r4': No space left on device
rm: cannot remove '/var/lib/glpi/files/_sessions/sess_8ot3brvuraeqjam5941sqr1jb1': No space left on device
rm: cannot remove '/var/lib/glpi/files/_sessions/sess_s4bq72sq03bona13llbnu952j5': No space left on device
rm: cannot remove '/var/lib/glpi/files/_sessions/sess_abkvmd4kpn0fd4o3vivsio72l3': No space left on device
rm: cannot remove '/var/lib/glpi/files/_sessions/sess_2lqkddl9q639fpipgjnp8r1cq7': No space left on device
rm: cannot remove '/var/lib/glpi/files/_sessions/sess_r3te5ufefhgaunbmt771674do7': No space left on device
rm: cannot remove '/var/lib/glpi/files/_sessions/sess_89vtl2gcsnk05g2vgl7moo1ls0': No space left on device
rm: cannot remove '/var/lib/glpi/files/_sessions/sess_lg6510cufn5mlkl5l13167qub6': No space left on device
rm: cannot remove '/var/lib/glpi/files/_sessions/sess_msg4b2iskkkhdud8j20lccj0p5': No space left on device
rm: cannot remove '/var/lib/glpi/files/_sessions/sess_6su64jnkmjen3n20sgqpvuqs43': No space left on device
rm: cannot remove '/var/lib/glpi/files/_sessions/sess_3log16vb5at4kp2sff0o9jtpg7': No space left on device
^Xrm: cannot remove '/var/lib/glpi/files/_sessions/sess_9pm0r5gl6v452ujev1ek3297i4': No space left on device
rm: cannot remove '/var/lib/glpi/files/_sessions/sess_7tuck163sk0h6b77rbqqetvbg6': No space left on device
rm: cannot remove '/var/lib/glpi/files/_sessions/sess_cupii4nu2s9fh3cs5ob4sfr431': No space left on device

[helio]

Nota: Default user accounts are:
• glpi/glpi admin account,
• tech/tech technical account,
• normal/normal «normal» account,
• post-only/postonly post-only account.
Aviso: For obvious security concerns, you’ll have to delete or edit those accounts.
Before removing the glpi account, please make sure you have created another user with super-admin profile.
3.3. Install / Update 17

[helio]

--force did not not work:

root@3C62350D01347:/mnt/var/lib# rm -rf /var/lib/glpi --force
rm: cannot remove '/var/lib/glpi/files/_uploads': No space left on device
rm: cannot remove '/var/lib/glpi/files/_cache': No space left on device
rm: cannot remove '/var/lib/glpi/files/_log': No space left on device
rm: cannot remove '/var/lib/glpi/files/_cron': No space left on device
rm: cannot remove '/var/lib/glpi/files/_dumps': No space left on device
rm: cannot remove '/var/lib/glpi/files/_graphs': No space left on device
rm: cannot remove '/var/lib/glpi/files/_lock': No space left on device
rm: cannot remove '/var/lib/glpi/files/_sessions/sess_p8cf0j3pqkufhud0t8fk568sp3': No space left on device
rm: cannot remove '/var/lib/glpi/files/_sessions/sess_3vm5rbesuud5k508i0udqumqf4': No space left on device

[StephenB]

---

@helio wrote:

now I remember GLPI is an IT Department inventory application.

We will need to force the uninstall. here are the results:

rm: cannot remove '/var/lib/glpi/files/_sessions/sess_c1medjpofuh7i32bofmtd4mqu7': No space left on device

...

FWIW, the -f in my command is the same as --force.

I think your issue is that you have no free metadata space, so nothing can be deleted.

Can you try

echo > /var/lib/glpi/files/_sessions/sess_c1medjpofuh7i32bofmtd4mqu7

and see if you get the same error?

This won't delete the file, but it will truncate it to a size of 0.

[helio]

root@3C62350D01347:/mnt/var/lib# echo > /var/lib/glpi/files/_sessions/sess_c1medjpofuh7i32bofmtd4mqu7
bash: /var/lib/glpi/files/_sessions/sess_c1medjpofuh7i32bofmtd4mqu7: No space left on device

[StephenB]

---

@helio wrote:

root@3C62350D01347:/mnt/var/lib# echo > /var/lib/glpi/files/_sessions/sess_c1medjpofuh7i32bofmtd4mqu7
bash: /var/lib/glpi/files/_sessions/sess_c1medjpofuh7i32bofmtd4mqu7: No space left on device

---

We are running out of options

Try

cd //
umount /mnt
mount --bind -nodatacow // /mnt
btrfs balance start -dusage=0 -musage=0 /
btrfs balance start -dusage=0 -musage=100 /

Likely the last command will give an error.

Then try this one 

rm -rf /mnt/var/lib/glpi

[helio]

@StephenB 

root@3C62350D01347:/mnt/var/lib# cd //
root@3C62350D01347://# umount /mnt
root@3C62350D01347://# mount --bind -nodatacow // /mnt
root@3C62350D01347://# btrfs balance start -dusage=0 -musage=0 /
Done, had to relocate 0 out of 15 chunks
root@3C62350D01347://# btrfs balance start -dusage=0 -musage=100 /
ERROR: error during balancing '/': No space left on device
There may be more info in syslog - try dmesg | tail
root@3C62350D01347://# ^C
root@3C62350D01347://# dmesg | tail
[ 227.222284] disk 0, o:1, dev:sdb5
[ 227.222288] disk 1, o:1, dev:sdc5
[ 227.222292] disk 2, o:1, dev:sda5
[ 227.222295] disk 3, o:1, dev:sdd5
[ 227.222299] disk 4, o:1, dev:sde5
[ 227.222536] md125: detected capacity change from 0 to 5000299151360
[ 227.243720] EXT4-fs (md0): VFS: Can't find ext4 filesystem
[ 227.243915] BTRFS: device label 0ed5edda:root devid 1 transid 2603287 /dev/md0
[ 227.244525] BTRFS info (device md0): has skinny extents
[28355.767379] BTRFS info (device md0): 6 enospc errors during balance
root@3C62350D01347://# ^C
root@3C62350D01347://# rm -rf /mnt/var/lib/glpi
rm: cannot remove '/mnt/var/lib/glpi/files/_uploads': No space left on device
rm: cannot remove '/mnt/var/lib/glpi/files/_cache': No space left on device
rm: cannot remove '/mnt/var/lib/glpi/files/_log': No space left on device
rm: cannot remove '/mnt/var/lib/glpi/files/_cron': No space left on device
rm: cannot remove '/mnt/var/lib/glpi/files/_dumps': No space left on device
rm: cannot remove '/mnt/var/lib/glpi/files/_graphs': No space left on device
rm: cannot remove '/mnt/var/lib/glpi/files/_lock': No space left on device
rm: cannot remove '/mnt/var/lib/glpi/files/_sessions/sess_p8cf0j3pqkufhud0t8fk568sp3': No space left on device
rm: cannot remove '/mnt/var/lib/glpi/files/_sessions/sess_3vm5rbesuud5k508i0udqumqf4': No space left on device
^C

[StephenB]

Unfortunately I don't see a way past this.

Paid Netgear support could completely rebuild the OS partition, but that is not something I've ever tried.

We could mount a USB disk on the NAS, and copy data from the data volume to the USB drive.

[helio]

@StephenB 

We could mount a USB disk on the NAS, and copy data from the data volume to the USB drive

It sounds good, let's do that. Please.

[StephenB]

---

@helio wrote:

We could mount a USB disk on the NAS, and copy data from the data volume to the USB drive

It sounds good, let's do that. Please.

---

It'd be good to format the disk as NTFS, so you could also read it on a PC (Windows or Mac).

I'd need to do a test here to make sure I give you the right mount command.   That might take a while (I had an accident earlier this morning, and am dealing with a broken collarbone).  So it's unlikely to be today.  Perhaps @Sandshark or @mdgm will chime in to speed things up 

[helio]

@StephenB I am so sorry to read that.

I hope you are ok and not in too much pain.

Don't worry about it for now, take good care and get well first.

My best wishes of a prompt recovery!

[StephenB]

---

@helio wrote:

I hope you are ok and not in too much pain.

---

I am ok, and pain is managable so far. But I'll be one-handed for a bit, and not very mobile.

[Sandshark]

I've not done a lot of poking around in that mode and don't use USB drives with my NAS, so I'm not the guy to help on this one.

[StephenB]

---

@Sandshark wrote:

I've not done a lot of poking around in that mode and don't use USB drives with my NAS, so I'm not the guy to help on this one.

---

I know you need to use mount –t ufsd but it would be better to try it, and make sure it works in tech support mode.

[helio]

@StephenB Hey, I hope you are feeling better, get well.

@Sandshark hey, I tried but it did not work, am I doing it wrong?

root@3C62350D01347:/mnt# -ufsd -t

root@3C62350D01347:/mnt# -t ufsd
bash: -t: command not found
root@3C62350D01347:/mnt# ufsd -t
bash: ufsd: command not found

thanks

[StephenB]

---

@helio wrote:

I tried but it did not work, am I doing it wrong?

root@3C62350D01347:/mnt# -ufsd -t

root@3C62350D01347:/mnt# -t ufsd
bash: -t: command not found
root@3C62350D01347:/mnt# ufsd -t
bash: ufsd: command not found

---

Yes you are doing it wrong.  But I'll need to play with these commands a bit in order to get it right.

[mdgm]

From the chroot you would need to modprobe the ufsd driver

# modprobe ufsd

Then if you use the correct command you should be able to mount the drive.