× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Weird Problems

f0rgiven
Aspirant

Weird Problems

ReadyNAs pro 4 with latest firmware and 4x 3TB drives. When i login to the admin console and click "shares", nothing shows there not even the default shares. If i use SSH to connect to the NAS, putty will close after i login. Using Linux SSH to connect to the NAS, it says "connection closed" immediately after login. A while back it seems like i remember the NAS acting like this when the filesystem was full. Is there any way to check the space left on the system?

Model: ReadyNASRNDP4220|ReadyNAS Pro 4
Message 1 of 38
mdgm-ntgr
NETGEAR Employee Retired

Re: Weird Problems

Sounds like you are trying to login to SSH as the admin user rather than the root user.

 

If you have SSH enabled and have logged in as 'root' you can run e.g.

 

# df -h

# df -i

 

Welcome to the community!

Message 2 of 38
f0rgiven
Aspirant

Re: Weird Problems

So it looks like the root filesystem is full:

 

#df /

Filesystem 1K-blocks Used Available Use% Mounted on
-                   4176564 4176564 0 100% /

 

How can i find what is taking up the space?

Message 3 of 38
mdgm-ntgr
NETGEAR Employee Retired

Re: Weird Problems

By using the du command.

 

e.g.

 

# du -csh /var/*
# du -csh /var/log/*

 

Note the df commands I provided in my previous post were to check for two different ways that the 4GB root partition may be full. Almost always it would be full one way, but not the other. There would be a different approach that would need to be taken if all the inodes were used up.

Message 4 of 38
f0rgiven
Aspirant

Re: Weird Problems

Thanks for your help! I found what was taking up the space. It was tmp. So i deleted some of the tmp files and restarted the NAS. Now, i can't connect via ssh (get connection refused). Not sure how to even get to it now without SSH. Is there a console that i can connect through?

Message 5 of 38
mdgm-ntgr
NETGEAR Employee Retired

Re: Weird Problems

Freeing up space is only the first step.

 

What was filling up /tmp ? Crashplan?

 

With your 4GB root partition full it's likely some config was corrupted.

 

There is tech support mode (a boot menu option). The credentials for logging into that using telnet are freely available on the web, but using that would have the same implications as using SSH.

It's likely your /etc/default/services file on the 4GB root partition is corrupt.

 

You could copy the default one from /etc/default/config/etc/default/services and add the lines

APACHESSL=1
SSH=1

 

Other config files may be corrupt too e.g. /etc/network/interfaces

Message 6 of 38
f0rgiven
Aspirant

Re: Weird Problems

Thanks! I have put the NAS into tech support mode and copied the services file to the suggested location. When  i looked at the services file located in /etc/default/services, nothing was in the file. If i looked at the default services file located in /etc/default/config/etc/default/services, it had text in it. So if i look at the interfaces file located in /etc/network, it DOES have the correct info in it. Should i reboot the NAS and see if SSH works now? I also started the RAID and mounted the "Data" drive.

 

Oh i realized i did not answer your question: about what was filling the tmp folder, we use a off-site backup software called Syncrify. This software stores its tmp and merge files in the tmp folder. I have since redirected the Syncrify storage location to a different section of the drive.

Message 7 of 38
mdgm-ntgr
NETGEAR Employee Retired

Re: Weird Problems

I suspected that had probably happened. As there was no room to write anything the file became empty.

 

There are other config files (of which the defaults are in /etc/default/config/ that could have got corrupted so you may wish to check some of those as well first).


You could then remember to sync your changes unmount the root volume, stop the RAID and reboot.

It is important to store as much as possible on the data volume rather than the 4GB root partition, but still keep an eye on what you put on the data volume from time to time.

Message 8 of 38
f0rgiven
Aspirant

Re: Weird Problems

I'm still confused as df / shows 100% useage. I moved the Syncrify tmp/merge files to /c/tmp. /c is the Data section right?

 

The NAs had a default share called backup created. I tried to delete that on and make one called Backup but it would not have any of it so i kept backup. I have put all the Syncrify backups into this backup folder. Is backup part of the Data drive? I think it is but for the life of me can't figure out what is taking up all the space on /

 

I should check the default files first..... When i do, i see a mysql/my.cnf file that i DON'T see in /etc/default. The problem with Syncrify, according to the log, is it can't connect to the DB. Does the default my.cnf go in the /etc/default folder?

Message 9 of 38
mdgm-ntgr
NETGEAR Employee Retired

Re: Weird Problems

Yes. Where did you move the files from again? Perhaps those were in a tmpfs or ramfs or already on the data volume.

You should look at e.g.

 

# cat /proc/mounts

 

What's filling up the root volume may be something else.

 

Have you tried both

# df -h

and
# df -i

as I suggested?

Message 10 of 38
f0rgiven
Aspirant

Re: Weird Problems

I moved the files from /tmp to /c/tmp (mv /tmp/*.* /c/tmp)

 

cat /proc/mounts
rootfs / rootfs rw 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
tmpfs /dev tmpfs rw,relatime,size=10240k,mode=755 0 0
devpts /dev/pts devpts rw,relatime,mode=600 0 0
/dev/md0 / ext3 rw,relatime,errors=continue,barrier=0,data=ordered 0 0
tmpfs /ramfs ramfs rw,relatime 0 0
tmpfs /USB tmpfs rw,relatime,size=16k 0 0
usbfs /proc/bus/usb usbfs rw,relatime 0 0
/dev/c/c /c ext4 rw,noatime,barrier=1,stripe=48,data=ordered,jqfmt=vfsv1,usrjquota=aquota.user,grpjquota=aquota.group 0 0

 

df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 16K 0 16K 0% /USB
/dev/c/c 8.2T 332G 7.8T 5% /c

 

df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
tmpfs 255692 1 255691 1% /USB
/dev/c/c 136970240 283556 136686684 1% /c

Message 11 of 38
StephenB
Guru

Re: Weird Problems

What about /dev/md0 ?

Message 12 of 38
f0rgiven
Aspirant

Re: Weird Problems

df /dev/md0
Filesystem 1K-blocks Used Available Use% Mounted on
- 10240 588 9652 6% /dev

 

Permissions are all jacked on existing shares and i suspect the mysql config is hosed which makes me worry about the existing DB's...........

I will just put this on the table; i can put it in tech support mode if you could remote to it and take a look?

Message 13 of 38
StephenB
Guru

Re: Weird Problems


@f0rgiven wrote:

df /dev/md0
Filesystem 1K-blocks Used Available Use% Mounted on
- 10240 588 9652 6% /dev


Something wrong here.  /dev/md0 should be 4176564 1K-blocks.  

 

It should be something more like this

# df /dev/md0
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/md0 4176564 1531416 2435488 39% /

 

What firmware are you running?

Message 14 of 38
f0rgiven
Aspirant

Re: Weird Problems

4.2.28

Message 15 of 38
StephenB
Guru

Re: Weird Problems

Very odd.  Maybe try cd /lib and then df .

 

Message 16 of 38
f0rgiven
Aspirant

Re: Weird Problems

df /lib
Filesystem 1K-blocks Used Available Use% Mounted on
- 4176564 4175164 0 100% /

Message 17 of 38
mdgm-ntgr
NETGEAR Employee Retired

Re: Weird Problems

You'll need to use the du command and check to make sure that what you find is on the 4GB root volume.

Message 18 of 38
f0rgiven
Aspirant

Re: Weird Problems

I'm not sure i understand. Run df /? which i have done. Also, did you guys see my edit on my few posts up? Permissions are totally jacked, can't get to any existing shares and i suspect the mysql conf file is gone?

Message 19 of 38
f0rgiven
Aspirant

Re: Weird Problems

Should i reinstall the OS or would that just break things more?

Message 20 of 38
StephenB
Guru

Re: Weird Problems


@f0rgiven wrote:

df /lib
Filesystem 1K-blocks Used Available Use% Mounted on
- 4176564 4175164 0 100% /


The OS volume remains full.

 

As mdgm says, you need to run du in the various folders in the OS partition, and find the files that have filled it up.

 

An OS reinstall will not help (and in fact will likely fail because there is no space).

Message 21 of 38
mdgm-ntgr
NETGEAR Employee Retired

Re: Weird Problems

The first step is to find what's filling the root volume and free some space

 

The next step is to fix broken config.

 

Only once you've done both of those an OS Re-install may be worth considering.

 

Doing an OS Re-install while the root volume is full will just make things worse.

Message 22 of 38
f0rgiven
Aspirant

Re: Weird Problems

I/m having a hard time figuring out where is the OS volume?

 

du -csh /*
0 /backup
3.6M /bin
332G /c
588K /dev
6.5M /etc
14M /frontview
8.0K /home
4.0K /initrd
25M /lib
2.2M /lib64
16K /lost+found
0 /media
4.0K /mnt
221M /opt
du: cannot access `/proc/10946/task/10946/fd/4': No such file or directory
du: cannot access `/proc/10946/task/10946/fdinfo/4': No such file or directory
du: cannot access `/proc/10946/fd/4': No such file or directory
du: cannot access `/proc/10946/fdinfo/4': No such file or directory
0 /proc
28K /ramfs
44K /root
4.8M /sbin
0 /sys
0 /Tek-Nerds
256K /tmp
0 /USB
198M /usr
67M /var
332G total

 

fdisk -l
fdisk: device has more than 2^32 sectors, can't use all of them
Found valid GPT with protective MBR; using GPT

Disk /dev/sda: 4294967295 sectors, 2047G
Logical sector size: 512
Disk identifier (GUID): 09b1b2a5-a63d-4a14-9885-00353d8705b0
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 5860533134

Number Start (sector) End (sector) Size Code Name
1 64 8388671 4096M 0700
2 8388672 9437247 512M 0700
3 9437248 5860529072 2790G 0700
fdisk: device has more than 2^32 sectors, can't use all of them
Found valid GPT with protective MBR; using GPT

Disk /dev/sdb: 4294967295 sectors, 2047G
Logical sector size: 512
Disk identifier (GUID): f1514482-3365-420c-bedf-264d7ba43a61
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 5860533134

Number Start (sector) End (sector) Size Code Name
1 64 8388671 4096M 0700
2 8388672 9437247 512M 0700
3 9437248 5860529072 2790G 0700
fdisk: device has more than 2^32 sectors, can't use all of them
Found valid GPT with protective MBR; using GPT

Disk /dev/sdc: 4294967295 sectors, 2047G
Logical sector size: 512
Disk identifier (GUID): 68e62ebb-685e-4a42-af66-247d86417f24
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 5860533134

Number Start (sector) End (sector) Size Code Name
1 64 8388671 4096M 0700
2 8388672 9437247 512M 0700
3 9437248 5860529072 2790G 0700
fdisk: device has more than 2^32 sectors, can't use all of them
Found valid GPT with protective MBR; using GPT

Disk /dev/sdd: 4294967295 sectors, 2047G
Logical sector size: 512
Disk identifier (GUID): e532ea7e-5ef2-426d-8b40-4ebcd6bd55be
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 5860533134

Number Start (sector) End (sector) Size Code Name
1 64 8388671 4096M 0700
2 8388672 9437247 512M 0700
3 9437248 5860529072 2790G 0700

Disk /dev/md0: 4293 MB, 4293906432 bytes
2 heads, 4 sectors/track, 1048317 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md0 doesn't contain a valid partition table
fdisk: device has more than 2^32 sectors, can't use all of them

Disk /dev/dm-0: 2199.0 GB, 2199023255040 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/dm-0 doesn't contain a valid partition table

Message 23 of 38
f0rgiven
Aspirant

Re: Weird Problems

It looks like i have 2 backup folders. 1 in /c and 1 in /backup. If i look inside both folders, they have the same folders and content? Frontview shows 1 share folder named backup..... I tried to mv -v dir1 dir2 /c/backup and got:

 

mv: `Commercial' and `/c/backup/Commercial' are the same file
mv: `DAG' and `/c/backup/DAG' are the same file
mv: `randy@lovenc.com' and `/c/backup/randy@lovenc.com' are the same file
mv: `Residential' and `/c/backup/Residential' are the same file
mv: `SDR' and `/c/backup/SDR' are the same file

 

Is 1 of the backup folders a symlink or something?

 

du -csh /backup/*
244G /backup/Commercial
42G /backup/DAG
45G /backup/randy@lovenc.com
399M /backup/Residential
6.1M /backup/SDR
330G total

 

du -csh /c/backup/*
244G /c/backup/Commercial
42G /c/backup/DAG
45G /c/backup/randy@lovenc.com
399M /c/backup/Residential
6.1M /c/backup/SDR
330G total

And i am not even sure /backup is the OS volume as the /backup folder is 330G and the OS volume is 4G.....

I guess i could mv the /backup folder and all contents to /c/some.other.folder and see what happens (i could always move the files back)....

And so i did mv the files as i stated above and then ls /backup = empty then ls /c/backup = empty. So 1 of these is a symlink of some type.

 

I'm waiting to hear back from 1 of you guys on how to identify where the OS volume is on the ReadyNAS Pro 4? It looks to be /dev/md0 from fdisk -l above but i wait to get specific instructions from 1 of you

Message 24 of 38
mdgm-ntgr
NETGEAR Employee Retired

Re: Weird Problems

/backup is a symlink to /c/backup

 

You can see this when doing e.g.

 

# ls -la /

Message 25 of 38
Top Contributors
Discussion stats
  • 37 replies
  • 4370 views
  • 0 kudos
  • 3 in conversation
Announcements