ReadyNAS Ultra4 - Possible Corrupted Firmware - No Access to Shares

[jak0lantash]

---

@itsonlycomputer wrote:

# cd /var
# du -h -d1
12.0K   ./run
4.0K    ./log
16.0K   .

Wait you're not in the right place, you did "cd /var" instead of "cd var", you're out of the OS volume.

To get back to where you were: 

cd /sysroot/var

[jak0lantash]

---

@mdgm wrote:

On OS6 one would run start_raids, so it can be a bit confusing remembering the subtle difference in command to enter at times.

---

And using autocompletion every day doesn't help remembering 😉

And posting instructions past midnight either ^^ Even though I proofread it like four times!

[StephenB]

---

@mdgm wrote:

On OS6 one would run start_raids, so it can be a bit confusing remembering the subtle difference in command to enter at times.

---

Yes.  And even if you do remember, it is easy to type something wrong.

[itsonlycomputer]

Not sure why my reply keeps getting deleted,  I am including an image of the last steps.   Where do we go from here?

Thanks

[VAR FOLDER]

8.0K    ./lock
4.0K    ./ftp
24.6M   ./log
12.7M   ./run
4.0K    ./nmbd
54.9M   ./backups
17.1M   ./cache
4.0K    ./local
4.0K    ./opt
40.3M   ./lib
8.0K    ./replicate
4.0K    ./mail
3.2G    ./tmp
4.0K    ./netatalk
132.0K  ./spool
3.3G    .

[CACHE FOLDER]

 # du -h -d1
680.0K  ./samba
4.0K    ./apt
9.0M    ./frontview
1.9M    ./cups
4.0K    ./dbconfig-common
4.0K    ./news
492.0K  ./exim
4.0K    ./mysql
24.6M   .

[jak0lantash]

---

@itsonlycomputer wrote:

[VAR FOLDER]
3.2G    ./tmp

There (/sysroot/var/tmp)

[itsonlycomputer]

ok,  I'm there,  what next?   [STOPPING HERE FOR NEXT STEPS]

# cd /sysroot/var/tmp
# du -h -d1
3.2G    .
# ls -alh
drwxrwxrwt    2    4.0K Apr 17 12:52 .
drwxr-xr-x   17    4.0K Sep 15  2009 ..
-rw-------    1   21.6M Dec  5 13:48 sort0HXBI1
-rw-------    1   21.4M Mar 23  2016 sort0j9CIj
-rw-------    1   21.2M Apr  3  2016 sort192xcR
-rw-------    1   21.6M Jul 26  2016 sort1nfHIp
-rw-------    1   21.6M Jul 26  2016 sort1pzCav
-rw-------    1   21.2M Mar 12 15:11 sort2e2O0c
-rw-------    1   21.3M Mar 23  2016 sort2xWn9g
-rw-------    1   21.4M Jul 25  2016 sort3CgJIV
-rw-------    1   21.6M Jul 25  2016 sort3q6ENr
-rw-------    1   21.6M Apr 16  2016 sort4KcmCV
-rw-------    1   21.5M Mar 12 13:35 sort4befgS
-rw-------    1   21.3M Jul 26  2016 sort4nwX0H
-rw-------    1   21.8M Apr  9 12:48 sort4oRAGD
-rw-------    1   21.4M Apr  3  2016 sort5Gk4Z7
-rw-------    1   20.5M Mar 24 12:35 sort5xecrP
-rw-------    1   21.8M Apr 16  2016 sort6GOVJe
-rw-------    1   19.9M Dec  5 13:52 sort6nlv1K
-rw-------    1   21.8M Mar 12 13:01 sort6oYUHj
-rw-------    1   21.8M Mar 24 12:56 sort6zFjqB
-rw-------    1   21.4M Apr 16  2016 sort8ayKYJ
-rw-------    1   19.5M Apr 16  2016 sort8y94qk
-rw-------    1   21.4M Apr  3  2016 sort9JN5vI
-rw-------    1   22.0M Apr  3  2016 sort9g7PAh
-rw-------    1   21.2M Mar 12 12:58 sortA2vVa4
-rw-------    1   21.4M Apr 16  2016 sortAGdG0x
-rw-------    1   21.5M Apr  9 12:49 sortAbmu9u
-rw-------    1   21.2M Jul 25  2016 sortBBIlWb
-rw-------    1   21.8M Mar 24 12:56 sortBIccn1
-rw-------    1   21.4M Jul 25  2016 sortBPwmZe
-rw-------    1   21.9M Mar 12 15:21 sortC3mIAR
-rw-------    1   21.7M Mar 12 12:59 sortCf16qF
-rw-------    1   21.2M Dec  5 14:10 sortEgAUSy
-rw-------    1   21.7M Mar 12 12:58 sortEhTXhS
-rw-------    1   21.8M Mar 12 13:02 sortErxpXr
-rw-------    1   21.4M Jul 25  2016 sortFAxg9O
-rw-------    1   21.8M Mar 24 12:57 sortFY9tqM
-rw-------    1   20.3M Mar 24 12:59 sortFa3P7j
-rw-------    1   20.5M Jul 26  2016 sortGCn1gc
-rw-------    1   21.6M Dec  5 13:47 sortGQfAYI
-rw-------    1   19.9M Jul 25  2016 sortHkuND2
-rw-------    1   21.7M Mar 12 14:53 sortIY6DlD
-rw-------    1   21.0M Jul 25  2016 sortJvID8Q
-rw-------    1   21.6M Mar 23  2016 sortK1R251
-rw-------    1   21.6M Apr 16  2016 sortKtxi6c
-rw-------    1   21.7M Mar 24 12:54 sortKxyEqo
-rw-------    1   21.0M Apr  3  2016 sortL5sss9
-rw-------    1   21.5M Mar 24 13:19 sortLcb5Vr
-rw-------    1   21.7M Mar 24 12:56 sortMcjOZW
-rw-------    1   19.9M Jul 26  2016 sortNSs6hX
-rw-------    1   21.4M Jul 25  2016 sortNa6sbE
-rw-------    1   21.3M Apr 16  2016 sortOPvJ3F
-rw-------    1   21.8M Apr  9 12:47 sortOfjSeS
-rw-------    1   21.5M Mar 24 12:58 sortQRK7Ey
-rw-------    1   21.5M Mar 24 13:28 sortQYYxhL
-rw-------    1   21.4M Jul 26  2016 sortR1FZ09
-rw-------    1   21.4M Jul 25  2016 sortRcebSN
-rw-------    1   21.0M Mar 24 12:34 sortRsfFbv
-rw-------    1   21.2M Mar 23  2016 sortS7OMjl
-rw-------    1   21.6M Apr 16  2016 sortSAFPgO
-rw-------    1   21.3M Mar 12 13:06 sortSCKiEk
-rw-------    1   19.8M Mar 23  2016 sortSN1PiC
-rw-------    1   21.0M Apr 16  2016 sortSiX13t
-rw-------    1   21.9M Jul 25  2016 sortT80iDs
-rw-------    1   20.3M Apr  9 12:50 sortU2YQlj
-rw-------    1   20.2M Mar 12 13:08 sortU9FgUb
-rw-------    1   21.7M Apr 16  2016 sortU9RLYc
-rw-------    1   21.0M Mar 24 12:35 sortUApUtR
-rw-------    1   21.8M Mar 12 13:03 sortUJUuza
-rw-------    1   21.6M Apr  3  2016 sortVDENFD
-rw-------    1   21.6M Jul 25  2016 sortVrqXoI
-rw-------    1   21.4M Jul 26  2016 sortVt5YoK
-rw-------    1   21.6M Dec  5 13:48 sortWu4MKA
-rw-------    1   21.3M Jul 25  2016 sortXRqthM
-rw-------    1   20.7M Mar 24 12:40 sortXhOxR8
-rw-------    1   21.6M Mar 12 15:16 sortY4o9Sm
-rw-------    1   21.3M Dec  5 14:20 sortY59uBq
-rw-------    1   20.5M Apr 16  2016 sortYKc2mP
-rw-------    1   22.0M Mar 12 15:19 sortYgmbl7
-rw-------    1   21.3M Jul 25  2016 sortZ6Zblj
-rw-------    1   21.1M Jul 25  2016 sortZm1j0C
-rw-------    1   21.5M Apr  9 13:20 sortaDiOoa
-rw-------    1   21.5M Mar 12 13:48 sortaEXuzn
-rw-------    1   20.4M Mar 24 12:35 sortaLmGVb
-rw-------    1   21.3M Mar 24 12:52 sortacWwdM
-rw-------    1   21.3M Mar 23  2016 sortaz34ur
-rw-------    1   21.6M Apr  3  2016 sortbW9GZF
-rw-------    1   21.8M Apr  9 12:47 sortcY3Cgs
-rw-------    1   21.4M Mar 23  2016 sortcaOZZZ
-rw-------    1   21.7M Apr  9 12:45 sortcc3P2u
-rw-------    1   21.6M Jul 25  2016 sortdYeVnA
-rw-------    1   21.2M Jul 26  2016 sortdwSv0P
-rw-------    1   21.0M Mar 23  2016 sorteD0beO
-rw-------    1   21.4M Apr  3  2016 sortfW8S3D
-rw-------    1   21.4M Apr  3  2016 sortfuJf1j
-rw-------    1   21.3M Mar 24 13:08 sortg0Dxh6
-rw-------    1   21.4M Apr 16  2016 sortgRr9p3
-rw-------    1   21.6M Mar 12 14:05 sortgdRfuG
-rw-------    1   21.4M Dec  5 13:46 sortgqMQUD
-rw-------    1   21.3M Apr 16  2016 sorti6uLSV
-rw-------    1   21.8M Mar 12 13:03 sorti7rmSF
-rw-------    1   21.7M Apr  9 12:46 sortiDCn3P
-rw-------    1   21.0M Apr 16  2016 sortiRscNw
-rw-------    1   21.2M Apr 16  2016 sortihWfHv
-rw-------    1   20.7M Mar 23  2016 sortioBl47
-rw-------    1   21.2M Apr  9 12:59 sortk0d5Yb
-rw-------    1   21.6M Jul 26  2016 sortkhNLri
-rw-------    1   21.6M Mar 23  2016 sortkiXxgq
-rw-------    1   21.4M Apr 16  2016 sortkjf2nk
-rw-------    1   21.6M Mar 23  2016 sortmD88Fg
-rw-------    1   21.2M Mar 12 12:56 sortmKcdBA
-rw-------    1   21.7M Apr  9 12:46 sortmTNUWv
-rw-------    1   21.1M Mar 23  2016 sortmiAhb9
-rw-------    1   21.0M Apr 16  2016 sortmzOChc
-rw-------    1   21.4M Apr 16  2016 sorto3CfSM
-rw-------    1   21.0M Dec  5 14:01 sorto4pGrd
-rw-------    1   21.7M Mar 12 14:29 sortoaEwhN
-rw-------    1   21.5M Mar 12 13:22 sortoeB1UP
-rw-------    1   21.3M Jul 26  2016 sortommPiB
-rw-------    1   20.7M Apr 16  2016 sortosijJf
-rw-------    1   21.6M Apr  3  2016 sortpRnAfm
-rw-------    1   20.6M Apr  3  2016 sortpwM7s1
-rw-------    1   21.4M Apr 16  2016 sortqvJRJ3
-rw-------    1   21.3M Jul 25  2016 sortr9sxIF
-rw-------    1   21.3M Apr  3  2016 sortrfIjFB
-rw-------    1   21.4M Mar 23  2016 sorts4kPo7
-rw-------    1   20.6M Apr 16  2016 sorts8ejFz
-rw-------    1   21.1M Apr 16  2016 sortsZQ7LQ
-rw-------    1   19.8M Apr 16  2016 sortsePCVP
-rw-------    1   21.7M Mar 12 15:23 sortslim2O
-rw-------    1   21.0M Jul 26  2016 sortsvseOz
-rw-------    1   21.1M Jul 25  2016 sorttRBa4X
-rw-------    1   21.2M Apr  3  2016 sorttuhGVm
-rw-------    1   20.0M Apr 16  2016 sortuPIpkN
-rw-------    1   22.0M Apr 16  2016 sortuX5SXR
-rw-------    1   21.2M Apr 16  2016 sortuqnqtn
-rw-------    1   21.7M Apr  3  2016 sortvBHQnx
-rw-------    1   21.7M Mar 24 12:55 sortvFuwDv
-rw-------    1   20.6M Jul 25  2016 sortvPE57K
-rw-------    1   21.1M Jul 26  2016 sortw9My1R
-rw-------    1   21.3M Dec  5 13:51 sortwFifLP
-rw-------    1   21.3M Apr 16  2016 sortwMMZ1T
-rw-------    1   20.5M Mar 23  2016 sortwO5xL3
-rw-------    1   21.5M Apr  9 13:11 sortwOyDHY
-rw-------    1   21.0M Apr  9 12:45 sortwkR1kt
-rw-------    1   19.8M Apr  3  2016 sortxImGUL
-rw-------    1   21.3M Apr  3  2016 sortxKZJMZ
-rw-------    1   21.0M Mar 24 12:54 sortxaWecf
-rw-------    1   21.6M Jul 25  2016 sortxvacZa
-rw-------    1   20.8M Jul 26  2016 sortxwTAPf
-rw-------    1   21.2M Mar 23  2016 sorty7Id6x
-rw-------    1   21.4M Mar 23  2016 sortyCy44q
-rw-------    1   21.4M Jul 26  2016 sortyI3Xg6
-rw-------    1   21.4M Apr  3  2016 sortzi75u1
# 

[mdgm-ntgr]

Well I guess you want to remove those sort files, but also figure out which service or add-on is writing those files and get the service or add-on to write to the data volume instead and clean up after itself.

Looks like it's temp files for some kind of sorting being done.

Do you recall which add-ons you have installed and which services you are running?

[jak0lantash]

You need to delete those files.

updatedb seems to be the culprit: https://community.netgear.com/t5/Using-your-ReadyNAS/sortXXXX-files-in-var-tmp/m-p/949775#M80531

[itsonlycomputer]

Ok that's done.  Proceeding to final steps

15. Move back to the root: cd /

16. Sync: sync

17. Unmount the OS volume: umount /sysroot

18. Stop the RAID arrays: mdadm -S /dev/md*

19. Reboot the machine: reboot -f

FINGERS CROSSED.

[itsonlycomputer]

Directory not found using:    Stop the RAID arrays: mdadm -S /dev/md*

[mdgm-ntgr]

Another option is

# mdadm --stop --scan

Then check to confirm that all the arrays have been stopped

# cat /proc/mdstat

[itsonlycomputer]

Ok,  I've rebooted the device and attempted to use raidar to find the device which it does successfully.   I changed the IP Address back to it's original value and I now have EVERYTHING back exactly the way it was.  FINGERS UNCROSSED.

THANK YOU THANK YOU THANK YOU!!!!!!!!!!

Questions:

1.   Is there anything I could have done beforehand to avoid this problem?   Why would this happen

2.   Should I remove or truncate any of the log files,  seems there were quite a few, to ensure they get recycled properly moving forward?

3.   and lastly,  Can this device be upgrade to the latest OS without losing data?   I can certainly start a new thread for this if you are unable to point me to step by step instructions.

Again,  I cannot thank you enough for your time and committment to my issue.

David

[jak0lantash]

I should have mentioned that mdadm -S /dev/md* is expected to show some errors. Checking that all the arrays are stopped with cat /proc/mdstat is a good step that could be added (though I was trying to use only steps that require little to no explanation).

1. Not too sure. Did you do anything via SSH prior to the issue?
2. The log files are OK. Those sort files aren't.
3. Within RAIDiator 4.2 generation, yes. To ReadyNAS OS6, no.

Glad your data is accessible again.

[StephenB]

---

@itsonlycomputer wrote:

1.   Is there anything I could have done beforehand to avoid this problem?   Why would this happen?

I'm not sure why it happened either.  Usually when the OS fills it is

• an add-on that writes to the OS partition
• A runaway log file
• the ReadyDLNA cache 

In your case it seems to be "none of the above".

While I have no advice on how you could have prevented it, I do have two suggestions:

1. I suggest you monitor the OS partition fullness for a while, since we really don't have a grip on the root cause.  You'd install EnableRootSSH from https://kb.netgear.com/24551/ReadyNAS-Add-ons-for-RAIDiator-4-2-x86?cid=wmt_netgear_organic.  Then you can access the NAS command line using putty.exe on a windows PC.  On a mac, you can simply use terminal.  If you see the tmp folder filling up again, post back hopefully we can find the root cause before it gets so full.

2.  We didn't talk about backup on this thread.  But if you aren't maintaining backups of the data on the NAS, I suggest you put a backup plan in place.  There was no data loss this time.  But lots ot things can go wrong, and RAID isn't enough to keep your data intact.