RN212 all files gone except iCloud

jimk1963 · ‎2020-09-24

Hi again, concerning issue with RN212. Had about 4TB stored, all are seemingly gone except a lone iCloud folder under Pictures share. History:

1) 3 NAS boxes have been idling for over a month without access

2) Last week usin gReadyCloud iOS app, attempted to pull a file - couldn't login

3) Selected "reset password", email sent, link followed, password updated, able to access RN212 again

4) But instead of all the files I expected, only a lonely iCloud Photos folder (with data)

5) Default Shares (Documents, Music, Pictures, Videos) are all visible using Windows Explorer and using Chrome Browser (admin page). All are empty now

6) Confirmed firmware is up to date, as is anti-virus scanner

7) Also had a scare on other two NAS's but after rebooting them (in addition to RN212) and rebooting my PC, was able to see all files on RN314 and RN528X. However, RN212 remains blank

😎 Downloaded log files, no idea what to look for

Please help!

StephenB · ‎2020-09-25

On the problem itself - though it's too early to say what the chance of getting the data back is, it doesn't sound good. Anything you write to the NAS will lower the odds. So I would either shut it down, or reboot it in read-only mode.

It sounds you are saying that the NAS suddenly has a healthy but empty volume (apart from that one picture folder) I am wondering

If you had other shares (not the default ones) before that have now disappeared.
if the failure might have given you a completely empty volume, with the iClouds photos resynced by ReadyCloud from your phone after the failure occured.

On the logs: One option is to contact the mods ( @JohnCM_S and @Marc_V ) via PM (private message), and ask them to analyze the logs. Send them a link to this thread and a download link (google drive, icloud, dropbox, etc) link to the log zip file. Don't post the log zip here.

When I'm looking at them, I always look in disk_info.log at the disk health, and mdstat.log to see what disks are currently in the volume RAID array (md12x RAID groups are the ones in the data volume). I generally also check the bottom of volume.log, as that will tell me the OS partition fullness, and the maintenance bit at the bottom can also be useful.

Readynasd.log and status.log will give you a high-level view of what the ReadyNAS application was doing, and that might give you some clues on what else went on around the time you had to reset your ReadyCloud password.

You can delve deeper by looking in system.log, kernel.log and perhaps dmesg.log though there is a lot in there.

Sandshark · ‎2020-09-25

Are you using the admin credentials for any/all of these? If not, do, in case it's just a permissions issue. I had some folders get the permissions changed without me knowing why, and a few others have reported the same. Resetting them fixed it.

Are you comfortable with SSH? That would be the place to absolutely see whether the files are really still there, but have been rendered invisible or inaccessible to you via the other protocols.

Could this be from some sync process going horribly wrong? Files were removed from the other location, or the other location became inaccessible, and the sync removed them from the NAS?

jimk1963 · ‎2020-09-25

Thanks @Sandshark and @StephenB .

Haven't ran SSH in a few months, forgot most of it but will try to bring it up again.

Re: admin, I am completely confused on this. Two of the NAS's login using my email address, and one uses "admin".

The RN212 (with the problem): when I log in, under "Accounts" it just shows "Admin" and then to the right it shows my email under "Secondary Members".

Is there a simple way to "reset all permissions" as you indicated?

Sorry for naive questions, I'm really poor at understanding permissions, admin vs. users, etc., especially on this NAS platform.

jimk1963 · ‎2020-09-25

Thankfully I created an SSH "how-to" document. Logged into the RN212 using admin, interestingly the password was my old password and not the new password I recently updated to. That made me hopeful...

Executed

cd ..

ls

cd Pictures

ls

And the only folders it lists are the iCloud Photos folder and one more folder I added as a test (unfortunately I did that prior to @StephenB 's advice not to do that). Also checked Documents folder, same story. Nothing.

Does that conclusively prove the entire machine has been wiped out? If so, I'd sure as heck like to understand how/why... You guys may recall I upgraded this box with 2 Seagate Ironwolf ST12000VNZ008 12TB replacement drives. I did it by swapping in one at a time, so the rebuild process took days but it all worked fine and continued to be fine for at least a month.

I'm concerned someone hacked into my home account, or more likely, there's just something wrong with the NAS. But it's weird that the NAS is working fine, and yet all the files seem to be gone. Ugh.

jimk1963 · ‎2020-09-25

Thanks @StephenB for the info below. To answer your first question, there are no new shares, just the defaults.

Re: logs, checked the following:

1) disk_info: Health Data on both drives is similar, and looks benign:

Health data
ATA Error Count: 0
Reallocated Sectors: 0
Reallocation Events: 0
Spin Retry Count: 0
Command Timeouts: 0
Current Pending Sector Count: 0
Uncorrectable Sector Count: 0
Temperature: 37
Start/Stop Count: 2
Power-On Hours: 2085
Power Cycle Count: 2
Load Cycle Count: 35104

2) mdstat: Looks normal from what I can see, RAID1 with 2 disks, State="clean", Failed Devices=0, Active Devices=2

md127 : active raid1 sda3[0] sdb3[1]
11714035648 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda2[0] sdb2[1]
523712 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sda1[0] sdb1[1]
4190208 blocks super 1.2 [2/2] [UU]

3) volume: shows very little data used, ugh

=== df -h ===
Filesystem Size Used Avail Use% Mounted on
udev 10M 4.0K 10M 1% /dev
/dev/md0 3.7G 1.1G 2.5G 30% /
tmpfs 1009M 4.0K 1009M 1% /dev/shm
tmpfs 1009M 560K 1009M 1% /run
tmpfs 505M 1.1M 504M 1% /run/lock
tmpfs 1009M 0 1009M 0% /sys/fs/cgroup
/dev/md127 11T 20G 11T 1% /data
/dev/md127 11T 20G 11T 1% /apps
/dev/md127 11T 20G 11T 1% /home
=== df -i ===
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 187927 228 187699 1% /dev
/dev/md0 1048576 12024 1036552 2% /
tmpfs 188818 2 188816 1% /dev/shm
tmpfs 188818 403 188415 1% /run
tmpfs 188818 25 188793 1% /run/lock
tmpfs 188818 9 188809 1% /sys/fs/cgroup
/dev/md127 0 0 0 - /data
/dev/md127 0 0 0 - /apps
/dev/md127 0 0 0 - /home

=== maintenance history ===
device operation start_time end_time result details
---------- --------- ------------------- ------------------- --------- ----------------------------------------------------------------
data resilver 2020-06-29 20:38:58 2020-07-01 00:09:36 completed

Sandshark · ‎2020-09-25

Unfortunately, that does appear that your diagnosis is correct -- your files have vanished. You can try to see if ReclaiME can recover any files that you don't have backed up. You can download a free copy and see if it will do the job before you pay for the actual recovery: https://www.reclaime.com/library/nas-recovery.aspx .

jimk1963 · ‎2020-09-25

Thanks @Sandshark , I have the files saved on the other two NAS's, believe everything is there. Before I try to run a recovery program though, or just copy all the files back onto the NAS, I really want to figure out the HOW... how did these files vanish? Because without root cause, I fear it's going to happen again, either on this one or on the other two, or all three. I've had the RN314 and RN212 for 7 and 5 years, respectively, without ever losing a byte. This year I upgraded both with new drives (you both were very helpful with the data migration). And within a few months I already have one bad NAS. And by the way, it's the NAS with the supposedly bulletproof Ironwolf drives. The other NAS with the more questionable WD Red 4TB's is just fine. Is there any way to look through the log files to determine if there was a purposeful erase?

StephenB · ‎2020-09-25

@jimk1963 wrote:

Is there any way to look through the log files to determine if there was a purposeful erase?

There should be some evidence if the RAID array being recreated.

FWIW, you have very unusual symptoms. I've seen many cases where the volume goes inactive - but I've not seen a case of a spontaneous factory reset (or a volume being spontaneously destroyed and recreated) on an OS-6 ReadyNAS.

I am wondering if something else might have gone wrong - a run-away sync program that for some reason decided to delete the files on your NAS??? Does anyone else have access to the NAS?

jimk1963 · ‎2020-09-26

Thanks @StephenB :

1) I'm the only one who accesses these NAS boxes

2) PC antivirus just expired in Aug, didn't renew (McAfee) - Windows Security and Malware Bytes are active

3) Don't have any backups scheduled, which is where I would normally sync stuff

Just found two "issues" that may provide a clue:

1) Under "Performance" (see below) the usual Volume/Network/Utilization/Temperature graphs are completely gone. They show fine on the other two NAS boxes

2) The antivirus scanner update began logging an error in July and continued every day up to now, and still continues. Firmware is firmly up to date, and once I manually click on the antivirus scanner status, it then updates it a few minutes later. But then the next day, same issue. Read the 100+post thread on this issue, didn't find any magic answers other than rebooting the machine and keeping the OS up to date. OS has been up to date all along and I rebooted the machine yesterday; this morning, same issue. See below for the log list illustrating this.

jimk1963 · ‎2020-09-26

jimk1963 · ‎2020-09-26

Sorry not sure what's going on with the site... usually I can insert JPEG's into the message, this time it's not working so am attaching the two pics as files in two seperate posts.

Sandshark · ‎2020-09-27

The only ARM NAS I have is a 102, and I know it doesn't have the graphs because of insufficient RAM and/or CPU power, and I believe they are absent from all ARM units. The virus protection update issue does affect a lot of other users, and none have reported anything similar to your issue, so I don't think they are related.

Do the log entries you posted go back to a date you know the files were intact?

jimk1963 · ‎2020-09-27

Good question, I am not sure. I went through many of the log files but didn't find any clear information on files deleted/moved/etc. nor any "snapshot" type information to indicate a change in volume storage. I'm sure the info is there, I'm likely just missing it.

RN212 all files gone except iCloud

RN212 all files gone except iCloud

Re: RN212 all files gone except iCloud

Re: RN212 all files gone except iCloud

Re: RN212 all files gone except iCloud

Re: RN212 all files gone except iCloud

Re: RN212 all files gone except iCloud

Re: RN212 all files gone except iCloud

Re: RN212 all files gone except iCloud

Re: RN212 all files gone except iCloud

Re: RN212 all files gone except iCloud

Re: RN212 all files gone except iCloud

Re: RN212 all files gone except iCloud

Re: RN212 all files gone except iCloud

Re: RN212 all files gone except iCloud