Permission Issues: ReadyNAS RR4312X goes total read only mode, inoperable

[ddx]

We have two of these RR4312X (ReadyNAS 4312 with 10GBASE/T not showing up in the model list above).

One is working fine still, the other one went totally read only all of a sudden.  Please help!

1. Firmware was updated to 6.7.5
2. SMB Shares set up, was working fine for almost a year, suddenly this issue.
3. Was on Active Directory, I took it off AD for troubleshooting.
4. When it was on Active Directory, it was suddenly read only access regardless of actual permissions setup.
5. Permission changing was not possible at all (Error Code 1002030001 Commit Failed)
6. Creating new share is not possible (Always with error 1013050001 "Please verify that the share name is valid and unique")
7. Reset permissions function is not working ("Failed to change permission and ownership of the share -sharename-")
8. Rebooting has no effect.

Please advise, this thing holds our backups and we can't continue to run our backups.

[anna_arun]

How much space is left in your volume ?

[ddx]

Thanks for reply...   The NAS has 8TB left of 36TB

[evan2]

@ddx

Cloud do you please send to logs? 

How do I send all logs to ReadyNAS Community moderators?
https://kb.netgear.com/21543/How-do-I-send-all-logs-to-ReadyNAS-Community-moderators

[ddx]

Hi evan2, thanks for your reply.

I have sent the log file with your handle in the subject.

Thanks

[evan2]

@ddx

Check system logs, filesystem bterfs has errors, then forced readonly,

Jul 03 12:16:11 NSCOL kernel: BTRFS error (device md127): parent transid verify failed on 17416348631040 wanted 95453 found 294341
Jul 03 12:16:11 NSCOL kernel: BTRFS error (device md127): parent transid verify failed on 17416348631040 wanted 95453 found 294341
Jul 03 12:16:11 NSCOL kernel: BTRFS warning (device md127): Skipping commit of aborted transaction.
Jul 03 12:16:11 NSCOL kernel: BTRFS: error (device md127) in cleanup_transaction:1855: errno=-5 IO failure

See mount info, it is ro,

/dev/md127 on /run/nfs4/home type btrfs (ro,noatime,nodiratime,nospace_cache,subvolid=257,subvol=/home)

Could do you please create a support case through MyNETGEAR?

https://kb.netgear.com/26905/Creating-a-Support-Case-through-MyNETGEAR

[ddx]

Is my data safe? Can I do like check disk on btrfs?

Also do I have to pay $500 for support from netgear in this case? (My 90 day expired)

Thanks

[StephenB]

---

@ddx wrote:
Is my data safe? 

---

The file system is damaged, so It's at risk. You should back it up.

[ddx]

Does this happen often?  I feel its quite unreliable, especially when it happens its a total lockdown even I can't enable SSH.

[StephenB]

Well, since I don't work for Netgear I don't have any way to gather stats.

But BTRFS corruption doesn't show up very often here.

[ddx]

So I have been having trouble communicating with L3 Support staff, so I might just ask my question here.

So we paid for the premium support with the hope of resolving this issue, and it came back basically like this:

- File System is in a very bad shape: 
Data, single: total=30.85TiB, used=28.25TiB 
System, DUP: total=8.00MiB, used=3.34MiB 
Metadata, DUP: total=27.00GiB, used=24.21GiB 
GlobalReserve, single: total=512.00MiB, used=201.25MiB

After a little back and forth, the L3 staff followed up with:

- Poor setup: the customer ended up with almost 30 GB of metadata that are quite enough to cause troubles.

I have not had any luck getting the L3 staff to explain further what all these means.  All I have access to (as the customer) is the GUI, and in the GUI there is no information about any of those things.  How am I supposed to monitor or do anything about those???

I guess I could probably enable SSH service and check there.  However when I try to enable SSH it says "For security reasons, we recommend that you do not enable SSH. If you enable SSH root access, NETGEAR reserves the right to deny you technical support."  So it appears we're not supposed to do that.

I'm very concerned because we have two of these NAS, and I'm afraid the other one will be down soon as well.  Could you help me understand this issue?

Another thing is the L3 Staff mentioned that the hard drives used in the NAS is not in the HCL.  I swear the hard drives were in the list last year when we bought it, otherwise we wouldn't spend the $$$$$ to buy two of these.  Do hard drives gets pulled out of HCL ever?

Thanks for your help.

[StephenB]

I can't speak for L3, but I will see if I can find someone who can.  I do see your volume is quite full (91%), and that could be part of it.

BTW what hard drives did you purchase?

[ddx]

Thanks for your reply,

How do you see that it was 91%?  Is it from this line:  Data, single: total=30.85TiB, used=28.25TiB 

I'm not sure why it is shown total 30TiB, because the total (according to GUI) is 36TB.

As far as I know according to the GUI, it never went any lower than 6TB free (of 36TB), so never any more than 83% full.  At that point, we had a 4TB iSCSI LUN that was suddenly crashing our ESXi server, so we removed the 4TB LUN (The LUN was almost full at 3.5TB actual data).  When we did that, the free space only went down to 8TB, which where it is currently at (77% full).  And then all these mess started happening.  So that about 2TB space is unaccounted for, which may be the symptom of some bad stuff happening.

Our other same model NAS also had the same issue, suddenly it crashed the ESXi (a different ESXi host).  So on that one we took the LUN down, but it is still working fine (although we are really worried now!!).  That particular one does have more space left at any time (currently 61% full).  The still working NAS also has lower firmware version 6.7.4., I don't know if the newer firmware are more prone to this issue (the broken one on latest 6.7.5).

Other than the LUN (at the time) it is just normal SMB shares.

The hard drives are Hitachi Deskstar NAS HDN724040ALE640, 4TB x12 (using Netgear's X-RAID single 36TB volume).  The L3 staff keep saying it is a regular desktop drive, do not use regular desktop drive, etc.  However it is not a regular desktop drive, its a NAS drive, and I'm sure last year I checked the HCL and it was there at the time.  It does not show up in the HCL currently though, so the L3 staff basically said (paraphrasing), "well you screw up, sorry."  However it is a bit funny that the HCL does have one approved regular Deskstar HDS721010CLA332 (not NAS) hard drive in there, with the note "This disk should be compatible based on the test result of another disk in the same series from the disk manufacturer," yet there are no other Deskstar drives in the list!

A bit of suggestion to the forum viewers, save/make a screenshot of the HCL at the time of purchase! 

Thanks

[StephenB]

---

@ddx wrote:

The hard drives are Hitachi Deskstar NAS HDN724040ALE640, 4TB x12 (using Netgear's X-RAID single 36TB volume).  The L3 staff keep saying it is a regular desktop drive, do not use regular desktop drive, etc.  However it is not a regular desktop drive, its a NAS drive, 

---

Certainly it is a NAS drive, though it's spec sheet specifies Applications/Environments as desktop NAS - https://www.hgst.com/sites/default/files/resources/DS_NAS_spec.pdf.  Not sure it's intended for a 12 bay rackmount.  On the other hand, I don't know that it won't function in a rack mount either.  How do the SMART stats look?

---

@ddx wrote:

How do you see that it was 91%?  Is it from this line:  Data, single: total=30.85TiB, used=28.25TiB 

I'm not sure why it is shown total 30TiB, because the total (according to GUI) is 36TB.

That was it.  I don't understand the discrepancy either.

[jak0lantash]

It's because this extract is not enough. It only shows the allocated data blocks and the data inside it.
In btrfs.log from the log zip, also paste the section above this extract please.

[jak0lantash]

Do you have snapshots enabled on the LUN? Bit rot protection? Compression? Metadata usage is indeed quite high.

Do you run balances? I was going to ask if you run any defrag, but if you use CoW LUNs, I'm not sure if defrag can help that much.

Is RAID10 an option?

@ddx wrote:

The hard drives are Hitachi Deskstar NAS HDN724040ALE640, 4TB x12 (using Netgear's X-RAID single 36TB volume).  The L3 staff keep saying it is a regular desktop drive, do not use regular desktop drive, etc.

Maybe he confuses Deskstar with Desktop... Because they're clearly NOT desktop drives, it takes about 6.5 seconds on Google to confirm that. First result = "Deskstar NAS Internal Drive Kit Data Sheet - HGST"
Though I give the thumbs up to HGST for finding possibly the most stupid name for a non-desktop drive.

[jak0lantash]

---

@ddx wrote:

I guess I could probably enable SSH service and check there.  However when I try to enable SSH it says "For security reasons, we recommend that you do not enable SSH. If you enable SSH root access, NETGEAR reserves the right to deny you technical support."  So it appears we're not supposed to do that.

---

SSH is a feature. You CAN use it.

Sometimes, you hear things like "it's forbidden" or "not allowed", it's not. It's a feature available for everyone, you can even enable it from the GUI.

What this warning means is that if you perform intrusive actions via SSH and break your system, it's not NETGEAR Support's job to figure it out for you. You broke it, you fix it.

You can use SSH freely as long as it's nonintrusive actions. The rest is your own responsibility.

An example of a nonintrusive command you can use:

btrfs filesystem usage /data

[ddx]

@StephenB

How do I check the SMART stat on the GUI?  Otherwise is says everything is normal.

Is there a big difference between NAS drives for the form factor of "Desktop" or for "Rackmount"?   Not Netgear product, but Synology has a 12-bay "Desktop" NAS called "Synology DiskStation DS2415+" that has my exact hard drive on their compatibility list (and many other Deskstar NAS series hard drives).

@jak0lantash

Here's the above part in the btrfs.log:

Label: '2fe77d04:root' uuid: f64a0e0e-8d77-4f2a-8ba4-c61db0d62943
Total devices 1 FS bytes used 511.58MiB
devid 1 size 4.00GiB used 1.63GiB path /dev/md0

Label: '2fe77d04:data' uuid: 2f0b2f9c-cbfd-43fe-85af-39ebb45250d7
Total devices 1 FS bytes used 28.27TiB
devid 1 size 36.34TiB used 30.90TiB path /dev/md127

We do not use snapshots on anything.

Bit rot protection is enabled on all the shares.

Compression is not enabled.

I don't believe I had CoW on the LUN.

About SSH, does the NAS keep logs of what commands we run?  Otherwise, how would one tell?

I do want to check our still running NAS if their data usage is in check, but want to make sure before we will be denied support.

Thanks for your responses,

[StephenB]

---

@ddx wrote:

How do I check the SMART stat on the GUI?  Otherwise is says everything is normal.

Unfortunately there's no good way - an odd omission.  It is in the log zip though - disk_info.log and smart_history.log

---

@ddx wrote:

@StephenB

How do I check the SMART stat on the GUI?  Otherwise is says everything is normal.

Is there a big difference between NAS drives for the form factor of "Desktop" or for "Rackmount"?   Not Netgear product, but Synology has a 12-bay "Desktop" NAS called "Synology DiskStation DS2415+" that has my exact hard drive on their compatibility list (and many other Deskstar NAS series hard drives).

---

I don't know about HGST specifically, but generally "desktop NAS" drives are intended for smaller NAS.  WDC Reds for instance say "built for single-bay to 8-bay". Vibration protection might be weaker than you'd see in an enterprise class drive.

How much of this is sales hype and how much is real depends on who you ask.  BackBlaze for instance doesn't buy it - they use desktop-class drives in their 60-bay storage pods. 

---

@ddx wrote:

About SSH, does the NAS keep logs of what commands we run?  Otherwise, how would one tell?

IMO The warning is a bit too strong. There's a better formulation here: https://kb.netgear.com/30068/ReadyNAS-OS-6-SSH-access-support-and-configuration-guides

• Access by SSH is not discouraged, but is recommended for advanced users only.
  • As such, using SSH is at the user's own risk.
• Support is offered for configuring SSH using available options provided in the Admin Page.
• Support will not be offered for using SSH.
• Support may be denied if it is determined that actions taken through SSH have contributed to problems encountered on the ReadyNAS. However, you may factory reset the ReadyNAS and restore data from backup at which point normal support may resume.
• SSH access does not affect the hardware warranty.

Of course there is information in the downloaded log zip file (which is rotated).  That includes apt-history.log.  I don't think there's anything hidden.

[ddx]

I checked disk_info.log

All drives seem to check out fine, all 0 on these:

ATA Error Count, Reallocated Sectors, Reallocation Events, Spin Retry Count, Current Pending Sector Count, and Uncorrected Sector Count.

All on the same firmware.

About hard drive choice, I guess now it depends on how NAS manufacturers see it in this kind of situation.  I'm not sure if necessarily Netgear is denying support because the hard drive is not (longer) in their HCL list.  That is certainly my impression for now.

I'll try the SSH then.

[jak0lantash]

Label: '2fe77d04:data' uuid: 2f0b2f9c-cbfd-43fe-85af-39ebb45250d7
Total devices 1 FS bytes used 28.27TiB
devid 1 size 36.34TiB used 30.90TiB path /dev/md127

Data, single: total=30.85TiB, used=28.25TiB 
System, DUP: total=8.00MiB, used=3.34MiB 
Metadata, DUP: total=27.00GiB, used=24.21GiB 
GlobalReserve, single: total=512.00MiB, used=201.25MiB

36.34TiB is the capacity of the volume.
28.25TiB is the amount of data (of data type).
30.85TiB is the sum of allocated chunks of data type (the footprint, the "data allocation").
24.21GiB is the amount of metadata.
27.00GiB is the sum of allocated chunks of metadata type (the footprint, the "metadata allocation"), here in DUP mode, so stored twice on the volume (standard on ReadyNAS).
30.90TiB is the footprint of allocated chunks of all types (data, metadata and system) = 30.85TiB + 8.00MiB x 2 + 27.00GiB x 2.
28.27TiB is the sum of used capacity inside all allocated chunks = 28.25TiB + 3.34MiB x 2 + 24.21GiB x 2.
Global reserve is accounted for within metadata.
Afaik, system chunks are accounted for in the volume, not outside, so I included it in the calculations above, though it wouldn't make much difference otherwise.

[ddx]

Thanks for your responses,

So this is regarding my other (apparently) still working RR4312X NAS.

- I enabled SSH and tried to run the SSH command, but "btrfs" is not recognized as a proper command.

- I also downloaded the log, in btrfs.log here's what I see.

Label: '2fe72ca4:root' uuid: 56a086c1-9729-4317-8fbc-ea04d86eded1
Total devices 1 FS bytes used 527.59MiB
devid 1 size 4.00GiB used 2.45GiB path /dev/md0

Label: '2fe72ca4:data' uuid: 5668e473-9290-4b4c-85b8-e4078ac6bb88
Total devices 1 FS bytes used 26.22TiB
devid 1 size 36.34TiB used 27.39TiB path /dev/md127

=== filesystem /data ===
Data, single: total=27.33TiB, used=26.20TiB
System, DUP: total=40.00MiB, used=3.09MiB
Metadata, DUP: total=29.50GiB, used=27.35GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

Looks like this one has even larger Metadata!

Any suggestion on how to resolve this issue (if this is an issue)?

Thanks

[StephenB]

---

@ddx wrote:

- I enabled SSH and tried to run the SSH command, but "btrfs" is not recognized as a proper command.

Did you log in as "root"? Or "admin'.

[ddx]

Hi @StephenB

Oops, my bad.  I relogin with root and it works, thanks.

This is what I got from my damaged NAS:

Overall:
Device size: 36.34TiB
Device allocated: 28.52TiB
Device unallocated: 7.82TiB
Device missing: 0.00B
Used: 25.93TiB
Free (estimated): 10.40TiB (min: 6.49TiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)

Data,single: Size:28.46TiB, Used:25.89TiB
/dev/md127 28.46TiB

Metadata,DUP: Size:27.00GiB, Used:21.34GiB
/dev/md127 54.00GiB

System,DUP: Size:8.00MiB, Used:3.19MiB
/dev/md127 16.00MiB

Unallocated:
/dev/md127 7.82TiB

When browsing the command btrfs I see that there is a "restore" function ("Try to restore files from a damaged filesystem (unmounted)"), is that something I can try?  (without breaking support)

The support team have access to the NAS and on occassion they are trying something by unmounting the volume.  I haven't got any response yet on what they are actually doing, perhaps running this command.

So far they are not very helpful in explaining whats going on.

Thanks,

[jak0lantash]

btrfs restore is to dump data from the (unmonted) volume to another storage.

Without having a look at your volume directly, I cannot recommend one approach or another, but usually the first step is to try mount the volume in read-only mode, if doesn't work then in recovery mode, if doesn't work then btrfs restore.

I don't think it's wise to interfere with their work though.

Can you post here your case number please?