How to get volume back to read-write mode

---

tijgert wrote:

I'm not too proud to grasp at straws...

So, how do I do that?

---

Start by downloading the log zip file, and post partition.log and mdstat.log. That will provide the information needed to manually assemble the RAID array(s).

I'll give you other steps once I have that info.

Appreciate it. Here goes.

Mdstat.log

Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md127 : active raid5 sda3[0] sdf3[5] sde3[4] sdd3[3] sdc3[2] sdb3[1]
78105144000 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]

md1 : active raid10 sda2[0] sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb2[1]
1566720 blocks super 1.2 512K chunks 2 near-copies [6/6] [UUUUUU]

md0 : active raid1 sda1[1] sdb1[2] sdc1[3] sdd1[4] sde1[5] sdf1[6]
4190208 blocks super 1.2 [6/6] [UUUUUU]

unused devices: <none>
/dev/md/0:
Version : 1.2
Creation Time : Wed Feb 15 19:51:42 2023
Raid Level : raid1
Array Size : 4190208 (4.00 GiB 4.29 GB)
Used Dev Size : 4190208 (4.00 GiB 4.29 GB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent

Update Time : Fri Aug 2 15:06:20 2024
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Consistency Policy : unknown

Name : 7c6e3766:0 (local to host 7c6e3766)
UUID : bce2e184:a906144d:1961ca2d:1a43aebf
Events : 319

Number Major Minor RaidDevice State
1 8 1 0 active sync /dev/sda1
6 8 81 1 active sync /dev/sdf1
5 8 65 2 active sync /dev/sde1
4 8 49 3 active sync /dev/sdd1
3 8 33 4 active sync /dev/sdc1
2 8 17 5 active sync /dev/sdb1
/dev/md/1:
Version : 1.2
Creation Time : Wed Feb 15 19:57:40 2023
Raid Level : raid10
Array Size : 1566720 (1530.00 MiB 1604.32 MB)
Used Dev Size : 522240 (510.00 MiB 534.77 MB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent

Update Time : Fri Jul 26 00:31:30 2024
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Layout : near=2
Chunk Size : 512K

Consistency Policy : unknown

Name : 7c6e3766:1 (local to host 7c6e3766)
UUID : 8942c986:4e610533:bc3bf4e8:1c3a56ea
Events : 19

Number Major Minor RaidDevice State
0 8 2 0 active sync set-A /dev/sda2
1 8 18 1 active sync set-B /dev/sdb2
2 8 34 2 active sync set-A /dev/sdc2
3 8 50 3 active sync set-B /dev/sdd2
4 8 66 4 active sync set-A /dev/sde2
5 8 82 5 active sync set-B /dev/sdf2
/dev/md/NAS516-0:
Version : 1.2
Creation Time : Wed Feb 15 19:57:40 2023
Raid Level : raid5
Array Size : 78105144000 (74486.87 GiB 79979.67 GB)
Used Dev Size : 15621028800 (14897.37 GiB 15995.93 GB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent

Update Time : Fri Aug 2 10:22:09 2024
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

Consistency Policy : unknown

Name : 7c6e3766:NAS516-0 (local to host 7c6e3766)
UUID : b3917b1f:62e0a85a:64524e32:dddac605
Events : 737

Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
1 8 19 1 active sync /dev/sdb3
2 8 35 2 active sync /dev/sdc3
3 8 51 3 active sync /dev/sdd3
4 8 67 4 active sync /dev/sde3
5 8 83 5 active sync /dev/sdf3

Partitions.log

major minor #blocks name

8 0 15625879552 sda
8 1 4194304 sda1
8 2 524288 sda2
8 3 15621160904 sda3
8 16 15625879552 sdb
8 17 4194304 sdb1
8 18 524288 sdb2
8 19 15621160904 sdb3
8 32 15625879552 sdc
8 33 4194304 sdc1
8 34 524288 sdc2
8 35 15621160904 sdc3
8 48 15625879552 sdd
8 49 4194304 sdd1
8 50 524288 sdd2
8 51 15621160904 sdd3
8 64 15625879552 sde
8 65 4194304 sde1
8 66 524288 sde2
8 67 15621160904 sde3
8 80 15625879552 sdf
8 81 4194304 sdf1
8 82 524288 sdf2
8 83 15621160904 sdf3
9 0 4190208 md0
9 1 1566720 md1
9 127 78105144000 md127

tijgert wrote:

If it would just let me wipe those emergency saves that I rescued to a new drive then it would instantly have 10TB of space and everything will be solved.

 

---

I don't know if the NAS will never put itself back in read/write mode because BTRFS doesn't.  If nothing is damaged on your BTRFS volume (and I don't think you know that to be the case), you may be able to do it yourself via SSH or in support mode, or maybe a re-boot will do it once the full status is remedied.  When this happened to me, the volume was definitely damaged, so my experience of not successfully re-mounting as read/write may not apply.  But once the volume was full, other damage may have occurred, even though it going read-only is an attempt to prevent that.

Here is a reddit post by somebody who solved the problem, though not on a ReadyNAS:  https://www.reddit.com/r/btrfs/comments/ipagcl/full_disk_stuck_as_read_only/ .  As you can see, it can require a lot of work via the command prompt.

If you can successfully mount it as read/write in tech support mode, you need to wait some time after deleting some files before you re-boot.  That's because you likely ran out of space for metadata, not normal data, and it takes BTRFS some time to recover metadata from deleted files.  You could execute top and see when that recovery process is done.  You may also want to do a balance, which will reclaim additional metadata space..

---

Sandshark wrote:

 If nothing is damaged on your BTRFS volume (and I don't think you know that to be the case),

tijgert: That is my concern also.  I suspect something did get damaged when the volume became full.

FWIW, dmesg.log likely will give a useful hint when the volume was mounted.

*double*

Weird thing happening, I posted the logs twice and twice they got removed.

Is there a system preventing logs to be posted?

I agree that damage *might* have occurred, but I'm expecting that damage to be in the emergency saved files that are no longer needed. Any damage would be erased with erasing those files and then scrubbing/balancing I would think as all other files never moved or were altered.

Regardless, it's worth a try and then maybe a bit by bit check with the mirror to confirm, even just for knowing for future events or to help others in this same mess.

*double*

The logs have been posted a few posts back.

---

tijgert wrote:

Weird thing happening, I posted the logs twice and twice they got removed.

Is there a system preventing logs to be posted?

 

---

For some reason the automated spam filter often triggers on logs.  I released the quarantined posts for you.

Appreciate it.

As for the Reddit post; I believe that problem was fixed by adding capacity.

All my 6 bays are populated and I could only add drives with an EDA500 expansion bay, which I don't have and can't find.

---

tijgert wrote:

 

I agree that damage *might* have occurred, but I'm expecting that damage to be in the emergency saved files that are no longer needed.

---

Often it's metadata, which isn't so easily cleaned.

However, here are the instructions:

Start by booting the NAS into tech support mode using the boot menu.

Instructions are on pages 21-22 here:

• https://www.downloads.netgear.com/files/GDC/RNDU2000/ReadyNAS_Ultra_UltraPlus_NVX_Pro_HW_en_8July11.pdf

Next you will need to access the NAS using Telnet.  If you are using a windows PC, I suggest you use the Putty client for that.

• https://www.chiark.greenend.org.uk/~sgtatham/putty/

When you connect to the NAS, log in as root.  The password is infr8ntdebug.

Once logged in, enter these commands.  

rnutil chroot
mdadm --assemble /dev/md127 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3 /dev/sde3 /dev/sdf3 
btrfs device scan
btrfs fi show
mount /dev/md127 /data

Any errors on the last three commands might give you a clue on what is going on.

You can then delete the files/folders using the linux command line.  After space is reclaimed, I do recommend doing a balance.

Solid HowTo, thanks!

I may have run into a noob problem right off the bat though.

In normal mode I can Telnet into the NAS just fine using the IP address shown on the NAS with the standard port 22.

In Debug Mode this no longer works.

I enter the IP address as shown on the NAS but I get a fatal Putty error "Network error: Connection refused".

It refuses it, so the address is correct, the NAS just rejects the attempt.

The NAS also shows DebugMode 63334 in brackets on the screen, figuring that may be the port, but it doesn't work either.

I may need the slightest of nudges to get me in so I can br.. fix stuff.

Did you go to tech support mode using the reset button menu or by selecting "Secure Diagnostic Mode" from the GUI?  You need to do it from the reset menu.

I followed the instructions as per manual for my specific NAS.

Pressed the reset switch on the back with a pointy thing, I kept it pressed while powering on.

When the Boot menu shows on the front I release the reset switch, I choose the Debug mode, press OK and let it boot. It then shows "Debug Mode 63334" and the IP address.

I start Putty, enter the IP address, leave the port at 22 as standard and click 'open' and it refuses me access.

When I exit the Debug mode (only possible with 4 second press of the power button) and restart, I can login with Putty on that same IP address just fine.

---

tijgert wrote:

 

In normal mode I can Telnet into the NAS just fine using the IP address shown on the NAS with the standard port 22.

 

---

No you can't.  The NAS won't accept telnet connections in the normal mode.  You are selecting ssh in the putty dialog.

You need to select "other", and then choose "telnet" in the pulldown.

Currently backing up the NAS first, so it'll take a day or two before I can continue. I shall return!

So, the last 27 hours or so I’ve been copying files off of the NAS to create an extra backup, just in case the backup NAS I already have goes **bleep** up at the worst possible time and leaves me in even greater, potentially total loss, trouble. Only the most critical of files though as backing up 96TB takes over a week.

Before that, like I wrote here, I got into debug mode but couldn’t figure out how to Telnet into the system as I was unaware that SSH and Telnet were not the same and required a different setting in Putty. You see, a monkey can fly an airplane but that doesn’t mean he knows what he’s doing, sorry about that and I really appreciate your help and patience.

While it was copying files from the NAS I wanted to watch a show I had stored on there as well when I got an ominous ‘unable network playback’ error in Kodi. I couldn’t open any video file I tried, also not from the computer, though a zip file did work.

Copying any video file from the NAS and then opening it also gave an error and playback wasn’t possible. I didn’t understand what had happened but it seemed like all my video files were corrupted in some way. Maybe the lack of metadata storage space broke something? This was strange as the NAS was in ReadOnly mode and I could watch that show the day before just fine.

When the backup was finished I figured I could try one last reset to see if that would somehow make the video files openable again before I wipe the system and start to rebuild it, taking up to two weeks… ugh.

The reboot was uneventful, the video files were playable again and I noticed the free space indicator in Total Commander showing a little bit more free space than before. It seemed like only a couple dozen or maybe hundred Megabytes extra, but I noticed it because the free space amount was a fixed value for days. I figured maybe the Debug Mode might have allowed the system to do some stuff with writes and now it’s stuck again at this value?

Regardless, before a full wipe I wanted to check one last time if maybe this indicated that something got jarred and something changed. And it did.

Before, when I deleted files while it was in ReadOnly mode, it would delete the files like it always did, removing them visually from the folder in Total Commander while not adding to the free space, but then going out and back into the folder they would reappear.

This time, it ‘deleted’ the files visually (expecting them to reappear again), but the free space changed, it grew. I went out and back into the folder and the files were still gone. I back and forthed a few times, they stayed gone and the free space was greater than before and stayed there.

I erased some more files and they too stayed gone and added to the free space. I did this several times until I had about a TB of free space. I proceeded to download the latest logs to see what changed.

(I didn’t download the logs when I couldn’t open video files, maybe that would’ve showed something).

The logs show nothing that I can decipher as meaningful in ascertaining what changed. I'll post it here just in case a Guru can see what happened.

Regardless: The NAS works again as before, though I'll do a thorough check to find if there's any corruption.

I would love to hear what you guys think happened.

---

tijgert wrote:

 

I would love to hear what you guys think happened.

---

At this point it would be hard to sort out.  dmesg has useful information for the most recent boot, but not what happened earlier. The boot into tech support mode wouldn't have affected anything, since you never started RAID or attempted to mount the file system.

I suggest freeing up at least 15% free space.  Then run a BTRFS balance, followed by a scrub.  You start these from the settings wheel on the volume page.  They run in the background (you'll see an entry on the log page when they finish). If you've never run these, they likely will take quite a while (days).

I emptied 10TB and am running scans now.

I just can't wrap my head around what could've happened to suddenly have files eraseable again like nothing happened.

I feel this might be of use to many BTRFS users who find themselves in this kind of a situation.

Thank you for releasing the quarantined posts. Please let me know if there are any additional steps needed to ensure the volume is back to read-write mode.

Thank you for update!