× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Readynas RN204 - Stuck at Booting... 42%

davor242
Aspirant

Readynas RN204 - Stuck at Booting... 42%

RN204, OS 6.9.0

Noticed iSCSI misbehaving (no connection, normal shares operational), logged on web GUI, initiated reboot.

 

Reboot stuck on "see you soon". Power-button no reaction. Eventually power-cyled.

 

Now stuck on boot: "Booting... 42%". Blinking blue light, no disc activity. Pressing Power-button toggles led text to "systemd-udev-set".

Tried rebooting a few times, same problem.

 

HELP!

 

Model: RN204|ReadyNAS204
Message 1 of 19
davor242
Aspirant

Re: Readynas RN204 - Stuck at Booting... 42%

It finally booted but now "No volumes exists".

 

*panic*

Message 2 of 19
StephenB
Guru

Re: Readynas RN204 - Stuck at Booting... 42%

I suggest paid support (my.netgear.com)

Message 3 of 19
davor242
Aspirant

Re: Readynas RN204 - Stuck at Booting... 42%


@StephenB wrote:

I suggest paid support (my.netgear.com)


That sounds serious. 😮 

 

Is it "oh yeah, known problem, don't use iSCSI, you are f*cked but paid support can help you recover the data" or "Have not seen this particular sequence earlier, maybe support'll know"........?

Message 4 of 19
davor242
Aspirant

Re: Readynas RN204 - Stuck at Booting... 42%

Good news! 

I switched off NAS. Took out the drives and went to bed. And the good news part:

 

This morning I booted the NAS without drives.

Then I switched off the NAS.

Put in the (3) drives in again (in random order).

Switched on the NAS and it magically works again! 

Log: Disk: Disk in channel 1 (Internal) changed state from ONLINE to RESYNC.

 

 

Message 5 of 19
davor242
Aspirant

Re: Readynas RN204 - Stuck at Booting... 42%

The iSCSI just dropped out again. 😮

Message 6 of 19
davor242
Aspirant

Re: Readynas RN204 - Stuck at Booting... 42%

Restaring NAS after iSCSI problems -> stuck on "Restarting Now See You Soon".

Note: the NAS thinks there is an iSCSI connection from the PC, the PC thinks otherwise and is not able to reconnect. 

 

Noticed that the btrfs-transacti command was using 100% CPU before I was kicked out of SSH.

 

Booting. Stuck at 42% for long time again. So same symptoms as before. *waiting*. NAS is not visible to RAIDar at this point. udev-settle.service issues? 25+ minutes now. 

 

 

 

 

Message 7 of 19
davor242
Aspirant

Re: Readynas RN204 - Stuck at Booting... 42%

Another stop at 92% (mysql.service).

 

At this point SSH is availale.

mount  is using 95% (total time used 33:48), pilgrim is around 20% with occasional jumps to 35% (total time used 7:41). boot.png

 

 

 

 

 

 

Message 8 of 19
davor242
Aspirant

Re: Readynas RN204 - Stuck at Booting... 42%

Next stop is 98% (upnphttpd.servic). 55 minutes since power on. top paints the same picture as before.

 

RADar can access the NAS.

 

raidar_98boot.png

Successfully completed diagnostics System

  • Volume data is inactive
  • Volume data is inactive

Logs

  • No errors found

 

 

 

Message 9 of 19
StephenB
Guru

Re: Readynas RN204 - Stuck at Booting... 42%


@davor242 wrote:

@StephenB wrote:

I suggest paid support (my.netgear.com)


That sounds serious. 😮 

 

Is it "oh yeah, known problem, don't use iSCSI, you are f*cked but paid support can help you recover the data" or "Have not seen this particular sequence earlier, maybe support'll know"........?


Closer to the latter.  Failure to boot with possibility of data loss, complicated by the fact that iSCSI data is opaque to the NAS.  I was assuming that you had no backup (though that was a guess).

 

Though it looks like you've also installed stuff with ssh  (pilgrim anyway), which could complicate things further.

 

Any idea what mount is mounting?

Message 10 of 19
davor242
Aspirant

Re: Readynas RN204 - Stuck at Booting... 42%



Closer to the latter.  Failure to boot with possibility of data loss, complicated by the fact that iSCSI data is opaque to the NAS.  I was assuming that you had no backup (though that was a guess).

 

Though it looks like you've also installed stuff with ssh  (pilgrim anyway), which could complicate things further.

 

Any idea what mount is mounting?


I only installed mc (midnight commander). 

pilgrim appears to be what controls the front panel LCD of the NAS.

 

Usage: pilgrim [-c] [-l path] [ -p path] [-s hex] [-t "term cmd"] [-u file]
    -c Clear LCD on termination
    -l Message list file path(/var/readynasd/pilgrim.list)
    -p Socket path (\0/org/freedesktop/plymouthd)
    -s Solid box character code in hex (ff)
    -t Termnation cmd (none)
    -u Termnation unit ("readynasd.service")

 

Still at 98%. 

 

Message 11 of 19
davor242
Aspirant

Re: Readynas RN204 - Stuck at Booting... 42%

 /bin/mount /dev/disk/by-label/1177b17c:data /data -t btrfs

Message 12 of 19
davor242
Aspirant

Re: Readynas RN204 - Stuck at Booting... 42%

After 80 minutes, boot finishes.

This time volume is there, but shares are not. Interestingly, shares can be seen from SSH. 

 

raidar_missing_shares.png

Message 13 of 19
davor242
Aspirant

Re: Readynas RN204 - Stuck at Booting... 42%

Reboot without disks.

Let it run dry.

Shutdown. Insert disk, switching 2 disk positions.

Boot and voila it works again. 

Message 14 of 19
davor242
Aspirant

Re: Readynas RN204 - Stuck at Booting... 42%

In essence, I believe there are issues with iSCSI implementation. 😕

Message 15 of 19
StephenB
Guru

Re: Readynas RN204 - Stuck at Booting... 42%


davor242 wrote: 

I only installed mc (midnight commander). 

pilgrim appears to be what controls the front panel LCD of the NAS.

 


So it is (it's not something I'd noticed before).

 

Is top telling you what the mount command is doing?  The "c" toggle should give you it's command line.

 


@davor242 wrote:


 

In essence, I believe there are issues with iSCSI implementation. 😕

 


Clearly something is wrong, but I don't think you've isolated it to iSCSI just yet.  Lots of things will misbehave when the system is running at 98% CPU.

Message 16 of 19
davor242
Aspirant

Re: Readynas RN204 - Stuck at Booting... 42%

 

I have created a new LUN with thick privisioning and moved data there. This seems more stable, however iSCSI dropped out again (normal shares work fine).

 

What I have noticed is that every time the iSCSI drops the btrfs-transacti is running at 100% (i.e one core full load).

The btrfs-transacti will eventually calm down and iSCSI pops up again without NAS restart (although Windows may or may not be confused at that point and PC restart may be necessary, have not kept track of this fully).

 

So my current guess is:

 

* Something triggers this btrfs storm

* This affects iSCSI which drops

* Restarting the NAS at this point should be avoided

 

I am guessing that this btrfs-transacti is doing something useful. But I would consider dropping the iSCSI as a product bug. 

 

btrfs.png

Message 17 of 19
kohdee
NETGEAR Expert

Re: Readynas RN204 - Stuck at Booting... 42%

Did you create a Thick or a Thin LUN?
You are probably experiencing fragmentation. btrfs-transaction is probably having a hard time with the level of fragmentation on your LUN and causing instability. 

Send us your logs.

Message 18 of 19
kohdee
NETGEAR Expert

Re: Readynas RN204 - Stuck at Booting... 42%

We use pilgrim to display stuff on the LCD during boot, so if the unit was still loading, then it was pilgrim reading the startup status. 

Message 19 of 19
Top Contributors
Discussion stats
  • 18 replies
  • 2482 views
  • 0 kudos
  • 3 in conversation
Announcements