NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.

Forum Discussion

groundswells's avatar
groundswells
Follower
Feb 04, 2017

Readynas RN314 reshape hanging for days

I've been swapping out 3Tb for 6Tb drives on this unit for a few months.  One drive every few weeks.  Before I swapped the third drive, I upgraded the system to 6.6.1.  It rebooted and seemed fine, so I proceeded to swap out the third disk.  The previous 2 rebuilds took no more then 9 hours.  This third drive never showed the status on the LED, it always just flashed DEGRADED.  After a day, I logged in the admin interface only to see the rebuild at about 50% saying it had 2 days left.  After another day the web enterface stopped reponding.  Sshing to the host showed the volumes intact and a mdadm_reshape on eating 99% CPU.  I let it sit for another day or 2.  Still no change on the host, I tried to reboot it never shut down completely so I hard reboot it.

 

After it came back up, no data volumes were present and I recieved emails saying it was resyncing.  Logging onto the host I saw md127_raid5 process on top. 6-7 hours later, it showed complete, but still no data volumes mounted, and an md126_raid5 processes on top.  I tried running a few mdadm commands, but all seemed to hang.  I installed systat to see if any IO was happening, and very little was writing to the disk.  /proc/mdstat showed the. following, but never updated the status even though the md126_raid5 processes was at 99% for hours:

 

Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md126 : active raid5 sda3[4] sdd3[3] sdc3[5] sdb3[6]
8776243968 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/4] [U_UU_]
[>....................] reshape = 0.0% (0/2925414656) finish=16793708509.6min speed=0K/sec

md127 : active raid5 sda4[0] sdc4[1]
5860253824 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [UU_]

md1 : active raid1 sda2[0] sdd2[2] sdc2[1]
523712 blocks super 1.2 [3/3] [UUU]

md0 : active raid1 sda1[4] sdd1[3] sdc1[5]
4192192 blocks super 1.2 [5/3] [U_UU_]

unused devices: <none>

 

I"ve tried rebooting, booting without newly added third disk.  Rebooting in "volume readonly" mode, and nothing seems to help.  I fear I've proded this system too much already, but I'm hoping to salvage my data.  Hopefully I haven't made it even more difficult to troubleshoot.  I'll include a number of commands on the system that may be helpfull.  I'd appreciate any input others may have to at least get me to a point where I can and export the data off this unit.  I'm new to mdadm utilities, but I'm fairly well experienced in linux in general.

 

the first odd dmesg errors:

[ 1199.754989] INFO: task md126_reshape:1383 blocked for more than 200 seconds.
[ 1199.755000] Tainted: P O 4.1.30.x86_64.1 #1
[ 1199.755004] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1199.755010] md126_reshape D ffff88007964bb38 0 1383 2 0x00000000

 

cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md126 : active raid5 sda3[4] sdd3[3] sdc3[5] sdb3[6]
8776243968 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/4] [U_UU_]
[>....................] reshape = 0.0% (0/2925414656) finish=20017759245.0min speed=0K/sec

md127 : active raid5 sda4[0] sdc4[1]
5860253824 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [UU_]

md1 : active raid1 sda2[0] sdd2[2] sdc2[1]
523712 blocks super 1.2 [3/3] [UUU]

md0 : active raid1 sda1[4] sdd1[3] sdc1[5]
4192192 blocks super 1.2 [5/3] [U_UU_]

unused devices: <none>

 

mdadm --query --detail /dev/md126
/dev/md126:
Version : 1.2
Creation Time : Wed Jul 3 17:24:07 2013
Raid Level : raid5
Array Size : 8776243968 (8369.68 GiB 8986.87 GB)
Used Dev Size : 2925414656 (2789.89 GiB 2995.62 GB)
Raid Devices : 5
Total Devices : 4
Persistence : Superblock is persistent

Update Time : Sat Feb 4 13:20:00 2017
State : clean, FAILED, reshaping
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1

Layout : left-symmetric
Chunk Size : 64K

Reshape Status : 0% complete
Delta Devices : 1, (4->5)

Name : 7c6dfb8e:data-0 (local to host 7c6dfb8e)
UUID : 0c0fedf9:4770c7fd:19c55022:ee329eee
Events : 24712

Number Major Minor RaidDevice State
4 8 3 0 active sync /dev/sda3
6 8 19 1 spare rebuilding /dev/sdb3
5 8 35 2 active sync /dev/sdc3
3 8 51 3 active sync /dev/sdd3
8 0 0 8 removed

 

systemctl status readynasd
● readynasd.service - ReadyNAS System Daemon
Loaded: loaded (/lib/systemd/system/readynasd.service; enabled; vendor preset: enabled)
Active: activating (start) since Sat 2017-02-04 15:10:16 EST; 35s ago
Main PID: 14601 (readynasd)
Status: "Importing DB"
CGroup: /system.slice/readynasd.service
└─14601 /usr/sbin/readynasd -v 3 -t

Feb 04 15:10:16 NAS systemd[1]: Starting ReadyNAS System Daemon...
Feb 04 15:10:16 NAS readynasd[14601]: readynasd log started
Feb 04 15:10:16 NAS readynasd[14601]: readynasd started. (restarted=0)
Feb 04 15:10:16 NAS readynasd[14601]: DB (main) schema version: 23 ==> 23
Feb 04 15:10:16 NAS readynasd[14601]: DB (queue) schema version: new ==> 0

 

Any help would be apreciated.  Thanks.

 

Carl

 

 

 

 

1 Reply

Replies have been turned off for this discussion
  • mdgm-ntgr's avatar
    mdgm-ntgr
    NETGEAR Employee Retired

    How did you get on? Your system thinks you have a 5 disk array rather than a 4 disk one.

NETGEAR Academy

Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology! 

Join Us!

ProSupport for Business

Comprehensive support plans for maximum network uptime and business peace of mind.

 

Learn More