I've been swapping out 3Tb for 6Tb drives on this unit for a few months. One drive every few weeks. Before I swapped the third drive, I upgraded the system to 6.6.1. It rebooted and seemed fine, so I proceeded to swap out the third disk. The previous 2 rebuilds took no more then 9 hours. This third drive never showed the status on the LED, it always just flashed DEGRADED. After a day, I logged in the admin interface only to see the rebuild at about 50% saying it had 2 days left. After another day the web enterface stopped reponding. Sshing to the host showed the volumes intact and a mdadm_reshape on eating 99% CPU. I let it sit for another day or 2. Still no change on the host, I tried to reboot it never shut down completely so I hard reboot it.
After it came back up, no data volumes were present and I recieved emails saying it was resyncing. Logging onto the host I saw md127_raid5 process on top. 6-7 hours later, it showed complete, but still no data volumes mounted, and an md126_raid5 processes on top. I tried running a few mdadm commands, but all seemed to hang. I installed systat to see if any IO was happening, and very little was writing to the disk. /proc/mdstat showed the. following, but never updated the status even though the md126_raid5 processes was at 99% for hours:
md127 : active raid5 sda4[0] sdc4[1] 5860253824 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [UU_]
md1 : active raid1 sda2[0] sdd2[2] sdc2[1] 523712 blocks super 1.2 [3/3] [UUU]
md0 : active raid1 sda1[4] sdd1[3] sdc1[5] 4192192 blocks super 1.2 [5/3] [U_UU_]
unused devices: <none>
I"ve tried rebooting, booting without newly added third disk. Rebooting in "volume readonly" mode, and nothing seems to help. I fear I've proded this system too much already, but I'm hoping to salvage my data. Hopefully I haven't made it even more difficult to troubleshoot. I'll include a number of commands on the system that may be helpfull. I'd appreciate any input others may have to at least get me to a point where I can and export the data off this unit. I'm new to mdadm utilities, but I'm fairly well experienced in linux in general.
the first odd dmesg errors:
[ 1199.754989] INFO: task md126_reshape:1383 blocked for more than 200 seconds. [ 1199.755000] Tainted: P O 4.1.30.x86_64.1 #1 [ 1199.755004] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1199.755010] md126_reshape D ffff88007964bb38 0 1383 2 0x00000000
md127 : active raid5 sda4[0] sdc4[1] 5860253824 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [UU_]
md1 : active raid1 sda2[0] sdd2[2] sdc2[1] 523712 blocks super 1.2 [3/3] [UUU]
md0 : active raid1 sda1[4] sdd1[3] sdc1[5] 4192192 blocks super 1.2 [5/3] [U_UU_]
unused devices: <none>
mdadm --query --detail /dev/md126 /dev/md126: Version : 1.2 Creation Time : Wed Jul 3 17:24:07 2013 Raid Level : raid5 Array Size : 8776243968 (8369.68 GiB 8986.87 GB) Used Dev Size : 2925414656 (2789.89 GiB 2995.62 GB) Raid Devices : 5 Total Devices : 4 Persistence : Superblock is persistent
Update Time : Sat Feb 4 13:20:00 2017 State : clean, FAILED, reshaping Active Devices : 3 Working Devices : 4 Failed Devices : 0 Spare Devices : 1
Layout : left-symmetric Chunk Size : 64K
Reshape Status : 0% complete Delta Devices : 1, (4->5)
Name : 7c6dfb8e:data-0 (local to host 7c6dfb8e) UUID : 0c0fedf9:4770c7fd:19c55022:ee329eee Events : 24712
Number Major Minor RaidDevice State 4 8 3 0 active sync /dev/sda3 6 8 19 1 spare rebuilding /dev/sdb3 5 8 35 2 active sync /dev/sdc3 3 8 51 3 active sync /dev/sdd3 8 0 0 8 removed
systemctl status readynasd ● readynasd.service - ReadyNAS System Daemon Loaded: loaded (/lib/systemd/system/readynasd.service; enabled; vendor preset: enabled) Active: activating (start) since Sat 2017-02-04 15:10:16 EST; 35s ago Main PID: 14601 (readynasd) Status: "Importing DB" CGroup: /system.slice/readynasd.service └─14601 /usr/sbin/readynasd -v 3 -t
Feb 04 15:10:16 NAS systemd[1]: Starting ReadyNAS System Daemon... Feb 04 15:10:16 NAS readynasd[14601]: readynasd log started Feb 04 15:10:16 NAS readynasd[14601]: readynasd started. (restarted=0) Feb 04 15:10:16 NAS readynasd[14601]: DB (main) schema version: 23 ==> 23 Feb 04 15:10:16 NAS readynasd[14601]: DB (queue) schema version: new ==> 0
