Following an abrupt power-off (powercord was unplugged), my ReadyNas NV has been busy doing an fsck for the past 6 days. I'm able to monitor progress by ssh into the box and looking at /var/log/front/enclosure.log and /tmp/fsck.log. Progress was slow but steady, until 3 days ago when it got to "Pass 2: Checking directory structures". It has been making 0.1% progress each day... Right now it is at 72.8%. CPU usage has been consistently at 95-99%.
Is it normal for an fsck to take this long? Will it ever finish? Can I interrupt it and restart it later? Will it restart where it left off, or have the past 6 days been a waste if I kill the fsck process?
Prior to this last fsck I had logged in and performed a 'kill' on the fsck process because I needed to get access to my files. While I understand the file system may have had errors, I had to take the risk. It worked out, I accessed the files I needed and rebooted... with the intent of letting fsck finish.
So far it has been over a week... progress right now is at 73.4%.. that's 0.6% since Friday evening (it's now Sunday morning).
Could fsck take longer because the filesystem contains many hard-links? I primarily use the device for backups and each night a hardlink is made to all files from the night before. All together I keep 7 daily backups and 4 weekly backups, so each file is basically hardlinked 11 times. (The exception are new, changed or deleted files)
nas1:/var/log/frontview# cat /tmp/fsck.log ***** File system check performed at Sat Oct 9 20:24:13 CST 2010 ***** fsck 1.40.2 (12-Jul-2007) e2fsck 1.40.2 (12-Jul-2007) c was not cleanly unmounted, check forced. Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure
I had to cancel the fsck because I needed access to my files. Next I reboot the device I'll have to ssh into it again and manually kill the fsck.. I don't think I'll ever be able to wait over a week and let it finish....
I have no ReadyNAS but can confirm the problem with high numbers of hardlinks. An old Fileserver (Single-Core-Xeon @2,8Ghz) crashed its external RAID which we used to store rsync-snapshot-backups. Ext4 clearly is not made for very high numbers of hardlinks, we are talking about one million files hardlinked in four complete backuptrees totaling three to six million files. For every single inode fsck has to compute the dangling links which makes 6.000.000 x 6.000.000 = 36.000.000.000.000 calculations.
When this happened first the Xeon took nearly a month for repair. Btw, it was the RAID-controller which spilled bad data over the drive.
My solution: take the raid or an image of the partition and put it on a faster system. My Corei7-2600k@4Ghz made it in sixs days.
Alternatve approach: Let the fsck run until you enter the lengthy "multiply claimed blocks" cycle. At this point your filesystem should(!!!) be mostly(!?!?!) readable as long as you mount it read only. Stop fsck, copy everything, format the corrupted drive, be happy. But beware, hardlinks will not be recognized after stopping fsck, so you will copy every single file as often as it is hardlinked.
Firstly the NV uses EXT3 not EXT4 (I presume EXT3 would probably support less hardlinks than EXT4). The NV does has an Infrant Sparc CPU which is fairly slow.
An NV would be pretty old. Probably would be a good idea for the OP to upgrade e.g. to a Pro 6 (RNDP6000-200 series) and transfer his/her data across the network. As well as a dual-core 2.6Ghz CPU, it does have nice features like an Online Filesystem Check and Raid Scrubbing that you can schedule which I expect would considerably reduce the likelihood of needing to have a lengthy offline filesystem check.
...and as apparently it has been fixed, could we please get this fix included in the next RAIDiator update? I'd guess a lot of people use backup strategies that create lots of hard links, so this is a real gotcha!
