NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Power went out during defrag, no space left
Hi guys.
I have a ReadyNas RN104 with 2 x 4 TB drives (WD Red) running in raid.
I'm using the scheduler to automate defrag, scrub and balance. During the defrag the power went out and when it came back on, the defrag process was stopped. Before the drag I had about 1TB of space left, now after it has booted up again it said I had 132kb. I tried to do a disk balance which increased the free disk space to about 2GB.
I've SSH'ing into the NAS and tried to figure out where the space went, but I haven't been able to figure it out.
I've experienced something like this before, but that was due to the snapshotting not working properly and keeping old snapshots even though they should have been deleted. I've disabled snapshotting for all shares and deleted all old snapshot files, so it's not the same problem again.
Any help would be greatly appreciated :)
I have a ReadyNas RN104 with 2 x 4 TB drives (WD Red) running in raid.
I'm using the scheduler to automate defrag, scrub and balance. During the defrag the power went out and when it came back on, the defrag process was stopped. Before the drag I had about 1TB of space left, now after it has booted up again it said I had 132kb. I tried to do a disk balance which increased the free disk space to about 2GB.
I've SSH'ing into the NAS and tried to figure out where the space went, but I haven't been able to figure it out.
/# du -sh /
du: cannot access `/proc/8090/task/8090/fd/4': No such file or directory
du: cannot access `/proc/8090/task/8090/fdinfo/4': No such file or directory
du: cannot access `/proc/8090/fd/4': No such file or directory
du: cannot access `/proc/8090/fdinfo/4': No such file or directory
2.8T /
/# df -H
Filesystem Size Used Avail Use% Mounted on
rootfs 4.0G 959M 2.9G 25% /
tmpfs 10M 4.0K 10M 1% /dev
/dev/md0 4.0G 959M 2.9G 25% /
tmpfs 248M 0 248M 0% /dev/shm
tmpfs 248M 564K 248M 1% /run
tmpfs 248M 0 248M 0% /sys/fs/cgroup
tmpfs 248M 0 248M 0% /media
/dev/md127 3.7T 3.7T 2.1G 100% /data
/dev/md127 3.7T 3.7T 2.1G 100% /apps
/dev/md127 3.7T 3.7T 2.1G 100% /home
/# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
rootfs 65536 19181 46355 30% /
tmpfs 63410 273 63137 1% /dev
/dev/md0 65536 19181 46355 30% /
tmpfs 63410 1 63409 1% /dev/shm
tmpfs 63410 398 63012 1% /run
tmpfs 63410 4 63406 1% /sys/fs/cgroup
tmpfs 63410 1 63409 1% /media
/dev/md127 0 0 0 - /data
/dev/md127 0 0 0 - /apps
/dev/md127 0 0 0 - /home
/# dumpe2fs /dev/md127
dumpe2fs 1.42.10 (18-May-2014)
dumpe2fs: Bad magic number in super-block while trying to open /dev/md127
Couldn't find valid filesystem superblock.
/# fdisk -l
Disk /dev/mtdblock0: 1 MB, 1572864 bytes
255 heads, 63 sectors/track, 0 cylinders, total 3072 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xf7ff1c18
Disk /dev/mtdblock0 doesn't contain a valid partition table
Disk /dev/mtdblock1: 0 MB, 131072 bytes
255 heads, 63 sectors/track, 0 cylinders, total 256 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x322e3535
Disk /dev/mtdblock1 doesn't contain a valid partition table
Disk /dev/mtdblock2: 6 MB, 6291456 bytes
255 heads, 63 sectors/track, 0 cylinders, total 12288 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xe1a0f004
Disk /dev/mtdblock2 doesn't contain a valid partition table
Disk /dev/mtdblock3: 4 MB, 4194304 bytes
255 heads, 63 sectors/track, 0 cylinders, total 8192 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x6e907a39
Disk /dev/mtdblock3 doesn't contain a valid partition table
Disk /dev/mtdblock4: 121 MB, 121634816 bytes
255 heads, 63 sectors/track, 14 cylinders, total 237568 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/mtdblock4 doesn't contain a valid partition table
WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sda: 4000.8 GB, 4000787030016 bytes
256 heads, 63 sectors/track, 484501 cylinders, total 7814037168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sda1 1 4294967295 2147483647+ ee GPT
Partition 1 does not start on physical sector boundary.
WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sdb: 4000.8 GB, 4000787030016 bytes
256 heads, 63 sectors/track, 484501 cylinders, total 7814037168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdb1 1 4294967295 2147483647+ ee GPT
Partition 1 does not start on physical sector boundary.
Disk /dev/md0: 4292 MB, 4292804608 bytes
2 heads, 4 sectors/track, 1048048 cylinders, total 8384384 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Disk /dev/md0 doesn't contain a valid partition table
Disk /dev/md1: 536 MB, 536543232 bytes
2 heads, 4 sectors/track, 130992 cylinders, total 1047936 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Disk /dev/md1 doesn't contain a valid partition table
Disk /dev/md127: 3995.8 GB, 3995818655744 bytes
2 heads, 4 sectors/track, 975541664 cylinders, total 7804333312 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Disk /dev/md127 doesn't contain a valid partition table
I've experienced something like this before, but that was due to the snapshotting not working properly and keeping old snapshots even though they should have been deleted. I've disabled snapshotting for all shares and deleted all old snapshot files, so it's not the same problem again.
Any help would be greatly appreciated :)
24 Replies
I'll try run the defrag again. It was the first power outage in years, so I doubt it'll happen again.mdgm wrote: Factory reset (wipes all data, settings, everything) and restore data from backup.
Or try to expand your volume somehow, run a defrag again, this time preferably with the NAS on a UPS to reduce the risk of an unsafe shutdown in the middle of a defrag and see if the result is any better than after the previous one.
I've tried googling the problem and some suggest that it could be due to the mounts being mounted on top of already existing data.
Like the md127 mount is placed at /data, but before the mount some data could potentially be stored directly at /data and thereby hidden from the 'du' command but not the 'df' command.. Is this something you've heard about before? They mentioned something about binding the mount to somewhere else and checking, but I wasn't sure exactly how to do that :/
Do you know which drives they've used? Like I'm no hardware expert, so I just bought the drives I could find with the best reviews, the Red drives are optimized or something like that for NAS, so I thought that I would be safe - guess it's just a false sense of security then :/StephenB wrote:
There are other failure modes (for instance NAS chassis, lightening/power surge) which could corrupt/destroy data - not to mention theft/fire/flood.ahkjeldsen wrote: I kinda thought the whole "RAID" thing would take care of when drives died, like giving me a chance to replace the single drive that died. I'm have two WD Red 4 TBs in my NAS and they should be pretty durable - hence I'd be pretty unlucky if they both died at the same time, right?
Also, there are many posters here who had a second drive die while waiting for the first replacement to arrive. So it isn't as unusual as you might think. After all, the drives are identical, are in the same chassis, and (thx to the mirroring) have exactly the same usage pattern.
So RAID is useful, but not a substitute for backup.
Most of the files on the NAS is just media files that can be retrieved again from other sources if need be, therefore I didn't really see the need of a backup - though I'd still prefer not having to wipe the whole NAS as it would take plenty of time getting all the media files again.ahkjeldsen wrote: I'll try run the defrag again. It was the first power outage in years, so I doubt it'll happen again.
Just to be clear in my post above I suggested expanding the volume then trying a defrag again.ahkjeldsen wrote:
Like the md127 mount is placed at /data, but before the mount some data could potentially be stored directly at /data and thereby hidden from the 'du' command but not the 'df' command..
That's possible, but if that happened it would only explain less than 4GB. The OS partition is 4GB so with OS files already there....ahkjeldsen wrote:
Do you know which drives they've used? Like I'm no hardware expert, so I just bought the drives I could find with the best reviews, the Red drives are optimized or something like that for NAS, so I thought that I would be safe - guess it's just a false sense of security then :/
The RED drives are good, but no drive is perfect. If you look at places like backblaze you will see that even the best drives can fail. Disks can and do fail at any time.ahkjeldsen wrote:
Most of the files on the NAS is just media files that can be retrieved again from other sources if need be, therefore I didn't really see the need of a backup - though I'd still prefer not having to wipe the whole NAS as it would take plenty of time getting all the media files again.
Backing up and doing a factory reset (wipes all data, settings, everything) then restoring from backup is one option that I would suggest considering.mdgm wrote: Just to be clear in my post above I suggested expanding the volume then trying a defrag again.
Ahh my bad, I need to read closer then.
I'll try by all means to avoid a factory reset, so when you say expanding the volume, that would be inserting a new blank 4TB WD Red disk, right?
When inserting a new disk and expanding, it would also have to resync the data, right? So the defrag should be run after the resync?- Yes
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
