NETGEAR is aware of a growing number of phone and online scams. To learn how to stay safe click here.
Forum Discussion
Solved
Detected high command timeouts, short spin downs and where is readynas OS installed please?
Hi all,
Started getting these emails recently
Detected increasing command timeouts errors: [25] on disk 2 (Internal) [ST4000VN000-1H4168, Z3018KBN] 21 times in the past 30 days. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
TLDr
1) Should I be worried about the high command timeout warnings?
2) Where is the Readynas OS stored, on installed disks or on the user installed ones?
Setup info:
- Readynas RN102 - 6.9.3
- No installed apps but isc-dhcp-server and bind installed through ssh.
- Spindown set to 25 min
- JBOD, no snapshots
I'm also getting some other warnings in journalctl
ug 27 12:38:41 NETNAS noflushd[18778]: Spinning up disk 1 (/dev/sdb) after 0:01:33.
Aug 27 12:38:41 NETNAS noflushd[18778]: Spinning up disk 2 (/dev/sdc) after 0:01:30.
Aug 27 12:39:55 NETNAS kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Aug 27 12:39:55 NETNAS kernel: ata2.00: failed command: SMART
Aug 27 12:39:55 NETNAS kernel: ata2.00: cmd b0/d0:01:00:4f:c2/00:00:00:00:00/00 tag 9 pio 512 in
res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Aug 27 15:38:43 NETNAS noflushd[18778]: Spinning up disk 1 (/dev/sdb) after 0:00:47.
Aug 27 15:38:44 NETNAS noflushd[18778]: Spinning up disk 2 (/dev/sdc) after 0:00:46.
Aug 27 15:38:44 NETNAS systemd[1]: Started Radar Update.
Aug 27 15:39:01 NETNAS kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Aug 27 15:39:01 NETNAS kernel: ata2.00: failed command: SMART
Aug 27 15:39:01 NETNAS kernel: ata2.00: cmd b0/d0:01:00:4f:c2/00:00:00:00:00/00 tag 17 pio 512 in
res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Aug 27 15:39:01 NETNAS kernel: ata2.00: status: { DRDY }
Aug 27 15:39:01 NETNAS kernel: ata2: hard resetting link
Aug 27 15:39:01 NETNAS kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Aug 27 15:39:01 NETNAS kernel: ata2.00: configured for UDMA/133
Aug 27 15:39:01 NETNAS kernel: ata2: EH complete
Aug 27 15:39:18 NETNAS readynasd[2334]: Detected increasing command timeouts on disk 2 [25, 26]
Aug 27 15:39:18 NETNAS readynasd[2334]: Detected high command timeouts: [26] on disk 2 (Internal) [ST4000VN000-1H4168, Z3018KBN]. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy..
Aug 27 16:36:22 NETNAS noflushd[18778]: Spinning up disk 1 (/dev/sdb) after 0:02:53.
Aug 27 16:36:22 NETNAS noflushd[18778]: Spinning up disk 2 (/dev/sdc) after 0:02:51.
Aug 27 16:37:33 NETNAS kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Aug 27 16:37:33 NETNAS kernel: ata2.00: failed command: SMART
Aug 27 16:37:33 NETNAS kernel: ata2.00: cmd b0/d0:01:00:4f:c2/00:00:00:00:00/00 tag 2 pio 512 in
res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Aug 27 16:37:33 NETNAS kernel: ata2.00: status: { DRDY }
Aug 27 16:37:33 NETNAS kernel: ata2: hard resetting link
Aug 27 16:37:33 NETNAS kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Aug 27 16:37:33 NETNAS kernel: ata2.00: configured for UDMA/133
Aug 27 16:37:33 NETNAS kernel: ata2: EH complete
Aug 27 16:37:50 NETNAS readynasd[2334]: Detected increasing command timeouts on disk 2 [26, 27]
Aug 27 16:37:50 NETNAS readynasd[2334]: Detected high command timeouts: [27] on disk 2 (Internal) [ST4000VN000-1H4168, Z3018KBN]. This condition often indicates an impending failure. Be prepared to replace this disk to maintain data redundancy.
Aug 27 16:37:52 NETNAS msmtpq[31120]: mail for [ -C /etc/msmtprc email@replaced.com --timeout=60 ] : send was successful
Aug 27 16:37:52 NETNAS readynasd[2334]: Current (temp, start_stop_cnt, power_on_hrs, power_cycle_cnt, load_cycle_cnt) = (35, 25776, 32977, 232, 25784)
Spindowns
I'm noticing that my spin down time is very short
Aug 27 14:23:27 NETNAS noflushd[18778]: Spinning down disk 2 (/dev/sdc). Aug 27 14:24:45 NETNAS noflushd[18778]: Spinning up disk 1 (/dev/sdb) after 0:01:17. Aug 27 14:25:04 NETNAS noflushd[18778]: Spinning up disk 2 (/dev/sdc) after 0:01:35. Aug 27 14:55:04 NETNAS noflushd[18778]: Spinning down disk 1 (/dev/sdb). Aug 27 14:55:07 NETNAS noflushd[18778]: Spinning down disk 2 (/dev/sdc). Aug 27 15:00:15 NETNAS noflushd[18778]: Spinning up disk 1 (/dev/sdb) after 0:05:08. Aug 27 15:00:16 NETNAS noflushd[18778]: Spinning up disk 2 (/dev/sdc) after 0:05:06.
I'd expect this for Disk 2 which has scripts writing to it, but not Disk1, or are both Disks always spun up/ down at the same time?
Readynas OS
I was thinking I could move some scripts, DHCP lease files/ logs to a usb stick to try to reduce the amount Disk2 is accessed but that led me me to wonder where the OS is installed?
I'm not quite understanding the disk layout and was hoping someone could help interpret it for me. Am I close?
My best guess is that
- There's no inherent/ internal storage for the readynas without Disks installed?! This is what's confusing me, how does it boot first time?
Disks are installed with btfrs file system
- A 4Gb partition is created on each disk with RAID for the OS and mirrored? Hence my spindowns not lasting due DHCP leases, or similar files being written?
- I'm wondering whether these disk timouts are therefore not a problem... it's just the OS trying to write to disk and having to wait for the disk to spin up but registering that as a delay?
- Shares are created with /dev/md* but not mirrored
root@NETNAS:/var/log# df -H
Filesystem Size Used Avail Use% Mounted on udev 11M 4.1k 11M 1% /dev /dev/md0 4.0G 876M 2.9G 24% / tmpfs 261M 0 261M 0% /dev/shm tmpfs 261M 586k 260M 1% /run tmpfs 131M 8.9M 122M 7% /run/lock tmpfs 261M 0 261M 0% /sys/fs/cgroup /dev/md126 4.0T 498G 3.5T 13% /Disk2 /dev/md126 4.0T 498G 3.5T 13% /home /dev/md126 4.0T 498G 3.5T 13% /apps /dev/md127 6.0T 5.3T 740G 88% /Disk1 /dev/md126 4.0T 498G 3.5T 13% /run/nfs4/Disk2/local-software /dev/md126 4.0T 498G 3.5T 13% /run/nfs4/Disk2/share2 /dev/sda2 6.1T 973G 5.1T 17% /media/USB_HDD_1 /dev/sdd1 2.0G 3.0M 2.0G 1% /media/USB_FLASH_3
fdisk -l
Disk /dev/mtdblock0: 1.5 MiB, 1572864 bytes, 3072 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mtdblock1: 512 KiB, 524288 bytes, 1024 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mtdblock2: 6 MiB, 6291456 bytes, 12288 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mtdblock3: 4 MiB, 4194304 bytes, 8192 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mtdblock4: 116 MiB, 121634816 bytes, 237568 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sda: 5.5 TiB, 6001175125504 bytes, 11721045167 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: 6A61C6DE-6790-4915-9F55-3C755619AC7A Device Start End Sectors Size Type /dev/sda1 34 262177 262144 128M Microsoft reserved /dev/sda2 264192 11721043967 11720779776 5.5T Microsoft basic data Partition 2 does not start on physical sector boundary. Disk /dev/sdb: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: ABFCFBEA-DF0A-4F79-841B-43EDB02B151B Device Start End Sectors Size Type /dev/sdb1 64 8388671 8388608 4G Linux RAID /dev/sdb2 8388672 9437247 1048576 512M Linux RAID /dev/sdb3 9437248 11721045119 11711607872 5.5T Linux RAID Disk /dev/sdc: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: A9F95F28-4E6C-4ADB-B618-E9C68D96BFEC Device Start End Sectors Size Type /dev/sdc1 64 8388671 8388608 4G Linux RAID /dev/sdc2 8388672 9437247 1048576 512M Linux RAID /dev/sdc3 9437248 7814037119 7804599872 3.6T Linux RAID Disk /dev/md0: 4 GiB, 4290772992 bytes, 8380416 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/md1: 511.4 MiB, 536281088 bytes, 1047424 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/md127: 5.5 TiB, 5996208979968 bytes, 11711345664 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/md126: 3.6 TiB, 3995820883968 bytes, 7804337664 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/sdd: 1.9 GiB, 2031091712 bytes, 3966976 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x0d8b086c Device Boot Start End Sectors Size Id Type /dev/sdd1 2048 3966975 3964928 1.9G 83 Linux
cat /etc/fstab
LABEL=0e35e4f0:Disk2 /Disk2 btrfs defaults,nodatasum 0 0 LABEL=0e35e4f0:Disk1 /Disk1 btrfs defaults,nodatasum 0 0
mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sat Dec 24 16:08:02 2016
Raid Level : raid1
Array Size : 4190208 (4.00 GiB 4.29 GB)
Used Dev Size : 4190208 (4.00 GiB 4.29 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Mon Aug 27 18:39:46 2018
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : 0e35e4f0:0 (local to host 0e35e4f0)
UUID : c0f20fb2:52af152f:7b54b5bb:2b846f3c
Events : 89151
Number Major Minor RaidDevice State
3 8 17 0 active sync /dev/sdb1
2 8 33 1 active sync /dev/sdc1
sc101hell wrote:
1) Should I be worried about the high command timeout warnings?
Yes. 25 iisn't a huge count, but I do recommend monitoring the rate of increase.
sc101hell wrote:
2) Where is the Readynas OS stored, on installed disks or on the user installed ones?
Not sure what you mean by "installed disks" vs "user-installed disks". The only disks are the two you installed. The OS is stored on both of them (on a 4 GB RAID-1 volume - /dev/md0).
4 Replies
Replies have been turned off for this discussion
sc101hell wrote:
1) Should I be worried about the high command timeout warnings?
Yes. 25 iisn't a huge count, but I do recommend monitoring the rate of increase.
sc101hell wrote:
2) Where is the Readynas OS stored, on installed disks or on the user installed ones?
Not sure what you mean by "installed disks" vs "user-installed disks". The only disks are the two you installed. The OS is stored on both of them (on a 4 GB RAID-1 volume - /dev/md0).
Thanks Stephen,
1) No timeouts since I turned off the disk spin down on 28th. The disk is getting the same level of access from the scripts during that time so it's likely related to an error being logged specifically when spinning up (?)
That's helpful now though, I'll start moving files that change somewhere else and see whether I can get then to properly spin down for more than a couple of minutes and whether I then get the errors when they spin back up
2) Thanks, sorry for the lack of clarity there. I was wondering whether it had some small internal memory/ disk installed.
So... being really stupid. How is the OS installed on first boot? Let's say I put two new disks in there, where is the image/ bootstrap held to install onto those?On initial install (or a factory reset), the drives are formatted and the OS is installed from flash memory. When/if the system boots up in safe mode it is also running from flash. Otherwise it boots from the disks.
NETGEAR Academy
Boost your skills with the Netgear Academy - Get trained, certified and stay ahead with the latest Netgear technology!
Join Us!
