This topic has been marked solved and closed to new posts due to inactivity. We hope you'll join the conversation by posting to an open topic or starting a new one.
Readynas OS6 6.10 t185 beta 2 Data Tiering error message
Hi,
Hopefully I can get some assistance here. I have been running 6.10 t185 Beta 2 for a while now and I just ran into my second weird issue. The first was when the unit rebooted in the middle of trying to run some HCI bench tests against it, that was a month or so ago. Today I spun up a new template and thick provisioned 100G VM to do some testing and once the 100G vm was written to the thin provisioned iscsi volume, the unit tried to do a data migration (I have it set to move at 75% full) and then a few seconds later failed with the following:
Dec 16, 2018 08:18:01 AM
Volume: Data tier migration failed to start for volume VOL01.
Dec 16, 2018 08:17:59 AM
Volume: Data tier migration started for volume VOL01.
It has now done this 4x since I created the vm. I'm pretty certain that a graceful restart will force the migration to run and it will work fine. When I had the issue with HCI bench the logs showed a tier migration right before the NAS crashed and it started a migration that was successful after rebooting from the crash. The migration was sucessful and I have seen no other alerts or error messages until this am.
I have the logs harvested and saved off.
Build:
628x
2x 1Gbe LACP layer 2+3 cifs/smb (Primarily Plex)
2x 10Gbe iscsi non jumbo frames connected to ESXi 6.5u3 host with 2x 10Gbe iscsi ports.
6x WD Red 8GB
2x Samsung 512GB 860Pro SSD set for meta data and tiering
The migration took 15 hours and finished successfully. That seems awful long especially considering that it was at 38% one hour in. I left all vm's but 2 off for 11 hours of the migration as when i had first started the vm's after the NAS reboot they were all running very poorly with 20-160ms disk latency. I don't recall the migration having such a performance penalty in the past. 15 hours seems very long for what could not have been more than 384GB worth of data.
Thanks,
Kirk
Model: RN628X|ReadyNAS 628X - Ultimate Performance Business Data Storage - 8-Bay
I don't recall the migration having such a performance penalty in the past. 15 hours seems very long for what could not have been more than 384GB worth of data.
RAID sync works below the file-system level, so it really doesn't matter if the volume is empty or full. Every block in the data volume is either read or written, including free space.
Thanks for the reply. I was not aware that the migration of data from the ssd tier to spindles was a raid sync/full sysnc. I thought it would only read / move the blocks that are on SSD. I also expected it to work as more of a background sync and not as a resync with the resync performance penalty. I'm must have spent too much time with Netapp flex cache operations. I may have to reconsider the use of tiering on this device, especially with the repeated Tier migration failures requiring a reboot for sucessful migration.
Whoops, I meant to reply to you and replied to the thread instead, my bad.
Thanks for the reply. I was not aware that the migration of data from the ssd tier to spindles was a raid sync/full sysnc. I thought it would only read / move the blocks that are on SSD. I also expected it to work as more of a background sync and not as a resync with the resync performance penalty. I'm must have spent too much time with Netapp flex cache operations. I may have to reconsider the use of tiering on this device, especially with the repeated Tier migration failures requiring a reboot for sucessful migration.
Thanks, that would explain the long migration time for sure. But I'm still puzzled about the failures when starting the Tier migration until i rebooted. Not sure if there is a correlation. I never timed my prior Tier migrations but I don't remember them taking that long, nor incurring the perf hit that I saw.
I have reached out to our dev that works in this area to looks at the logs. He is interested in investigating your unit if you are to provide remove access via Secure Diagnostics Mode. You can PM me the 5 digit number from your unit.
It sounds like what you are running into is something that we would like to have a better understanding of your unit in this state.
The developer took a look at additional information from your unit and is able to have something internally reproduced. He is working on a fix for the next release.
When a migration is happening, the vm's on an iscsi volume are almost unusable, is there a way to change the priority of the tier migration so it doesn't have so much impact on the vms? Plex and normal cifs access works fine but vm's on an iscsi volume have major issues. Any thoughts about making this a definable parameter inside the os?
When I went in to turn off remote support, a notice to install Beta 3 popped up. I did so and the system launched into a tier migration upon reboot and completion of the install. The migration ran fairly quick and was at 89% about 10 minutes ago, I just looked at the LCD on the nas to check the status and it said booting... It has been stuck on booting for the last 6 minutes.
I waited until 10 minutes and it was still stuck booting. I powered it off and it booted back up to a Tier migration at 94% (Took much longer than normal and hung at 33% for a while) Kinda scarey for Beta 3.
After the reboot and the 94% restart the Tier migration finished and all now seems normal. Concerned about what occured though. Can we have a dev look into why it rebooted / crashed during the migration please?
There is an option to show more information during the bootup process in cases like this. If you quickly tap the power button will show more information regarding the booting process and what process the system is trying to work on.
If you rebooted the unit, did the beta 3 get installed? Or did that part of the boot get interuped and you are still on beta 2? There are not specific fixes to tiering in beta 3. There is the new auditting feature and fixes around apps.
The reboot does disable the Secure Diagnostics Mode, so you would need to reenable that for us to look into the unit further.
The unit is on Beta 3. The Beta 3 install went smooth afaik. Is was the sync that auto started after the Beta 3 install and reboot that died at 94% due to crash or system reboot. Good to know about the power button trick for more info. Secure Diag is reenabled, I will sent the port over via PM.
I am currently in a state where the migration tried to kick off again and is failing. It retrys and fails every 10 minutes. I did not reboot yet in case your team want remote access or needs logs from prior to the reboot.
Thanks,
Kirk
Model: RN628X|ReadyNAS 628X - Ultimate Performance Business Data Storage - 8-Bay
