Reply
Highlighted
Aspirant

Readynas OS6 6.10 t185 beta 2 Data Tiering error message

Hi,

 

Hopefully I can get some assistance here. I have been running 6.10 t185 Beta 2 for a while now and I just ran into my second weird issue. The first was when the unit rebooted in the middle of trying to run some HCI bench tests against it, that was a month or so ago.  Today I spun up a new template and thick provisioned 100G VM to do some testing and once the 100G vm was written to the thin provisioned iscsi volume, the unit tried to do a data migration (I have it set to move at 75% full) and then a few seconds later failed with the following:

 

 

Dec 16, 2018 08:18:01 AM   Volume: Data tier migration failed to start for volume VOL01.
Dec 16, 2018 08:17:59 AM   Volume: Data tier migration started for volume VOL01.

 

 

It has now done this 4x since I created the vm.  I'm pretty certain that a graceful restart will force the migration to run and it will work fine. When I had the issue with HCI bench the logs showed a tier migration right before the NAS crashed and it started a migration that was successful after rebooting from the crash.  The migration was sucessful and I have seen no other alerts or error messages until this am. 

 

I have the logs harvested and saved off. 

 

Build:

628x

2x 1Gbe LACP layer 2+3 cifs/smb  (Primarily Plex)

2x 10Gbe iscsi non jumbo frames connected to ESXi 6.5u3 host with 2x 10Gbe iscsi ports.

6x WD Red 8GB

2x Samsung 512GB 860Pro SSD  set for meta data and tiering

 

Thanks,

 

Kirk 

Model: RN628X|ReadyNAS 628X - Ultimate Performance Business Data Storage - 8-Bay
Message 1 of 28

Accepted Solutions
Highlighted
NETGEAR Moderator

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

Message 27 of 28

All Replies
Highlighted
Aspirant

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

So I stopped my vm's and rebooted the NAS and as expected the migration kicked off without error. It is currently at 38%.

 

Thanks,

Kirk

Model: RN628X|ReadyNAS 628X - Ultimate Performance Business Data Storage - 8-Bay
Message 2 of 28
Highlighted
Aspirant

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

The migration took 15 hours and finished successfully. That seems awful long especially considering that it was at 38% one hour in.   I left all vm's but 2 off for 11 hours of the migration as when i had first started the vm's after the NAS reboot they were all running very poorly with 20-160ms disk latency. I don't recall the migration having such a performance penalty in the past. 15 hours seems very long for what could not have been more than 384GB worth of data.

 

Thanks,

Kirk

Model: RN628X|ReadyNAS 628X - Ultimate Performance Business Data Storage - 8-Bay
Message 3 of 28
Highlighted
Guru

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message


@CappyKD wrote:

I don't recall the migration having such a performance penalty in the past. 15 hours seems very long for what could not have been more than 384GB worth of data.

 

RAID sync works below the file-system level, so it really doesn't matter if the volume is empty or full. Every block in the data volume is either read or written, including free space.

Message 4 of 28
Highlighted
Aspirant

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

Thanks for the reply.  I was not aware that the migration of data from the ssd tier to spindles was a raid sync/full sysnc. I thought it would only read / move the blocks that are on SSD. I also expected it to work as more of a background sync and not as a resync with the resync performance penalty. I'm must have spent too much time with Netapp flex cache operations.  I may have to reconsider the use of tiering on this device, especially with the repeated Tier migration failures requiring a reboot for sucessful migration.

 

Thanks,

Kirk

Message 5 of 28
Highlighted
Aspirant

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

Whoops, I meant to reply to you and replied to the thread instead, my bad.

 

 

Thanks for the reply.  I was not aware that the migration of data from the ssd tier to spindles was a raid sync/full sysnc. I thought it would only read / move the blocks that are on SSD. I also expected it to work as more of a background sync and not as a resync with the resync performance penalty. I'm must have spent too much time with Netapp flex cache operations.  I may have to reconsider the use of tiering on this device, especially with the repeated Tier migration failures requiring a reboot for sucessful migration.

 

Thanks,

Kirk

 

Message 6 of 28
Highlighted
Guru

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message


@CappyKD wrote:

 

Thanks for the reply.  I was not aware that the migration of data from the ssd tier to spindles was a raid sync/full sysnc.

 


I could be wrong on that, but I think that is the case.

 

It's hard to explain a 15 hour migration otherwise (as you already pointed out).

Message 7 of 28
Highlighted
Aspirant

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

Thanks, that would explain the long migration time for sure. But I'm still puzzled about the failures when starting the Tier migration until i rebooted. Not sure if there is a correlation.  I never timed my prior Tier migrations but I don't remember them taking that long, nor incurring the perf hit that I saw. 

 

Thanks,

Kirk

Message 8 of 28
Highlighted
Aspirant

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

Any Netgear folks out there that can chime in please?

 

Thanks,

Kirk

Model: RN628X|ReadyNAS 628X - Ultimate Performance Business Data Storage - 8-Bay
Message 9 of 28
Highlighted
Aspirant

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

Are all Netgear employees off for the Holidays?  I would expect someone to want to review the logs to validate if this is bug related, or other.

 

Thanks,

Kirk

Model: RN628X|ReadyNAS 628X - Ultimate Performance Business Data Storage - 8-Bay
Message 10 of 28
Highlighted
NETGEAR Moderator

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

Hi CappyKD,

 

You may upload the logs to a file sharing site and then PM me the download link so we can review it.

 

Regards,

JohnCM_S
NETGEAR Community Team
Message 11 of 28
Highlighted
Aspirant

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

Thank you. The link to the logs has been sent.

 

 

Kirk

Message 12 of 28
Highlighted
NETGEAR Moderator

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

I have reached out to our dev that works in this area to looks at the logs. He is interested in investigating your unit if you are to provide remove access via Secure Diagnostics Mode. You can PM me the 5 digit number from your unit.

It sounds like what you are running into is something that we would like to have a better understanding of your unit in this state.
Message 13 of 28
Highlighted
Aspirant

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

PM Sent.

 

Thank You,

Kirk

Model: RN628X|ReadyNAS 628X - Ultimate Performance Business Data Storage - 8-Bay
Message 14 of 28
Highlighted
Aspirant

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

I can likely induce it to occur again by filling up the SSD tier with data. Would be more than happy to help if needed.

 

Thanks,

Kirk

Model: RN628X|ReadyNAS 628X - Ultimate Performance Business Data Storage - 8-Bay
Message 15 of 28
Highlighted
NETGEAR Moderator

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

Thank you for providing the remote access.

The developer took a look at additional information from your unit and is able to have something internally reproduced. He is working on a fix for the next release.
Message 16 of 28
Highlighted
Aspirant

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

Great news. Glad to hear that.

 

Thanks,

Kirk

Model: RN628X|ReadyNAS 628X - Ultimate Performance Business Data Storage - 8-Bay
Message 17 of 28
Highlighted
Aspirant

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

When a migration is happening, the vm's on an iscsi volume are almost unusable, is there a way to change the priority of the tier migration so it doesn't have so much impact on the vms?  Plex and normal cifs access works fine but vm's on an iscsi volume have major issues. Any thoughts about making this a definable parameter inside the os? 

 

Thanks,

Kirk

Message 18 of 28
Highlighted
Aspirant

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

When I went in to turn off remote support, a notice to install Beta 3 popped up. I did so and the system launched into a tier migration upon reboot and completion of the install. The migration ran fairly quick and was at 89% about 10 minutes ago, I just looked at the LCD on the nas to check the status and it said booting... It has been stuck on booting for the last 6 minutes.

Message 19 of 28
Highlighted
Aspirant

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

I waited until 10 minutes and it was still stuck booting. I powered it off and it booted back up to a Tier migration at 94% (Took much longer than normal and hung at 33% for a while)  Kinda scarey for Beta 3.

Message 20 of 28
Highlighted
Aspirant

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

After the reboot and the 94% restart the Tier migration finished and all now seems normal. Concerned about what occured though. Can we have a dev look into why it rebooted / crashed during the migration please?

Message 21 of 28
Highlighted
NETGEAR Moderator

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

There is an option to show more information during the bootup process in cases like this. If you quickly tap the power button will show more information regarding the booting process and what process the system is trying to work on.

If you rebooted the unit, did the beta 3 get installed? Or did that part of the boot get interuped and you are still on beta 2? There are not specific fixes to tiering in beta 3. There is the new auditting feature and fixes around apps.

 

The reboot does disable the Secure Diagnostics Mode, so you would need to reenable that for us to look into the unit further.

Message 22 of 28
Highlighted
Aspirant

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

The unit is on Beta 3. The Beta 3 install went smooth afaik. Is was the sync that auto started after the Beta 3 install and reboot that died at 94% due to crash or system reboot. Good to know about the power button trick for more info. Secure Diag is reenabled, I will sent the port over via PM. 

 

Thanks for the prompt reply.

 

Kirk

Message 23 of 28
Highlighted
Aspirant

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

Interesting note that the next migration took less than 4 hours. Hmm.

 

Thanks,

Kirk

Message 24 of 28
Highlighted
Aspirant

Re: Readynas OS6 6.10 t185 beta 2 Data Tiering error message

 I am currently in a state where the migration tried to kick off again and is failing. It retrys and fails every 10 minutes. I did not reboot yet in case your team want remote access or needs logs from prior to the reboot. 

 

Thanks,

Kirk

Model: RN628X|ReadyNAS 628X - Ultimate Performance Business Data Storage - 8-Bay
Message 25 of 28
Discussion stats
  • 27 replies
  • 3206 views
  • 3 kudos
  • 4 in conversation
Announcements