× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Re: ReadyNAS Balance and ESXI hosts

Retired_Member
Not applicable

ReadyNAS Balance and ESXI hosts

Model ReadyNAS 4312 (did not have that in Model selection, when creating this discussin).

Firmware 6.7.5

 

*My "setup"

ESXi 6.0

EMC SAN for Production Servers (10GB Base-T)

Readynas 4312 is used for non-productions servers and SMB Shares for backups.  The Readynas has two 10GB base t, one of those NIC's is in a physically seperate (different switches) and seperate network (10.0.10.0/24) for Iscsi traffic (where VM's live), and the other NIC is on physically seperated network, used for SMB shares (10.3.81.1/24).

 

I have performed 2 balances, and each time I lose access to my ESXi hosts (they say "Not responding") and my only fix is to shut down and restart the NAS.  I do wait until the balance is complete to do a shutdown, but I still cannot get access to my ESXi hosts after the balance is complete.  

 

When I turn the ReadyNAS back on, all is well.  

 

During the time the balance starts and when complete, and I loose access to ESXi hosts, the VM's on the EMC SAN are still running and I can RDP into them.  I also get alerts of high access reads and writes to the datastores in the EMC SAN.

 

The VM's running on the Readnas, I can ping them...but I cannot RDP into them (until I reboot the ReadyNAS). 

 

 

I am assuming the balance may be saturating the wrong network with iScsi traffic, but I wanted to see if anyone else has any issues when running a balance?

Message 1 of 10

Accepted Solutions
Retired_Member
Not applicable

Re: ReadyNAS Balance and ESXI hosts

Just an update, since I really dont like when people dont update a discussion.

On firmeware 6.8.0, and have had a Balance and defrag without issues. (so it seems Netgear has fixed the issue IMO)

 

 

View solution in original post

Message 10 of 10

All Replies
jak0lantash
Mentor

Re: ReadyNAS Balance and ESXI hosts


@Retired_Member wrote:

 

I am assuming the balance may be saturating the wrong network with iScsi traffic, but I wanted to see if anyone else has any issues when running a balance?


The BTRFS balance won't  saturate any network interface, but it is stressful for the volume, so it may increase read and write latency, which maybe leads to issues. You said you lose access to ESXi hosts and that the RR4312 hosts datastore(s) and SMB shares. I'm not sure why you would lose access to the ESXi hosts while still reaching VMs, I would understand why some VMs would hang, but hosts? Are you running a nested environment? I don't understand why a balance on a storage device would cut you off from the hosts while you can still reach some VMs running on it.

 

Can you download the logs of the RR4312 and post here an extract of btrfs.log?

 

Message 2 of 10
Retired_Member
Not applicable

Re: ReadyNAS Balance and ESXI hosts

Label: '2fe72984:root'  uuid: b1398ab0-5356-42be-b23e-f414e56d581d

                Total devices 1 FS bytes used 561.41MiB

                devid    1 size 4.00GiB used 2.05GiB path /dev/md0

 

Label: '2fe72984:TSS'  uuid: 3709eb1d-0149-4884-8f68-3f11f380aecf

                Total devices 1 FS bytes used 22.97TiB

                devid    1 size 36.36TiB used 23.62TiB path /dev/md127

 

=== filesystem /TSS ===

Data, single: total=23.57TiB, used=22.94TiB

System, DUP: total=32.00MiB, used=3.00MiB

Metadata, DUP: total=27.50GiB, used=25.84GiB

GlobalReserve, single: total=512.00MiB, used=0.00B

=== subvolume /TSS ===

ID 257 gen 167454 top level 5 path .purge

ID 258 gen 204170 top level 5 path ._share

ID 259 gen 250423 top level 5 path .apps

ID 260 gen 255443 top level 5 path home

ID 261 gen 11 top level 5 path .vault

ID 262 gen 167447 top level 5 path .timemachine

ID 263 gen 255141 top level 5 path NAS_dev-ws

ID 264 gen 80205 top level 263 path NAS_dev-ws/.snapshots

ID 265 gen 32 top level 260 path home/TSS/gcarson

ID 280 gen 255306 top level 5 path NAS_veeam

ID 282 gen 52 top level 280 path NAS_veeam/.snapshots

ID 392 gen 254669 top level 5 path NAS_SQL

ID 393 gen 80194 top level 392 path NAS_SQL/.snapshots

ID 394 gen 254705 top level 5 path NAS_ORACLE

ID 395 gen 80185 top level 394 path NAS_ORACLE/.snapshots

ID 398 gen 373 top level 260 path home/TSS/veeam1_

ID 483 gen 1034 top level 260 path home/TSS/jharrison

ID 515 gen 250329 top level 5 path NAS_mdotbackup2

ID 516 gen 52358 top level 515 path NAS_mdotbackup2/.snapshots

ID 7840 gen 16938 top level 260 path home/TSS/yshao

ID 7873 gen 17056 top level 260 path home/TSS/rgreen

ID 8897 gen 255443 top level 5 path naslun1

ID 12780 gen 251293 top level 5 path NAS_mdotbackup1

ID 12782 gen 52352 top level 12780 path NAS_mdotbackup1/.snapshots

ID 18904 gen 204558 top level 5 path NAS_sccm

ID 18905 gen 80216 top level 18904 path NAS_sccm/.snapshots

ID 19190 gen 34811 top level 260 path home/TSS/sccm1_

ID 20249 gen 255443 top level 5 path naslun2

ID 21164 gen 49603 top level 260 path home/TSS/stg-app_

ID 21422 gen 52700 top level 260 path home/TSS/admin.service

ID 21666 gen 55685 top level 260 path home/TSS/dev-ws_

ID 33579 gen 167152 top level 5 path git

ID 33580 gen 167146 top level 33579 path git/.snapshots

=== btrfs dump-super /dev/md0 /dev/md127

superblock: bytenr=65536, device=/dev/md0

---------------------------------------------------------

csum_type                         0 (crc32c)

csum_size                           4

csum                                     0xb771823b [match]

bytenr                                  65536

flags                                       0x1

                                                ( WRITTEN )

magic                                    _BHRfS_M [match]

fsid                                         b1398ab0-5356-42be-b23e-f414e56d581d

label                                      2fe72984:root

generation                          152043

root                                       51445760

sys_array_size                  129

chunk_root_generation 16925

root_level                           0

chunk_root                        20987904

chunk_root_level            0

log_root                               0

log_root_transid              0

log_root_level                  0

total_bytes                         4290772992

bytes_used                        588685312

sectorsize                            4096

nodesize                              16384

leafsize                 16384

stripesize                             4096

root_dir                               6

num_devices                     1

compat_flags                     0x0

compat_ro_flags                              0x0

incompat_flags                 0x161

                                                ( MIXED_BACKREF |

                                                  BIG_METADATA |

                                                  EXTENDED_IREF |

                                                  SKINNY_METADATA )

cache_generation           18446744073709551615

uuid_tree_generation   152043

dev_item.uuid                  5ae5ecc8-acb7-4d3b-90df-57704d75e19a

dev_item.fsid                    b1398ab0-5356-42be-b23e-f414e56d581d [match]

dev_item.type                  0

dev_item.total_bytes    4290772992

dev_item.bytes_used    2198994944

dev_item.io_align           4096

dev_item.io_width         4096

dev_item.sector_size     4096

dev_item.devid                1

dev_item.dev_group      0

dev_item.seek_speed    0

dev_item.bandwidth     0

dev_item.generation     0

sys_chunk_array[2048]:

                item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520)

                                length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP

                                io_align 65536 io_width 65536 sector_size 4096

                                num_stripes 2 sub_stripes 0

                                                stripe 0 devid 1 offset 20971520

                                                dev_uuid 5ae5ecc8-acb7-4d3b-90df-57704d75e19a

                                                stripe 1 devid 1 offset 29360128

                                                dev_uuid 5ae5ecc8-acb7-4d3b-90df-57704d75e19a

backup_roots[4]:

                backup 0:

                                backup_tree_root:          50757632             gen: 152042        level: 0

                                backup_chunk_root:      20987904             gen: 16925           level: 0

                                backup_extent_root:     50692096             gen: 152042        level: 1

                                backup_fs_root:                               50397184             gen: 152042        level: 2

                                backup_dev_root:           30048256             gen: 147651        level: 0

                                backup_csum_root:       50462720             gen: 152042        level: 1

                                backup_total_bytes:      4290772992

                                backup_bytes_used:      588685312

                                backup_num_devices:   1

 

                backup 1:

                                backup_tree_root:          51445760             gen: 152043        level: 0

                                backup_chunk_root:      20987904             gen: 16925           level: 0

                                backup_extent_root:     51314688             gen: 152043        level: 1

                                backup_fs_root:                               51003392             gen: 152043        level: 2

                                backup_dev_root:           30048256             gen: 147651        level: 0

                                backup_csum_root:       51052544             gen: 152043        level: 1

                                backup_total_bytes:      4290772992

                                backup_bytes_used:      588685312

                                backup_num_devices:   1

 

                backup 2:

                                backup_tree_root:          49954816             gen: 152040        level: 0

                                backup_chunk_root:      20987904             gen: 16925           level: 0

                                backup_extent_root:     49790976             gen: 152040        level: 1

                                backup_fs_root:                               50003968             gen: 152041        level: 2

                                backup_dev_root:           30048256             gen: 147651        level: 0

                                backup_csum_root:       50102272             gen: 152041        level: 1

                                backup_total_bytes:      4290772992

                                backup_bytes_used:      588677120

                                backup_num_devices:   1

 

                backup 3:

                                backup_tree_root:          50348032             gen: 152041        level: 0

                                backup_chunk_root:      20987904             gen: 16925           level: 0

                                backup_extent_root:     50266112             gen: 152041        level: 1

                                backup_fs_root:                               50003968             gen: 152041        level: 2

                                backup_dev_root:           30048256             gen: 147651        level: 0

                                backup_csum_root:       50102272             gen: 152041        level: 1

                                backup_total_bytes:      4290772992

                                backup_bytes_used:      588681216

                                backup_num_devices:   1

 

 

superblock: bytenr=65536, device=/dev/md127

---------------------------------------------------------

csum_type                         0 (crc32c)

csum_size                           4

csum                                     0x430a8dee [match]

bytenr                                  65536

flags                                       0x1

                                                ( WRITTEN )

magic                                    _BHRfS_M [match]

fsid                                         3709eb1d-0149-4884-8f68-3f11f380aecf

label                                      2fe72984:TSS

generation                          255443

root                                       7407750316032

sys_array_size                  129

chunk_root_generation 253996

root_level                           1

chunk_root                        36156708421632

chunk_root_level            1

log_root                               0

log_root_transid              0

log_root_level                  0

total_bytes                         39983460909056

bytes_used                        25251452792832

sectorsize                            4096

nodesize                              32768

leafsize                 32768

stripesize                             4096

root_dir                               6

num_devices                     1

compat_flags                     0x0

compat_ro_flags                              0x0

incompat_flags                 0x161

                                                ( MIXED_BACKREF |

                                                  BIG_METADATA |

                                                  EXTENDED_IREF |

                                                  SKINNY_METADATA )

cache_generation           18446744073709551615

uuid_tree_generation   255443

dev_item.uuid                  8f73134e-0657-4750-a01c-ac6de62bbcd7

dev_item.fsid                    3709eb1d-0149-4884-8f68-3f11f380aecf [match]

dev_item.type                  0

dev_item.total_bytes    39983460909056

dev_item.bytes_used    26105952075776

dev_item.io_align           4096

dev_item.io_width         4096

dev_item.sector_size     4096

dev_item.devid                1

dev_item.dev_group      0

dev_item.seek_speed    0

dev_item.bandwidth     0

dev_item.generation     0

sys_chunk_array[2048]:

                item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 36156708225024)

                                length 33554432 owner 2 stripe_len 65536 type SYSTEM|DUP

                                io_align 65536 io_width 65536 sector_size 4096

                                num_stripes 2 sub_stripes 1

                                                stripe 0 devid 1 offset 604621504512

                                                dev_uuid 8f73134e-0657-4750-a01c-ac6de62bbcd7

                                                stripe 1 devid 1 offset 604655058944

                                                dev_uuid 8f73134e-0657-4750-a01c-ac6de62bbcd7

backup_roots[4]:

                backup 0:

                                backup_tree_root:          7407750316032                 gen: 255443        level: 1

                                backup_chunk_root:      36156708421632               gen: 253996        level: 1

                                backup_extent_root:     7407735472128                 gen: 255443        level: 2

                                backup_fs_root:                               644336418816    gen: 204171        level: 0

                                backup_dev_root:           6245190500352                 gen: 253996        level: 1

                                backup_csum_root:       7407717056512                 gen: 255443        level: 0

                                backup_total_bytes:      39983460909056

                                backup_bytes_used:      25251452792832

                                backup_num_devices:   1

 

                backup 1:

                                backup_tree_root:          7407741272064                 gen: 255440        level: 1

                                backup_chunk_root:      36156708421632               gen: 253996        level: 1

                                backup_extent_root:     7407725740032                 gen: 255440        level: 2

                                backup_fs_root:                               644336418816    gen: 204171        level: 0

                                backup_dev_root:           6245190500352                 gen: 253996        level: 1

                                backup_csum_root:       7407713648640                 gen: 255441        level: 0

                                backup_total_bytes:      39983460909056

                                backup_bytes_used:      25251452764160

                                backup_num_devices:   1

 

                backup 2:

                                backup_tree_root:          7407748972544                 gen: 255441        level: 1

                                backup_chunk_root:      36156708421632               gen: 253996        level: 1

                                backup_extent_root:     7407728394240                 gen: 255441        level: 2

                                backup_fs_root:                               644336418816    gen: 204171        level: 0

                                backup_dev_root:           6245190500352                 gen: 253996        level: 1

                                backup_csum_root:       7407716925440                 gen: 255442        level: 0

                                backup_total_bytes:      39983460909056

                                backup_bytes_used:      25251452768256

                                backup_num_devices:   1

 

                backup 3:

                                backup_tree_root:          7407747170304                 gen: 255442        level: 1

                                backup_chunk_root:      36156708421632               gen: 253996        level: 1

                                backup_extent_root:     7407737438208                 gen: 255442        level: 2

                                backup_fs_root:                               644336418816    gen: 204171        level: 0

                                backup_dev_root:           6245190500352                 gen: 253996        level: 1

                                backup_csum_root:       7407717056512                 gen: 255443        level: 0

                                backup_total_bytes:      39983460909056

                                backup_bytes_used:      25251452784640

                                backup_num_devices:   1

 

 

Message 3 of 10
Retired_Member
Not applicable

Re: ReadyNAS Balance and ESXI hosts

I am guessing at this too, since the ReadyNAS is the only new thing added to my network...when I had my support ticket with VMware the only thing I did to bring back my ESXi hosts....was turn off the ReadyNAS.  A few days later I did a "Balance" again.....with same resutls...In vCenter the ESXi hosts stated "not responding"...I could not log into the hosts of each ESXi server.  I again shut off the ReadyNAS and all came back online.

 

Message 4 of 10
mdgm-ntgr
NETGEAR Employee Retired

Re: ReadyNAS Balance and ESXI hosts

Your volume doesn't need balancing. It's well balanced looking at the output you've provided.

Message 5 of 10
Retired_Member
Not applicable

Re: ReadyNAS Balance and ESXI hosts

Well I did 2 balancing within a week, the second was really only a test to see if the "issue" would happen again, and it did.

 

I first did a balance by suggestion from Netgear support.

 

I will do some testing and disable NIC's on my next balance, I will post updates.

Message 6 of 10
jak0lantash
Mentor

Re: ReadyNAS Balance and ESXI hosts

Metadata allocation is a bit high. Are you using bit rot protection, compression, etc?

You should disable the quotas on the volume (if you don't have any quotas on the shares).

If you're running databases, maybe you need a faster volume, like RAID10.

 

That said, I still don't see the connection between the balance and your ESXi hosts.

Message 7 of 10
Retired_Member
Not applicable

Re: ReadyNAS Balance and ESXI hosts

I will try your suggestions in regards to the quotas and will look at RAID 10.

 

I will say, if you want to have a webex session, I can show the issue live.

Message 8 of 10
Retired_Member
Not applicable

Re: ReadyNAS Balance and ESXI hosts

Well My Scheduled Defrag (Aug 10th) and Balance (Aug 16th) happened. 

Datasore Read Latency issues, Host disk SCSI aborts, the ESXi Hosts "not responding" in vCenter.

 

I restart the NAS...everything is corrected. 

 

One thing i noticed, all the alerts from NAS came in at once...even from a week ago.  Defragmentation started, Defragmentation Complete, Balance Started, Balance Complete, and even the "system is rebooting" message.  All alerts came together, right after the NAS has been restarted.  I see a new update (6.8.0) came out, maybe this will fix my "werid" issue.  

Message 9 of 10
Retired_Member
Not applicable

Re: ReadyNAS Balance and ESXI hosts

Just an update, since I really dont like when people dont update a discussion.

On firmeware 6.8.0, and have had a Balance and defrag without issues. (so it seems Netgear has fixed the issue IMO)

 

 

Message 10 of 10
Top Contributors
Discussion stats
  • 9 replies
  • 3796 views
  • 0 kudos
  • 3 in conversation
Announcements