× NETGEAR will be terminating ReadyCLOUD service by July 1st, 2023. For more details click here.
Orbi WiFi 7 RBE973
Reply

Very poor performance on iSCSI

Rimbalza
Aspirant

Very poor performance on iSCSI

Starting situation: Windows 2012R2 with MPIO with 2 gigabit adapters. RN316 with 6.9.3, 6 Drives full.

When a backup starts from the other NAS (same model) to this one (both DR or rsync) the iSCSI completely hangs.

I have replicated machines from another Hyper-v on this iSCSI and the replica stops, the disk is lost, then re-appears but writes at some kb/sec. Just trying to create a directory on the disk takes hours (literally).

So I started removing MPIO, using only 1 adapter. Same thing. Changed adpaters/cabling/switch: same thing.

I removed the backups and only kept the iSCSI for the Hyper-V replica, it works but with very low performance. I mean 300 mbit peak going up and down to 0. It takes forever to replicate VMs. I use a thick 2.5T volume (with and without sync writes, problems are the same)

I already discarded ReadyNAS solutions to have live hyper-v machines because it was badly off with performance, but I thought having replicas can be done. I was wrong.

Looking at the data I get from SNMP I discovered the NAS is completely overloaded on the CPU side. The attached graph is from the NAS receiving a single VM replica and doing nothing else.

Capture.JPG

Capture2.JPG

The second graph is the network (peaks are at 300 mbit).

From the same server I can pump to another brand NAS near to full gigabit speed (same cisco switch, same network adapter).

What can I check to achieve decent performance and maybe have a backup starting without killing all the rest?

Thanks

Model: RN31600|ReadyNAS 300 Series 6- Bay
Message 1 of 22
Rimbalza
Aspirant

Re: Very poor performance on iSCSI

Just to add: the volume is not compressed

Message 2 of 22
hezner1
Tutor

Re: Very poor performance on iSCSI

I don't know if it would help you in your situation but we found that after we upgraded to 6.9.3 OS we had very slow save speeds.  Strict sync had been disabled in the 638 OS but is enabled by default in the 6.9 OS.  When we disabled the strict sync that problem was resolved.   We are not using iscsi so I don't know if this is your issue but It is an easy setting to change and might be helpful.  Here is the link to my post about the issue that references others who researched the problem originally.  https://community.netgear.com/t5/ReadyNAS-in-Business/Slow-saves-for-Revit-2018-after-OS-upgrade-fro...

Message 3 of 22
Rimbalza
Aspirant

Re: Very poor performance on iSCSI

Thanks for your reply, but I'm not using SMB and tested sync writes on and off ono iSCSI too with no success.

Today I'm trying to run a backup and I get horrible performance and not constant. In the graph below up is IOPS and down Network.

It goes up, then drops and then slowly gets better, but nothing acceptable.

Capture.JPG

We'll dump Netgear stuff asap...

Message 4 of 22
mdgm-ntgr
NETGEAR Employee Retired

Re: Very poor performance on iSCSI

What about CoW, bit-rot protection and snapshots? Have those always been disabled on this LUN?

Is the LUN thin/thick?

Message 5 of 22
Rimbalza
Aspirant

Re: Very poor performance on iSCSI

You can find the relevant info in the first post.

BTW, no CoW and compression, thick LUN.

Edit: and no snaps

Message 6 of 22
mdgm-ntgr
NETGEAR Employee Retired

Re: Very poor performance on iSCSI

Please send in your logs (see the Sending Logs link in my sig).

Message 7 of 22
Rimbalza
Aspirant

Re: Very poor performance on iSCSI

Right now I killed the long running backup, it will terminate in a month at this rate. I tried to offline the LUN from Windows disk manager and it is hung since 30 min.

This is pretty reproducible, when some sustained iSCSI activity is ongoing all other iSCSI tasks also on other LUNs & portals just hang crashing the apps on them.

I did the same things on the other identical device but with 6.9.0 and is is the same. On this unit also MPIO never catches up, only one link is used despite both being configured.

I repeat this again, the same servers on other vendor's devices on the same gear pump full gigabit speed and several times balanced MPIO reaches 1,5/1,7 gbit.

Message 8 of 22
Rimbalza
Aspirant

Re: Very poor performance on iSCSI

I would like to send, but since I stopped the backup the device is hung also in the web interface or very very very slow...

I'm currently stuck on http://mydevice/dbbroker after clicking download...

Message 9 of 22
Rimbalza
Aspirant

Re: Very poor performance on iSCSI

Log sent

Message 10 of 22
Rimbalza
Aspirant

Re: Very poor performance on iSCSI

Currently disk was not even responding, rescan is hung since 20 minutes

Untitled.png

and this is the cpu load in the last hours (top of scale is 100%)

cpu.JPG

 

 

Message 11 of 22
mdgm-ntgr
NETGEAR Employee Retired

Re: Very poor performance on iSCSI

Can you disable quotas on the data volume (go to System > Volumes, click on the Settings Wheel for the volume, go down to Settings, uncheck Quota and apply your changes)?

The logs show a large number of snapshots on your DR shares (see e.g. snapper.log).

Message 12 of 22
Rimbalza
Aspirant

Re: Very poor performance on iSCSI

You see a lot of snaps that are for "dead" DR shares from the other NAS. There is no single snap related to iSCSI.

Quota is now disabled, but the server is hung waiting for the rescan, so no operation is possible... I have to wait the time to reboot the production environment, also considering that in this way my replica VM are all in critical state...

 

Message 13 of 22
Rimbalza
Aspirant

Re: Very poor performance on iSCSI

Should I remove all the ReadyDR shares from the device?

Message 14 of 22
Rimbalza
Aspirant

Re: Very poor performance on iSCSI

I removed all the ReadyDR shares, no more snapshots on the device. None. Zero.

No way to mount the iSCSI volumes. I restarted the device and it sent me the "restarting" email but never restarted. Done this several times with no success, tried to login via SSH and suddenly the device stopped responding to pings but never rebooted.

I took my car and run 30 km to the device <insert really bad words here> to find it with the display saying "Rebooting". I power cycled it and now it boots, the interface goes up but there is no way to mount the iSCSI targets (that I see), Win server remain hung trying to mount.

This is an embarassing device to sell. Really.

Message 15 of 22
Rimbalza
Aspirant

Re: Very poor performance on iSCSI

After a night trying to have the environment work again (spoiler: we moved to competitor’s units) we have some findings that I think it’s useful to share.

First of all the device is heavily underpowered for what it’s supposed to do. Each time there’s a long running operation the cpu tops to 100% and any operation on the device is affected: mounting and dismounting other iSCSI volumes may take an hour under load. I don’t know about the memory but I suppose a lot more is needed for proper caching in a multitasking environment (look at the graphs in previous posts where performance slowly builds up and then drops instantly). If you start an rsync on a share during an iSCSI backup they both take ages and probably will not complete.

Disabling quotas (not used in any case) may have some effect but we disabled them while removing any sort of snapshot (not related to iSCSI) on the device, so I don’t know exactly. This is another big point and pretty reproducible: we had iSCSI volumes for VM replica and backup from several servers. You need to carefully time the backups to not overlap or you’ll crash all of them. I understand having performance issues, but crashes* are not allowed in any case. Btw if you use this nas to receive ReadyDR shares from another unit, same model, in theory you end up with having shares and their snapshots replicated. Here is where the problem gets bigger: using the Netgear default configuration, snaps accumulate and when replicated they completely hang the receiving device, kicking the CPU to steady 100% and locking any other activity like iSCSI.

So basically intend this as a single task / serial task device: try to never have two operations running at the same time and never mix iSCSI/shares activity (like rsync replicas). And if you use ReadyDR shares, be sure to trim the originating snaps and use the receiving unit as a real “disaster recovery” device and not for something live. We already tested it to have live VM from a couple of servers on iSCSI volumes and the result was “use it only for dead replicas”.

Now we use the units as iSCSI volumes to receive the backups from other storage devices (other brand).

 cpu.JPG

You can see highlighted the CPU of the Netgear while receiving data al 300 mbit (we’re throttling at source not to kill anything after a full night working…), from a single device. On the left the cpu when there were snaps from ReadyDR shares. Holes in the graph is the device crashing…

This is without using CoW&Compression, I suppose it can only get worse.

To recap: thanks to support for pointing me to the snaps problem (they also offered to connect remotely but our policy doesn’t allow that). Besides that consider this as a huge USB drive, they should advertise it like that. It is an inexpensive device, but you get what you pay for. And, try to not have it in a remote location, as when it’s hung you need to physically pull the cord to reboot.

 

(*) with crash on iSCSI I mean that no data is going to be read or written on the device, and operation like mount/unmount just hang.

Message 16 of 22
mdgm-ntgr
NETGEAR Employee Retired

Re: Very poor performance on iSCSI

Your configuration was less than optimal and for your use case a different model in our range would have been more suitable. Pre-Sales advice is available and the community is a great place to ask Pre-Sales questions as well.

Message 17 of 22
Rimbalza
Aspirant

Re: Very poor performance on iSCSI

Please highlight the "less than optimal" configuration points, so that anyone can benefit.

Also explain the 100% cpu usage and iSCSI fails, if you have some time.

Message 18 of 22
mdgm-ntgr
NETGEAR Employee Retired

Re: Very poor performance on iSCSI

Some things were already mentioned above e.g. quotas.

 

It does sound like you were trying to do too much with this unit. Using a less powerful unit than what you require to do what you want is going to lead to problems.

 

That’s why seeking Presales advice is recommended.

Message 19 of 22
WingDog
Guide

Re: Very poor performance on iSCSI

@Rimbalza
the only one way to fix it - switch to Synology. I have many RND devices and all of them has poor perfomance and unpredictable lags and hangs.

 

Message 20 of 22
Rimbalza
Aspirant

Re: Very poor performance on iSCSI

It is exactly what I did (without writing the name). Production is on big iron, replicas on $other_brand, big-flat-old-backups on 6x6T ReadyNas.

As someone wrote above probably pre-sales would have warned the device was not adequate for our scenario. Just noting that for about same money (sliglthy less to be precise) with $other_brand you have wildly faster devices on mpio iSCSI. Same load, same servers, same network and 3T disks instead of 6T on ReadyNAS.

Message 21 of 22
WingDog
Guide

Re: Very poor performance on iSCSI

addon:

and you don't need to spend time (that has a price and the price is big) for writting posts to forums from year to year to proof device problems.

"check this checkbox", "uncheck those checkbox", "ask network vendor", "ask SW vendor".

Synology just works.

Message 22 of 22
Top Contributors
Discussion stats
  • 21 replies
  • 5774 views
  • 0 kudos
  • 4 in conversation
Announcements