iSCSI w. Server 2008 R2 locks up on large writes

Odd. I can dump 500GB of data to an iSCSI LUN from server 2008 R2 without running into that. You mentioned that the UI is slow, is it always slow or only during/after the writes? Can you send me the logs as well(info is in my signature) along with an estimate of when the problem occurred(doesn't have to be precise but that way we at least know what day to look at)?

Thanks, I think I replicated the problem, started the copy yesterday around noon ET, locked up when I came in this morning.
I sent the logs.

PS - this could very well be a MS issue, it happens with their iSCSI initiator, not using a hardware one.

Going through the logs there are two things that we noted.

1. Your volume is completely full. That shouldn't cause a problem, but it's worth noting. How large is this iSCSI LUN anyway?
2. There are a lot of network packet errors, as well as the ReadyNAS complaining about receiving packets out of order. What switch are you using and has anything been changed from default on that switch if it's managed?

You can open Frontview and browse to the network settings page for ethernet 1(eth0), and hit the show errors button and you'll see the error counts that we saw in the logs. If you were to try and reproduce this again after resetting that error count, can you verify that no other access to the NAS was occurring(no one using CIFS or anything) and see if the error count is still increasing? If so, then it would indicate a network problem(likely a client side or switch configuration issue since it's all receive errors on the NAS side and only happens when you're writing)

Grievous wrote:

Going through the logs there are two things that we noted. 1. Your volume is completely full. That shouldn't cause a problem, but it's worth noting. How large is this iSCSI LUN anyway? 2. There are a lot of network packet errors, as well as the ReadyNAS complaining about receiving packets out of order. What switch are you using and has anything been changed from default on that switch if it's managed? You can open Frontview and browse to the network settings page for ethernet 1(eth0), and hit the show errors button and you'll see the error counts that we saw in the logs. If you were to try and reproduce this again after resetting that error count, can you verify that no other access to the NAS was occurring(no one using CIFS or anything) and see if the error count is still increasing? If so, then it would indicate a network problem(likely a client side or switch configuration issue since it's all receive errors on the NAS side and only happens when you're writing)

The volume is the entire RAID5 array, 9TB.

Nothing has been changed. The setup is it's on a port based VLAN on an HP ProCurve 1800-24G. There is an uplink to a L3 switch but both the NAS and the client are on the same switch, so the traffic should never leave that local VLAN.

The client is an HP DL380 G7.

Could this be a Jumbo Frames issue? I could never get Jumbo Frames working with my NIC.

Initially I tried using teamed NICs on both the client and the NAS, but it was also having a lot of issues so I just went down to one nic on each.

PS - how do I reset the error count?

You mean the iSCSI LUN is 9TB? We reserve 10GB, so you can't occupy the entire volume with an iSCSI LUN, so it should only end up 99.9999%(you get the idea) full. The only way to actually fill it completely would be to have 10GB of data copied to a share on the ReadyNAS via CIFS, FTP, or something else.

Does that HP procurve have any management settings configured for the port like flow control? Is the flow control setting on the port matching the iSCSI client(if you have flow control disabled on that port, then it should also be disabled on the client, and the port that the client uses).

Shouldn't be a jumbo frame issue unless you have jumbo frames enabled on something else, since they were disabled on the ReadyNAS when you sent us the logs. Are jumbo frames still enabled on the switch or client?

Rebooting the system will reset that error count. If you do see the error count increasing while you're writing to the readynas, you don't have to continue until it fails.

Thanks, the error log is helpful.

I'm trying something different, I've disabled all the windows TCP/IP services for that NIC and am trying using just the HP Initiator, see if I have better luck with that than with the MS one. I think the fact that they are all RX errors seems to point the finger away from the NAS and towards a networking or host issue.

Hello,

I'm seeing the same issue. When using Veeam and writing to my ReadyNAS 3200, the MS iSCSI initiator essentially locks up some times. The lock up is intermittent. It have tired different types of Server 2008 R2 virtual servers (different patch levels, etc.) and still have the same intermittent issues. When the issue occurs, in Resource Monitor, I can see the logical volume have 100% active time, 0 disk queue length and no changes.

When Veeam is working, it writes at a maximum of about 29 MBps.

I turned the NAS on yesterday and have 718554682 RX packets and 1202679 RX dropped packets. I have no dropped TX packets.

I do not have Jumbo Frames turned on on the switches or NAS. We have redundant HP switches, ESXi 4 servers and an iSCSI SAN. We have not had any issues with iSCSI for 2 years since the SAN's installation.

We are on RAIDiator 4.2.19.

Can you give me suggestions of some more options to try?

Thanks,
Todd

tbey wrote:

Hello, I'm seeing the same issue. When using Veeam and writing to my ReadyNAS 3200, the MS iSCSI initiator essentially locks up some times. The lock up is intermittent. It have tired different types of Server 2008 R2 virtual servers (different patch levels, etc.) and still have the same intermittent issues. When the issue occurs, in Resource Monitor, I can see the logical volume have 100% active time, 0 disk queue length and no changes. When Veeam is working, it writes at a maximum of about 29 MBps. I turned the NAS on yesterday and have 718554682 RX packets and 1202679 RX dropped packets. I have no dropped TX packets. I do not have Jumbo Frames turned on on the switches or NAS. We have redundant HP switches, ESXi 4 servers and an iSCSI SAN. We have not had any issues with iSCSI for 2 years since the SAN's installation. We are on RAIDiator 4.2.19. Can you give me suggestions of some more options to try? Thanks, Todd

Ensure you have flow control enabled on the switch and server NICS. This should mitigate the issue you are having.