We just bought three new Dell R610 servers with Broadcom iSCSI bootable NICs to replace a pair of much older HP DL380 servers. I have a pair of Netgear smart switches (GS748TR) and a pair of ReadyNAS 2100 storage devices, both are running 4.2.17.
On the old servers I am booting from local installs of ESX 3.5 u4, and everything is working perfectly, with VMDKs stored on NFS shares on the ReadyNAS devices. I'm giving the background to let you know that this setup works just great. With the new servers, the design was to boot from iSCSI LUNs on the ReadyNAS storage, so we didn't buy any local storage for the R610s.
Right now I'm not using any IQN ACLs, nor am I using any CHAP on the iSCSI connections, as I'm just trying to get it functional.
The Broadcom NIC is configured per Dells online guidance for setting it up, verified by Dell tech support. Jumbo frames are disabled on the SAN (although I get the same result mentioned below with or without Jumbo frames enabled on the ReadyNAS)
When booting, the R610 server shows that it successfully connects to the iSCSI target LUN, then goes into the ESX install sequence from the CD, which is second in the boot order. The NIC BIOS is configured to NOT boot from the iSCSI target until after I get ESX installed on the target, but the NIC is first in the boot order. Then when the ESX install gets to the point where it's ready to configure the disk (right after accepting the license agreement), it says that it cannot find any disk to install on. I get this with both ESX 3.5 u4 and ESXi 4.1 u1. When I download the logs from the ReadyNAS, and look in the system.log it shows a successful iSCSI connection.
Anyone know why ESX isn't seeing the iSCSI target? It's stumping me. Any help would be greatly appreciated. Thanks.
Now I've attempted an install with ESX 4.1 U1 (not i) and tried with installing additional Broadcom drivers, and without (but the preloaded Broadcom drivers appear to work just fine). I even tried removing the iSCSI target and recreating a larger one (and then changing the target name in the NIC BIOS), and I get the same result every time: The SAN system.log shows the connection from the NIC, the ESX install shows the NICs listed, and even sees that one of the NICs is connected, I configure the correct IP range for the NIC driver in the ESX install (both separate from, or the same as, the NIC IP), but when you go to select the LUN to install ESX to, it is blank. There are no drives listed...nothing.
I originally had thought it might be a NIC driver issue, but after seeing the ESX 4.1 u1 install steps, I know that the NIC drivers are just fine, and are being fully recognized by ESX.
The problem at this point is clearly something with how the ESX install talks to the ReadyNAS iSCSI target LUN, or how it's presented to the ESX installation. Because the Broadcom NIC BIOS appears to talk with the ReadyNAS iSCSI target LUN just fine.
The last five lines in the system.log, listed below, are the perplexing part. It appears that the connection to the LUN is disconnected as soon as the ESX loader "takes over". Am I doing something wrong, or is something not working correctly?
I'm hoping that a Netgear expert can look at the last five lines in the log and say, "Oh yeah, you need to do X." Any help would be appreciated.
This issue was resolved with a Broadcom whitepaper titled: "ESXi 4 1 iSCSI Boot Setup Guide v0 11 DRAFT"
It was an issue with the fact that you cannot boot from NIC Port 1, you have to have NIC Port 1 enabled and connected to the SAN with iSCSI turned off, then enable and connect NIC Port 2 to the SAN as well, and perform the iSCSI boot from NIC Port 2. Sort of a weird super secret handshake, but it works now.
This is exactly why I normally boot all my ESX boxes from a flash drive or SD card. That way after booting they all mount their iSCSI targets and everything is good. It also allows the servers to boot when their is an iSCSI outage and reattach the targets once they become available.
