m4300-24x: strange ARP forwarding effect

[jmozdzen]

Hi all,

seems I cannot get to the bottom of this by myself, although I stripped my config to (what I feel is) minimum:

- M4300-24x 10G switch

- on one port, I have a (newly attached) server, single 10G interface, untagged, port mapped to VLAN 7

- on different ports, there's an existing server with a VLAN 7 interface (switch is running 802.1q on link aggregation group, existing server is handling tagging)

- side note: on the existing server, there's also a DHCP server running on the same VLAN interface (see below why I mention this)

Problem:

- after booting the new server and setting up the IP, pinging the existing server fails at ARP level

- as soon as the exisiting server pings *any address in the subnet*, pinging works bi-directionally

Details:

- I have  a VLAN dedicated to PXE booting, with an existing server offering DHCP and TFTP to that VLAN 7.

- I want to PXE-boot a new server

  * the new server successfully goes through the DHCP stage, the new server receives a proper IP from the existing server's dhcpd

  * the TFTP stage fails with the new server reporting an ARP timeout

- stripped down the environment to have the new server configured with a static IP on its interface

- pinging from the new server to the existing server doesn't work ("unreachable"), until I ping that new server from the existing server

- I've run tcpdumps:

  * I can see the DHCP packets from and to the new server on the corresponding Ethernet interface of the existing server

  * I can see that the new server sends out ARP requests (via switch port mirroring and per tcpdump on ethX of new server), but the existing server never receives these packets
  * once an ARP broadcast for *any* IP from the existing server is forwarded to the new server (I see this in i.e. the tcpdump of the new server's local interface), also the ARP request from the new server for the IP of the existing server reaches the existing server, is answered and the ICMP packets flow.

Looked like some form of dynamic VLAN assignment - but what's strange then is that the initial DHCP (new to existing server) goes through on exactly that VLAN and is answered. It's only the following ARP request that's "blocked".

I have no ARP snooping configured on the switch, and tried setting up the new port via new style (switchport mode access + access vlan 7; switchport mode trunk + trunk native vlan 7) and old-style (vlan pvid 7 + vlan participation include 7), without seeing any difference.

No dynamic VLANs configured whatsoever (MAC- / protocol- / IP-based).

Any ideas on what I may have missed?

Regards

Jens

[jmozdzen]

Update:

I had tried rebooting and updating the switches to the latest software level (12.0.9.3), just in case... but that didn't help.

Running out of ideas, I broke up the redundant LACP connection of the existing server - and to my suprise, everything started to work.

The network setup is as follows:

• existing server connected to two M4300-24X (stacked) via two 10G links, LACP, hash mode 7

  • interface lag 12
    port-channel load-balance 7
    mtu 9216
    switchport mode trunk
    switchport trunk native vlan 190
    switchport trunk allowed vlan 7,100,190-193,300-301,1100-1199

    server is running a "virtual VLAN interface" on VLAN 7, and has dhcpd and TFPT daemon listening on that interface
• new server connected to same two M4300-24X (stacked) via two 10G links (indended: LACP, hash mode 7)
  • because of the problems, the LAG was de-configured, only one link is currently active, without LACP

  • interface 1/0/8
    vlan pvid 7
    vlan participation exclude 1
    vlan participation include 7

    new server is to PXE-boot via this interface
• DHCP from new server to existing works, but when TFTP is tried, the PXE code reports "ARP timeout"
• tracing via a mirror port for 1/0/8 (new server's port), I can see the ARP requests being sent out by the new server
• tracing via tcpdump on the existing server and other machines on VLAN 7, I only see the DHCP packets (both requests and responses), but no indication of the ARP requests from the new server
• if I take down one of the two LAG interfaces of the existing server (I chose the on on module 2, so both existing and new server are active on the same switch), communications work as expected.

As described in my first post, I had also tested via ICMP echo requests by booting an OS on the new server and testing manually... with redundant LAG for the existing server, packets from the new server to the existing server (both ARP requests and ICMP) were only seen once I had pinged in the reverse direction (existing to new server, or to *any* IP on VLAN 7). With only the single link in the existing server's LAG, this test also worked as expected (ARP from new to exisiting server work without any prior traffic from the existing node.

I'll have to dig into this much deeper, but if this does ring a bell for anyone here, I'd really appreciate any idea or pointer.

Best regards,

Jens

[jmozdzen]

Seems I'm not the only one affected by this - I received a message from another Netgear user reporting similar problems.

We're ran into more similar situations (it was reported to me by an admin "host cannot ping, but one I pinged from the other side, even the original ping started to work") and we're facing "obscure" connectivity problems (where connections are reported to run into time-outs by applications, when trying to talk to some service). All this on v12.0.9.3, so "latest firmware".

Anybody else using LACP links across multiple M4300 (stacked) switches, facing similar symptoms?

Regards,

J

[kevin_hong]

Hi @jmozdzen ,

Welcome to community,

Could you please provide me with "tech support" ?  it will be very helpful to check to find the casue of the issue you are facing.

https://kb.netgear.com/31439/How-do-I-send-diagnostic-files-from-my-Managed-Switch-to-NETGEAR-commun... 

Thank you

[jmozdzen]

Hi Kevin,

thank you for your offer to look into this. Since I've just had a round of extra reboots, I'll rather wait for the next occurance of this issue, so that the tech supp will contain a "proven" case of trouble - no use in someone wasting time of going through the logs to only find out nothing bad had happend since last reboot.

I'll update this thread once I can report back.

Thanks & best regards,

J