MS510TXUP compatibility issue with Mikrotik RB5009UG+S+IN via SFP?

[ccl13]

I think that my MS510TXUP is not compatible with MikroTik RB5009UG+S+IN when connected via SFP port.

The issue is this:

When I connect my RB5009UG+S+IN via any SFP connection cable I have to MS510TXUP, on RB5009UG side it looks like the connection is up but speed negotiation failed, but on MS510TXUP side the connection is only up for about a second then it is considered disconnected.

For detail, this is the log on MS510TXUP when I try to connect RB5009UG to it:

<181>1 2021-12-04T17:53:11.124-8:00Z 192.168.14.209-1 TRAPMGR-5-STP_NEW_ROOT ksi_snmp.c(484) %% This bridge has changed to be the root bridge.
<182>1 2021-12-04T17:53:11.814-8:00Z 192.168.14.209-1 LLDP-6-NEIGHBOR_DEL proto_lldp.c(4985) %% Neighbor deleted on port XGigabitEthernet10: Chassis ID 06:66:B1:C4:3C:49, Port ID brg1-lan/sfp-sfpplus1
<182>1 2021-12-04T17:53:10.654-8:00Z 192.168.14.209-1 STP-6-PORT_STATE proto_stp.c(835) %% Port XGigabitEthernet10 moving from Blocking to Disabled
<181>1 2021-12-04T17:53:10.194-8:00Z 192.168.14.209-1 TRAPMGR-5-PORT_LINK_DOWN ksi_snmp.c(230) %% Interface XGigabitEthernet10 link down
<181>1 2021-12-04T17:53:10.144-8:00Z 192.168.14.209-1 TRAPMGR-5-PORT_LINK_DOWN ksi_snmp.c(230) %% Interface XGigabitEthernet10 link down
<181>1 2021-12-04T17:53:10.144-8:00Z 192.168.14.209-1 TRAPMGR-5-PORT_LINK_UP ksi_snmp.c(232) %% Interface XGigabitEthernet10 link up
<182>1 2021-12-04T17:53:10.144-8:00Z 192.168.14.209-1 STP-6-PORT_STATE proto_stp.c(835) %% Port XGigabitEthernet10 moving from Forwarding to Blocking
<182>1 2021-12-04T17:53:09.614-8:00Z 192.168.14.209-1 LLDP-6-NEIGHBOR_DISCOVER proto_lldp.c(5188) %% New neighbor on port XGigabitEthernet10: Chassis ID 06:66:B1:C4:3C:49, Port ID brg1-lan/sfp-sfpplus1
<182>1 2021-12-04T17:53:09.074-8:00Z 192.168.14.209-1 STP-6-PORT_STATE proto_stp.c(835) %% Port XGigabitEthernet10 moving from Blocking to Forwarding
<182>1 2021-12-04T17:53:09.824-8:00Z 192.168.14.209-1 STP-6-PORT_STATE proto_stp.c(835) %% Port XGigabitEthernet10 moving from Disabled to Blocking
<181>1 2021-12-04T17:53:09.784-8:00Z 192.168.14.209-1 TRAPMGR-5-PORT_LINK_UP ksi_snmp.c(232) %% Interface XGigabitEthernet10 link up

As far as I can tell, there's no obvious reason mentioned as why it is disconnected, and apparantly it went up twice and shutdown twice while physically it is inserted only once and still is firmly sitting in the socket.

Following mitigation are tried but no success:

1) Update firmware to newest on both sides. 1.0.4.2 on MS510TXUP, 7.0.5/7.1rc5/rc6 on RB5009, all same issue.

2) Replaced cables. At least 5 different brands of confirmed working DAC are tried, all same issue; 1 confirmed working pair of 10GBASE-SR and 1 confirmed working pair of 10GBASE-LRM adapters are tried, both same issue.

3) Factory reset of both sides. Same issue with default configurations. (Of course with default configuration the log is simpler but I wasn't being able to find a copy.)

4) Canned air blasting to clean the ports on both sides, same issue.

Also why I think this is compatibility issue between RB5009UG and MS510TXUP:

1) The same configuration works fine when RB5009UG is connected to other switches, tested are Netgear M4200 and Netgear GC752X.

2) The same configuration works fine when MS510TXUP is connected to another router, Mikrotik RB4011iGS+RM, with same firmware and almost same configuration as RB5009UG.

(BTW, MS510TXUP is not in available options in "Model" field when editing this post)

[schumaku]

Reads to me like STP is blocking the port virtually as the connection does come up.

<182>1 2021-12-04T17:53:10.144-8:00Z 192.168.14.209-1 STP-6-PORT_STATE proto_stp.c(835) %% Port XGigabitEthernet10 moving from Forwarding to Blocking

....some kind of loop in the network, the Spannng Tree Protocol - on either one or both sides - is blocking/shutting the port in the test set-up I'd suspect.

[ccl13]

I can confirm this is not the case. Three reasons:

1) I do have RSTP enabled for the whole network.

2) When tried with factory reset environment, there's nothing else connected except this only one piece of link between the router and switch.

3) The shutdown of port is not reflected within STP status page of MS510TXUP. If I create a loop, it would show such in the status page on either side. This means the port gets disconnected somehow, then STP jumped in to mark the port disabled, just like I pulled the plug on the port.

[ccl13]

Also I think the log with same timestamp can be out of order. And apparantly, at the exact timestamp of "2021-12-04T17:53:10.144-8:00Z" there's also a link up and link down event. All three cannot be happening at the same time if that's STP causing connection status change.

[schumaku]

OK, makes sense.

The issue reminds me to some issue we've (nah, not Netgear) experienced on an SFP+ based network link on two security appliance/router brands, and I think the customer tried some Mikrotik with an SFP+ garage and a similar set of test switches as yours. One of the discoveries was that the router side flapped the links, and did not allow using short/passive DAC. 

However, you have tried industry standard fiber modules and experienced the same. 

I would suggest to file a support case with Netgear, convinced their switch engineering would be keen to look into this.

@YeZ can you assist to establish a link to the switch engineering?

[ccl13]

Thank you schumaku.

Yes I've even tried Gigabyte SFP adapters and they don't work too. And only between RB5009 and MS510TXUP.

If there's anything else I can help provide, please do let me know.

I've just managed to create the factory reset environment again and did a minimal system test, and this is the logs I got:

<181>1 2021-01-01T00:04:16.040Z 192.168.0.239-1 TRAPMGR-5-PORT_LINK_DOWN rsd_l3_intf.c(120) %% Interface VLAN-MGMT link down
<181>1 2021-01-01T00:04:16.030Z 192.168.0.239-1 TRAPMGR-5-PORT_LINK_DOWN ksi_snmp.c(230) %% Interface XGigabitEthernet10 link down
<181>1 2021-01-01T00:04:16.960Z 192.168.0.239-1 TRAPMGR-5-PORT_LINK_DOWN ksi_snmp.c(230) %% Interface XGigabitEthernet10 link down
<181>1 2021-01-01T00:04:16.960Z 192.168.0.239-1 TRAPMGR-5-PORT_LINK_UP ksi_snmp.c(232) %% Interface XGigabitEthernet10 link up
<182>1 2021-01-01T00:04:16.960Z 192.168.0.239-1 LLDP-6-NEIGHBOR_DEL proto_lldp.c(4985) %% Neighbor deleted on port XGigabitEthernet10: Chassis ID 2C:C8:1B:FF:AE:42, Port ID bridge/sfp-sfpplus1
<181>1 2021-01-01T00:04:16.950Z 192.168.0.239-1 TRAPMGR-5-PORT_LINK_UP rsd_l3_intf.c(132) %% Interface VLAN-MGMT link up
<181>1 2021-01-01T00:04:16.750Z 192.168.0.239-1 TRAPMGR-5-PORT_LINK_DOWN rsd_l3_intf.c(120) %% Interface VLAN-MGMT link down
<182>1 2021-01-01T00:04:15.980Z 192.168.0.239-1 LLDP-6-NEIGHBOR_DISCOVER proto_lldp.c(5188) %% New neighbor on port XGigabitEthernet10: Chassis ID 2C:C8:1B:FF:AE:42, Port ID bridge/sfp-sfpplus1
<181>1 2021-01-01T00:04:15.610Z 0.0.0.0-1 TRAPMGR-5-PORT_LINK_UP ksi_snmp.c(232) %% Interface XGigabitEthernet10 link up
<181>1 2021-01-01T00:04:14.600Z 0.0.0.0-1 TRAPMGR-5-PORT_LINK_UP rsd_l3_intf.c(132) %% Interface VLAN-MGMT link up
<181>1 2021-01-01T00:04:14.600Z 0.0.0.0-1 TRAPMGR-5-PORT_LINK_DOWN rsd_l3_intf.c(120) %% Interface VLAN-MGMT link down
<181>1 2021-01-01T00:04:14.600Z 0.0.0.0-1 TRAPMGR-5-PORT_LINK_UP rsd_l3_intf.c(132) %% Interface VLAN-MGMT link up
<181>1 2021-01-01T00:04:14.590Z 0.0.0.0-1 TRAPMGR-5-PORT_LINK_DOWN ksi_snmp.c(230) %% Interface XGigabitEthernet10 link down
<181>1 2021-01-01T00:04:14.590Z 0.0.0.0-1 TRAPMGR-5-PORT_LINK_UP ksi_snmp.c(232) %% Interface XGigabitEthernet10 link up

This is from two times I tried to cycle the port on RB5009. It might be hard to see, that the first time (at 2021-01-01T00:04:14.590Z) the switch did not even consider the link is up and directly considered the link to be down. And the second time (at 2021-01-01T00:04:15.610Z) the link was up for a very short moment and again, two link down happened.

Above log is captured with Debug level and not skipping anything, just so it might help with debug. I have the full log available if needed.

[ccl13]

Also please instruct on how to open a support case - the MS510TXUP is past the 3 month support period and on the support site it directs me to post here for help.

[schumaku]

I'm not Netgear - and there is no formal bug report path (AFAIK).

Let's wait for @YeZ or a Netgear moderator to chime in, they will require a full diag log (on any kind of cloud service, and share the link by PM - not in the public).

[ccl13]

Thanks again! That's super helpful already!

[ccl13]

Try bump the thread. @YeZ could you help take a look here?

[YeZ]

Will ask the team to respond on this. Thank you.

[WilsonS]

Hi @ccl13 

I'm looking into it to see how to find out a root cause. 

[ccl13]

Please do let me know what I can help. At this moment, my newest discovery is this:

After MikroTik RB4011 upgraded to 7.1.1, the DAC cable connection is also broken. However it is slightly different - the connection is seen as up by both sides, but both sides are only transmitting packets while 100% of receiving packets are with error.

I am sure this is a compatibility issue more on software level, however this only happens with MS510TXUP. I've tried the same on my other switches, M4200 and GC752X, both working totally fine.

[ccl13]

I've got an update. I can confirm that the issue mentioned in my yesterday's post is fixed after a few reboot on both side - I don't know which reboot fixed it, but I am sure that at least first reboot on both side did not fix the issue. And after reboot, I've tried to reboot both side a few times however I was not able to recreate the issue.

Generally speaking, I think my conclusion is there's some compatibility issue between MikroTik RouterOS 7 and MS510TXUP.

[WilsonS]

Hi @ccl13 

It looks the change comes from upgrading MikroTik RB4011 to 7.1.1.

Becuase you say "I was not able to recreate the issue", I assume the issue can be closed. 

Many thanks for your update.

[ccl13]

Hi @WilsonS ,

No I mean the new observation is not reproducing, the old issue still is there. RB5009 still does not work with MS510TXUP. I just mentioned the issue with RB4011 because this makes me think the issue might be on the RouterOS.

[WilsonS]

Hi @ccl13 

After MikroTik RB4011 is upgraded to 7.1.1, it works with MS510TXUP via SFP. 

However, after updating the latest 7.0.5/7.1rc5/rc6 on RB5009, RB5009 still has SFP issue with MS510TXUP. 

Based on current information, from SFP point of view, it looks RB5009 has some differences from RB4011 . Could you report the situation to MikroTik and ask MikroTik to make RB5009 SFP same as RB4011? Thanks

Meanwhile, I will also study it. 

[ccl13]

Hi @WilsonS,

Sorry if I caused some confusion. Just to make it clear, here's the current observations:

1) RB5009 starting from version 7.0.5 until 7.1.1 and probably later version, all having the same issue mentioned in my first post. That MS510TXUP does not recognize the SFP connection while RouterBoard side recognize it, and of course it does not work. This does not happen with other switched by Netgear.

2) RB4011 after upgraded to 7.1.1 showed a similar issue with MS510TXUP that although both side recognizes the connection, it does not work as both sides are only receving broken packets. It did not fix after one reboot on both sides but following reboot fixed it. And afterwards I haven't yet been able to reproduce it, but similar issue did not occur before RouterOS 7. So I suspect this is partially related to the MikroTik software, which might also be the case for RB5009 issue.

I have the issue reported to MikroTik as a support ticket and they are working on it too. I am reporting the issue here because this currently is a specific MS510TXUP issue, so I think Netgear might be able to provide help too.

[WilsonS]

Hi @ccl13 

Please kindly test MS510TXUP and RB5009 (latest FW version) at simple environment and settings again. It will be easier for troubleshooting and debugging. Thanks. 

1. Remove other devices from MS510TXUP and RB5009. Keep only 1 SFP connection between them and connect only 1 PC to MS510TXUP.
2. Reset two devices to default settings
3. Configure the followings to make MS510TXUP sending out as less as possible
   * System > Management > Switch Discovery > UPnP / SSDP => Set "Allow discovery by the Insight app" as Disable.
   * System > Management > DNS > DNS Configuration => Set "DNS Status" as Disable.
   * System > LLDP > Advanced > LLDP Port Settings => Set "Admin Status" of all ports to Disabled.
   * Switching > L2 Loopback Protection > L2 Loopback Protection Configuration => Set "Admin Mode" as Disable.

If the issue still happens, please help to go to Maintenance > Export > HTTP File Export to get Tech-Support file when the issue happens. Provide the Tech-Support file. Thanks a lot.

[ccl13]

Hi @WilsonS ,

Because the switch is in active use it took me a while to find a chunk of time to do this experiment. Please let me know how I can share the file because forum does not allow me to upload it without an image file extension.

I did this experiment with both side factory reset and on newest stable firmware (1.0.4.2 on MS510TXUP, 7.1.1 on MikroTik). All configurations requested is applied before experiment.

Because I need to access the switch, I connected with known good router first so I can make sure it is working and did configuration. And after experiment I connected it with the same router so I can login and pull the files.

During the experiment, I tried to do the DAC cable insert 3 times, first time on MS510TXUP it looks like the port was active for one or two seconds, but the following two attempts just does not do anything. But of course, on MikroTik side it looks like the connection is good for all the 3 attempts.

Also worth noting, that this time I am already using a replacement unit of RB5009UG router. So it is now precise to say that the issue is between my MS510TXUP unit and MikroTik RB5009UG model.

[ccl13]

Hi @WilsonS , private messaged you a link to download the tech support file.

[ccl13]

Also I tried the new firmware 1.0.4.6 released today, the issue is still there.

[clazzi]

I also had the same problem

5009+ms510txm

[WilsonS]

Hi @ccl13 

Sorry to delay the response becuase of too busy.

Can you configure Mikrotik RB5009UG SFP+ from auto-nego to forced 10G speed and duplex and test again? Thanks a lot

[ccl13]

@WilsonS that's already tried and it does not work.

Also at this time I am no longer with a RB5009. Due to I cannot use it I had to return it at my cost. Please still look into this issue though.