Unable to get SFP28 working on a M4350-24x4V

[MisterBumbles]

Hello,

Let me start by saying this is my first attempt to make fiber connections.

I recently acquired a M4350-24x4V and am attempting to utilize the SFP28 ports. Without being able to find any documentation about which SFP28 modules would be compatible for this switch, I originally purchased some https://www.fs.com/products/183299.html. After consulting Netgear support, I was informed the FS SFP28 modules may be problematic and that Flexoptix was the better choice, so I picked up some of these: https://www.flexoptix.net/en/p-8525g-01.html Later I found this post which say both should work: https://community.netgear.com/t5/Pro-AV-over-IP-Switches/M4500-SFP28-Model/m-p/2346445/highlight/tru... However, even after configuring the Flexoptix for 'Netgear' I'm having issues with these as well. All connections herein use OM4 fiber:

- https://www.fs.com/products/40254.html
- https://www.flexoptix.net/en/m4-ll-x.html?option570=258

So far I am attempting to connect the M4350-24x4V to a XS716T and an Intel https://ark.intel.com/content/www/us/en/ark/products/95260/intel-ethernet-network-adapter-xxv710-da2.... For the XS716T I ordered some https://www.fs.com/products/12345.html which look comparable to Netgear AXM761 SFP+ modules.

If I plug in the FS SFP+ modules between the M4350-24x4V and the XS716T, the link lights come on and the port status via 'Switching > Ports > Port Configuration' shows:

Media Type: 10gBase-SR
Speed: 10G
Physical Status: 10G Full
Link Status: Link Up

which looks to confirm the SFP+ modules work. If I connect a FS SFP+ module to the M4350 and a Flexoptix SFP28 configured for Intel in my adapter card, the port lights come on the switch and the configuration shows:

Media Type: 10gBase-SR
Speed: 10G
Physical Status: 10G Full
Link Status: Link Up

which looks to confirm the Intel card works. When plugging in any of the SFP28 modules and linking to nothing, or linked to the SFP+, or linking to the Intel card, the M4350 port configuration always shows:

Media Type: <empty>
Speed: 1000, 10G, 25G
Physical Status: Unknown
Link Status: Link Down

with no port lights. The result is also the same regardless of any other transceivers plugged into the M4350. If I have a working SFP+ connection between the switches and plug in an SFP28 to the M4350, the SFP+ link drops out, showing:

Media Type: <empty>
Speed: 10G
Physical Status: Unknown
Link Status: Link Down

The only way I can get any sign of life is if I plug in two SFP28 modules to the M4350 and link them to each other. This doesn't sound sensical to me but at least the port lights turn on, what I'm hoping indicates the transceivers are configured correctly, the fiber itself is OK, and the modules are talking, showing:

Media Type: 25GBase-SR
Speed: 1000, 10G, 25G
Physical Status: 25G Full
Link Status: Link Up

So what am I missing? One Netgear support representative told me the modules must be of the same type between the switches, i.e., SFP+, but I want a second a opinion. That also doesn't appear to be the case with my experiment confirming the Intel adapter works. And I thought it was the whole point of the SFP28 module supporting 1G/10G/25G so it could negotiate support on the other side. And why does even plugging in an SFP28 module drop my SFP+ link between the switches? Why is the only configuration that shows any type of SFP28 activity is when connected between 2 ports on the same switch?

I'm all question and no answers. Your help is greatly appreciated.

Thanks. Jerry.

[LaurentMa]

Hi @MisterBumbles ,

Thank you for your message, let me help you.

Did you review the M4350 datasheet ever? Maybe the problem is related to the information shown on page 2:

https://www.downloads.netgear.com/files/GDC/M4350/M4350_Datasheet.pdf

On the M4350-24X4V (XSM4328CV) and any other M4350 switch with SFP28 25G uplinks, there is a strong limitation as follows:  * ETHERNET Mode: Each 4x25G block is connected to a 100G SERDES. As such, each 4-port block can only work at the same speed: 4x1G, or 4x10G, or 4x25G. Since 25G takes precedence, when one 25G module is inserted, other ports with 1G or 10G modules get down in the same 4-port block.

This might explain why when you have a 10G connection to another switch, you are experiencing issues with the other 25G connection.

The switch won't so 10G and 25G simultaneously on the four SFP28 ports. I would advise the NETGEAR AXM765-20000S 10GBASE-T transceiver on the other switch, in order to establish a 10G link using one of the 24 10GBASE-T copper ports on the  M4350-24X4V (XSM4328CV). With Copper SFP or SFP+ transceivers, please always go with genuine NETGEAR modules, since third-party (cheap) copper transceivers never come with the correct hardware PHY to work great in our switches.

For fiber modules, we see less problems but we cannot guarantee the quality control at other vendors. I can confirm to you that the 10G fiber or 25G fiber third-party modules should work for the most part, unless they are defect or not following MSA standards. For 25G, we successfully tested all FlexOptix modules for your information.

Last, on 25G, don't forget you might need to configure another FEC mode than our default FEC mode on the SFP28 25G ports. Please upgrade your M4350 switch with our latest 14.0.2.17:

https://www.netgear.com/support/product/m4350#download

After the upgrade and the reboot, please go to the IT GUI, Switching\Ports\Port Configuration, and select the SFP28 port you want to test, then select maybe another FEC setting in the dropdown menu (FEC Mode column):

Please let us know (but you need to do either all 10G or all 25G on these four SFP28 ports!)

[MisterBumbles]

Hi @LaurentMa,

I appreciate the quick reply. I thought the details you mentioned would be in the user manual, which is where I looked, but  could not find. I missed the asterisk and corresponding footnote on page 2 of the datasheet. You pointing me to this information is helpful.

Also, I have little familiarity with switching equipment and in my naive exposure I could not imagine a situation where there would be such a limitation, i.e., in this case all optical ports must operate at the same speed. So I've approached this issue somewhat with blinders on.

My firmware needs updating as I don't see the 'FEC' columns in the port configuration page. But you said, "Don't forget you might need to configure another FEC mode." However, I don't see mention of FEC or Forward Error Correction in the user manual or data sheet for this device. Did I miss this somewhere?

I will update the firmware, try some different FEC setting[s] and report back.

Thanks. Jerry.

[LaurentMa]

Hi Jerry, thank you.

The datasheet is the repository for supported features and any known limitations that can come with a product.

The technical documentation is always trying to keep up with new features (such as FEC) knowing this is general documentation based on the managed switch platform that always evolve based on new series, maintenance releases, etc.

And FEC is a setting for 25G and 100G because optics used in transceivers, NICs, and other network devices can differ hence FEC sometimes need to be configured right for specific devices. We have default Auto-FEC but in our recent firmware you will find this setting. Please upgrade to 14.0.2.17 from here: https://www.netgear.com/support/product/m4350#download

[MisterBumbles]

Hi @LaurentMa. With the updated firmware, connecting my Intel card looks to work without issue. I didn't make any particular change to the FEC mode as one was chosen for me. In my mind this is proof enough that things are looking good.

Cheers. Jerry.

[LaurentMa]

Thank you for the update, @MisterBumbles 

With our latest firmware, we implemented some Auto mode for FEC and I'm happy to hear that the detection with your Intel NIC went smooth, then. Cheers