install Mellanox mft/Update FW/Change OEM PSID/infiniBand 啟用/SR_IOV

下載:

MFT Download

MLNX_OFED

編譯完成檔案:For 4.15.18-7

Linux x86 Debian/Ubuntu:mft-4.8

安裝:

apt-get install gcc make dkms pve-headers-$(uname -r)
wget http://www.mellanox.com/downloads/MFT/mft-4.8.0-26-x86_64-deb.tgz
tar zxvf mft-4.8.0-26-x86_64-deb.tgz
cd mft-4.8.0-26-x86_64-deb/
./install.sh
mst start
mst status

狀態:

# mst status
MST modules:
------------
    MST PCI module loaded
    MST PCI configuration module loaded

MST devices:
------------
/dev/mst/mt4099_pciconf0         - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:04:00.0 addr.reg=88 data.reg=92
                                   Chip revision is: 01
/dev/mst/mt4099_pci_cr0          - PCI direct access.
                                   domain:bus:dev.fn=0000:04:00.0 bar=0xf0c00000 size=0x100000
                                   Chip revision is: 01
---------------------------------------------
ConncectX4:
mst status
MST modules:
------------
    MST PCI module is not loaded
    MST PCI configuration module loaded

MST devices:
------------
/dev/mst/mt4117_pciconf0         - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:04:00.0 addr.reg=88 data.reg=92
                                   Chip revision is: 00

使用 flint 查看詳細資料:

flint -d /dev/mst/mt4099_pci_cr0 query
ConnectX4:
flint -d /dev/mst/mt4117_pciconf0 query

詳細資料:

# flint -d /dev/mst/mt4099_pci_cr0 query
Image type:            FS2
FW Version:            2.42.5000
FW Release Date:       5.9.2017
Product Version:       02.42.50.00
Rom Info:              type=PXE version=3.4.752 devid=4099
Device ID:             4099
Description:           Node             Port1            Port2            Sys image
GUIDs:                 00_______e34 00________e35 00_________36 0________0e37
MACs:                                   001_____35     00______36
VSD:
PSID:                  ORC1090120019
---------------------------------------
ConnextX4:
flint -d /dev/mst/mt4117_pciconf0 query
Image type:            FS3
FW Version:            14.16.1020
FW Release Date:       20.6.2016
Product Version:       rel-14_16_1020
Rom Info:              type=PXE version=3.4.812
Description:           UID                GuidsNumber
Base GUID:             248a0703008d63ba        4
Base MAC:              248a078d63ba            4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  MT_2420110034
Security Attributes:   N/A

使用 mlxfwmanager 檢查FW:

mlxfwmanager -d /dev/mst/mt4099_pciconf0

檢查範例:

Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX3
  Part Number:      7046442_Ax
  Description:      Oracle Dual port; ConnectX-3 ; Infiniband Adapter FDR/QDR
  PSID:             ORC1090120019
  PCI Device Name:  /dev/mst/mt4099_pciconf0
  Port1 GUID:       00___________375
  Port2 GUID:       000__________376
  Versions:     Current    Available
     FW         2.11.1280     N/A 
Status: No matching image found

先去下載對應網卡FW binary:

Updating Firmware for ConnectX®-3 VPI PCI Express Adapter Cards (InfiniBand, Ethernet, FCoE, VPI)

cd
wget http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip
apt install unzip
unzip fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip
rm *.zip

強制刷版:

flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn
ConnectX4:
flint -d /dev/mst/mt4117_pciconf0 -i fw-ConnectX4Lx-rel-14_23_1020-MCX4121A-ACA_Ax-UEFI-14.16.17-FlexBoot-3.5.504.bin burn

確認版號更新:

Current FW version on flash:  2.11.1280
    New FW version:           2.42.5000


    You are about to replace current PSID on flash - "ORC1090120019"
with a different PSID - "MT_1090120019".
    Note: It is highly recommended not to change the PSID.

 Do you want to continue ? (y/n) [n] : y
Burning FS2 FW image without signatures - OK
Restoring signature                     - OK

重新啟動:

init 6

檢查新版本是否正常:

mst start
flint -d /dev/mst/mt4099_pci_cr0 query

狀態:

Image type:            FS2
FW Version:            2.42.5000
FW Release Date:       5.9.2017
Product Version:       02.42.50.00
Rom Info:              type=PXE version=3.4.752 devid=4099
Device ID:             4099
Description:           Node             Port1            Port2            Sys image
GUIDs:                 001____374 001____375 0010_____376 0010e_____377
MACs:                                 00____375     001____76
VSD:
PSID:                  MT_1090120019 <-已經更換

設定接口模式:

  • 1-InfiniBand
  • 2-Enternet
  • 3-VPI(Auto)

修改接口模式:

mlxconfig -d /dev/mst/mt4099_pciconf0 set LINK_TYPE_P1=1 LINK_TYPE_P2=1

修改結果:

Device #1:
----------

Device type:    ConnectX3
PCI device:     /dev/mst/mt4099_pciconf0

Configurations:                              Next Boot       New
         LINK_TYPE_P1                        VPI(3)          IB(1)
         LINK_TYPE_P2                        VPI(3)          IB(1)

 Apply new Configuration? ? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.

開啟SR-IOV:

mlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1
mlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=8
CX3-Pro Max VF:3
mlxconfig -d /dev/mst/mt4103_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=3

修改結果:

Device #1:
----------

Device type:    ConnectX3
PCI device:     /dev/mst/mt4099_pciconf0

Configurations:                              Next Boot       New
         SRIOV_EN                            False(0)        True(1)

 Apply new Configuration? ? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.

添加VF:

/etc/modprobe.d/mlx4_core.conf
vfs=單口vf,單口vf,雙口共通vf
options mlx4_core num_vfs=4,4 port_type_array=2,2 probe_vf=4,4
options mlx4_core num_vfs=0,0,3 probe_vf=0,0,3 log_num_mgm_entry_size=-1

mlx5設置方式:
echo 2 > /sys/class/net/eno3/device/sriov_numvfs
ip link set eno3 vf 0 mac 50:6b:4b:bd:37:f4
ip link set eno3 vf 1 mac 50:6b:4b:bd:37:f5

重起生效:

service openibd restart
init 6

添加模塊

echo mlx4_ib >> /etc/modules
echo ib_umad >> /etc/modules
echo ib_cm >> /etc/modules

啟用模塊:

modprobe mlx4_ib; modprobe ib_umad; modprobe ib_cm;

延伸參考: Flash OEM (IBM) Mellanox

OEM PSID對應原廠PSID:PDF

HowTo Configure SR-IOV VFs on Different ConnectX-3 Ports

HowTo Configure SR-IOV for ConnectX-3 with KVM (InfiniBand)

HowTo Configure SR-IOV for ConnectX-4/ConnectX-5 with KVM (Ethernet)

Getting started with VXLAN over ConnectX-3 Pro adapters on CentOS 7.1

ConnectX_Family_Adapter_Cards_Firmware_Mapping_for_Lenovo
阅读剩余
THE END