install Mellanox mft/Update FW/Change OEM PSID/infiniBand 啟用/SR_IOV
下載:
編譯完成檔案:For 4.15.18-7
Linux x86 Debian/Ubuntu:mft-4.8
安裝:
apt-get install gcc make dkms pve-headers-$(uname -r)
wget http://www.mellanox.com/downloads/MFT/mft-4.8.0-26-x86_64-deb.tgz
tar zxvf mft-4.8.0-26-x86_64-deb.tgz
cd mft-4.8.0-26-x86_64-deb/
./install.sh
mst start
mst status狀態:
# mst status
MST modules:
------------
MST PCI module loaded
MST PCI configuration module loaded
MST devices:
------------
/dev/mst/mt4099_pciconf0 - PCI configuration cycles access.
domain:bus:dev.fn=0000:04:00.0 addr.reg=88 data.reg=92
Chip revision is: 01
/dev/mst/mt4099_pci_cr0 - PCI direct access.
domain:bus:dev.fn=0000:04:00.0 bar=0xf0c00000 size=0x100000
Chip revision is: 01
---------------------------------------------
ConncectX4:
mst status
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module loaded
MST devices:
------------
/dev/mst/mt4117_pciconf0 - PCI configuration cycles access.
domain:bus:dev.fn=0000:04:00.0 addr.reg=88 data.reg=92
Chip revision is: 00
使用 flint 查看詳細資料:
flint -d /dev/mst/mt4099_pci_cr0 query
ConnectX4:
flint -d /dev/mst/mt4117_pciconf0 query詳細資料:
# flint -d /dev/mst/mt4099_pci_cr0 query
Image type: FS2
FW Version: 2.42.5000
FW Release Date: 5.9.2017
Product Version: 02.42.50.00
Rom Info: type=PXE version=3.4.752 devid=4099
Device ID: 4099
Description: Node Port1 Port2 Sys image
GUIDs: 00_______e34 00________e35 00_________36 0________0e37
MACs: 001_____35 00______36
VSD:
PSID: ORC1090120019
---------------------------------------
ConnextX4:
flint -d /dev/mst/mt4117_pciconf0 query
Image type: FS3
FW Version: 14.16.1020
FW Release Date: 20.6.2016
Product Version: rel-14_16_1020
Rom Info: type=PXE version=3.4.812
Description: UID GuidsNumber
Base GUID: 248a0703008d63ba 4
Base MAC: 248a078d63ba 4
Image VSD: N/A
Device VSD: N/A
PSID: MT_2420110034
Security Attributes: N/A使用 mlxfwmanager 檢查FW:
mlxfwmanager -d /dev/mst/mt4099_pciconf0檢查範例:
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX3
Part Number: 7046442_Ax
Description: Oracle Dual port; ConnectX-3 ; Infiniband Adapter FDR/QDR
PSID: ORC1090120019
PCI Device Name: /dev/mst/mt4099_pciconf0
Port1 GUID: 00___________375
Port2 GUID: 000__________376
Versions: Current Available
FW 2.11.1280 N/A
Status: No matching image found先去下載對應網卡FW binary:
Updating Firmware for ConnectX®-3 VPI PCI Express Adapter Cards (InfiniBand, Ethernet, FCoE, VPI)
cd
wget http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip
apt install unzip
unzip fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip
rm *.zip強制刷版:
flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn
ConnectX4:
flint -d /dev/mst/mt4117_pciconf0 -i fw-ConnectX4Lx-rel-14_23_1020-MCX4121A-ACA_Ax-UEFI-14.16.17-FlexBoot-3.5.504.bin burn確認版號更新:
Current FW version on flash: 2.11.1280
New FW version: 2.42.5000
You are about to replace current PSID on flash - "ORC1090120019"
with a different PSID - "MT_1090120019".
Note: It is highly recommended not to change the PSID.
Do you want to continue ? (y/n) [n] : y
Burning FS2 FW image without signatures - OK
Restoring signature - OK重新啟動:
init 6檢查新版本是否正常:
mst start
flint -d /dev/mst/mt4099_pci_cr0 query狀態:
Image type: FS2
FW Version: 2.42.5000
FW Release Date: 5.9.2017
Product Version: 02.42.50.00
Rom Info: type=PXE version=3.4.752 devid=4099
Device ID: 4099
Description: Node Port1 Port2 Sys image
GUIDs: 001＿＿＿＿374 001＿＿＿＿375 0010＿＿＿＿＿376 0010e＿＿＿＿＿377
MACs: 00＿＿＿＿375 001＿＿＿＿76
VSD:
PSID: MT_1090120019 <-已經更換設定接口模式:
- 1-InfiniBand
- 2-Enternet
- 3-VPI(Auto)
修改接口模式:
mlxconfig -d /dev/mst/mt4099_pciconf0 set LINK_TYPE_P1=1 LINK_TYPE_P2=1修改結果:
Device #1:
----------
Device type: ConnectX3
PCI device: /dev/mst/mt4099_pciconf0
Configurations: Next Boot New
LINK_TYPE_P1 VPI(3) IB(1)
LINK_TYPE_P2 VPI(3) IB(1)
Apply new Configuration? ? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.開啟SR-IOV:
mlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1
mlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=8
CX3-Pro Max VF:3
mlxconfig -d /dev/mst/mt4103_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=3修改結果:
Device #1:
----------
Device type: ConnectX3
PCI device: /dev/mst/mt4099_pciconf0
Configurations: Next Boot New
SRIOV_EN False(0) True(1)
Apply new Configuration? ? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.添加VF:
/etc/modprobe.d/mlx4_core.conf
vfs=單口vf,單口vf,雙口共通vf
options mlx4_core num_vfs=4,4 port_type_array=2,2 probe_vf=4,4
options mlx4_core num_vfs=0,0,3 probe_vf=0,0,3 log_num_mgm_entry_size=-1
mlx5設置方式:
echo 2 > /sys/class/net/eno3/device/sriov_numvfs
ip link set eno3 vf 0 mac 50:6b:4b:bd:37:f4
ip link set eno3 vf 1 mac 50:6b:4b:bd:37:f5重起生效:
service openibd restart
init 6添加模塊
echo mlx4_ib >> /etc/modules
echo ib_umad >> /etc/modules
echo ib_cm >> /etc/modules啟用模塊:
modprobe mlx4_ib; modprobe ib_umad; modprobe ib_cm;延伸參考: Flash OEM (IBM) Mellanox
OEM PSID對應原廠PSID:PDF
HowTo Configure SR-IOV VFs on Different ConnectX-3 Ports
HowTo Configure SR-IOV for ConnectX-3 with KVM (InfiniBand)
HowTo Configure SR-IOV for ConnectX-4/ConnectX-5 with KVM (Ethernet)
Getting started with VXLAN over ConnectX-3 Pro adapters on CentOS 7.1
ConnectX_Family_Adapter_Cards_Firmware_Mapping_for_Lenovo阅读剩余
THE END