http://wiki.define-technology.com/mediawiki-1.35.0/index.php?action=history&feed=atom&title=Broadcom_install_for_RoCE_on_ROCm_devices Broadcom install for RoCE on ROCm devices - Revision history 2026-05-05T00:09:52Z Revision history for this page on the wiki MediaWiki 1.35.0 http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=Broadcom_install_for_RoCE_on_ROCm_devices&diff=32857&oldid=prev Antony: add script to build and 2024-04-25T08:23:02Z <p>add script to build and</p> <table class="diff diff-contentalign-left diff-editfont-monospace" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 08:23, 25 April 2024</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l136" >Line 136:</td> <td colspan="2" class="diff-lineno">Line 136:</td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>link bnxt_re2/1 state ACTIVE physical_state LINK_UP netdev enp180s0np0</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>link bnxt_re2/1 state ACTIVE physical_state LINK_UP netdev enp180s0np0</div></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>link bnxt_re3/1 state ACTIVE physical_state LINK_UP netdev enp139s0np0:</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>link bnxt_re3/1 state ACTIVE physical_state LINK_UP netdev enp139s0np0:</div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">&lt;/pre&gt;</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"></ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">this does do some useful things but on ubuntu22 is skips the driver build step. To fix this, use this script to complete the process. put it in the same dir where you extcracted the bcm_229.1.123.0b.tar.gz file (not inside that dir though)</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">in order for this to work you cannot be using the hwe kernels install linux-generic and remove all linux-image-6* kernels and make sure it boots corrrectly into a 5 kerenel. the makefile ONLY builds for the running kernel. IF you have a new kernel installed and set to boot next boot it is normal for it to be borked.</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">&lt;pre&gt;</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">#!/bin/bash</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">mypwd=$PWD</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">vers=229.1.123.0b</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">tarfile=bcm_${vers}.tar.gz</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">extractdir=&quot;bcm_${vers}&quot;</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">echo extracting $tarfile</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">tar zxf &quot;${tarfile}&quot;</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">cd &quot;$extractdir/Linux/Linux_Driver&quot;</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">drivertars=( *.tar.gz )</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">firstdriver=${drivertars[0]}</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">echo extracting $firstdriver</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">tar zxf &quot;$firstdriver&quot;</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">driver_extract=$(echo $firstdriver | sed -r &#039;s|(.+).tar.+|\1|g&#039;)</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">cd &quot;$driver_extract&quot;</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">echo building driver</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">#make clean &amp;&amp; make &amp;&amp; sudo make install</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">rc=$?</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">[[ $rc -gt 0 ]] &amp;&amp; exit $rc</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"></ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">echo installing userspace lib</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">cd &quot;$mypwd/$extractdir/Linux/KMP-RoCE-Lib/&quot;</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">libbnxt_re_tars=( *.tar.gz )</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">firstlibbnxt_re=${libbnxt_re_tars[0]}</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">echo extracting $firstlibbnxt_re</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">tar xf &quot;$firstlibbnxt_re&quot;</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">libbnxt_re1_extract=$(echo $firstlibbnxt_re | sed -r &#039;s|(.+).tar.+|\1|g&#039;)</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">cd &quot;$libbnxt_re1_extract&quot;</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"></ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">sh autogen.sh &amp;&amp; ./configure --sysconfdir=/etc &amp;&amp; make &amp;&amp;  sudo sh -c &quot;find /usr/lib64 -name libbnxt_re-rdmav\*.so; find /usr/lib -name libbnxt_re-rdmav\*.so&quot;| sudo xargs -i{} mv {} {}.bak &amp;&amp;  sudo make install all</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">[[ $rc -gt 0 ]] &amp;&amp; exit $rc</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">sudo sh -c &quot;echo /usr/local/lib &gt;&gt; /etc/ld.so.conf&quot;</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">sudo ldconfig</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">sudo cp bnxt_re.driver /etc/libibverbs.d/</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">sudo rmmod bnxt_re</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">sudo rmmod bnxt_en</ins></div></td></tr> <tr><td colspan="2"> </td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">sudo modprobe bnxt_re &amp;&amp; sudo depmod -a &amp;&amp; sudo update-initramfs -u</ins></div></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>&lt;/pre&gt;</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>&lt;/pre&gt;</div></td></tr> </table> Antony http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=Broadcom_install_for_RoCE_on_ROCm_devices&diff=32856&oldid=prev David: Created page with "<pre> ❯ cat broadcom-roce-setup.txt # ubuntu 22.04 apt install ibverbs-utils libibverbs1 rdmacm-utils librdmacm1 librdmacm-dev infiniband-diags perftest hwloc ibutils numa..." 2024-04-23T10:46:28Z <p>Created page with &quot;&lt;pre&gt; ❯ cat broadcom-roce-setup.txt # ubuntu 22.04 apt install ibverbs-utils libibverbs1 rdmacm-utils librdmacm1 librdmacm-dev infiniband-diags perftest hwloc ibutils numa...&quot;</p> <p><b>New page</b></p><div>&lt;pre&gt;<br /> ❯ cat broadcom-roce-setup.txt<br /> <br /> # ubuntu 22.04<br /> apt install ibverbs-utils libibverbs1 rdmacm-utils librdmacm1 librdmacm-dev infiniband-diags perftest hwloc ibutils numactl<br /> <br /> # latest version from: https://www.broadcom.com/products/ethernet-connectivity/network-adapters/p1200g<br /> curl -LO https://docs.broadcom.com/docs-and-downloads/ethernet-network-adapters/NXE/BRCM_229.1.123.0/bcm_229.1.123.0b.tar.gz<br /> tar zxvf bcm_229.1.123.0b.tar.gz<br /> cd bcm_229.1.123.0b/<br /> cd Linux/Linux_Installer/<br /> <br /> <br /> cd niccli/Linux_x86_64<br /> dpkg -i ./niccli_229.0.150.0-1_x86_64.deb<br /> <br /> cd niccli/<br /> tar zxvf sliff-229.0.150.0.tar.gz<br /> cd sliff-229.0.150.0/<br /> make<br /> make install<br /> modprobe sliff<br /> <br /> cd Linux/KMP-RoCE-Lib/<br /> tar zxvf libbnxt_re-229.0.139.0.tar.gz<br /> cd libbnxt_re-229.0.139.0/<br /> sh autogen.sh<br /> ./configure<br /> make<br /> make install<br /> cp bnxt_re.driver /etc/libibverbs.d/<br /> echo &quot;/usr/local/lib&quot; &gt;&gt; /etc/ld.so.conf<br /> ldconfig -v<br /> <br /> # little hacky bits, remove from install.sh script as its installed above.<br /> 229 ROCE_LINES=&quot;&quot;<br /> 230 if [ &quot;$ROCE&quot; == &quot;true&quot; ]; then<br /> 231 ROCE_LINES=&quot;<br /> 232 # - libbnxtre<br /> 233 - modules-load<br /> 234 - roce_cc<br /> 235 &quot;<br /> 236 fi<br /> <br /> <br /> cd Linux/Linux_Installer/<br /> bash install.sh -i enp139s0np0<br /> <br /> # reboot system<br /> # on reboot we should see<br /> root@gpu4:~/scratch/bcm_229.1.123.0b# ibstatus<br /> Infiniband device &#039;bnxt_re0&#039; port 1 status:<br /> default gid: fe80:0000:0000:0000:7ec2:55ff:feba:7388<br /> base lid: 0x0<br /> sm lid: 0x0<br /> state: 4: ACTIVE<br /> phys state: 5: LinkUp<br /> rate: 100 Gb/sec (4X EDR)<br /> link_layer: Ethernet<br /> <br /> <br /> root@gpu4:~/scratch/bcm_229.1.123.0b/Linux/Linux_Installer# rdma link show<br /> link bnxt_re0/1 state ACTIVE physical_state LINK_UP netdev enp139s0np0<br /> <br /> # benchmarks - standard bond1 numbers, no roce<br /> ubuntu@gpu3:~$ /opt/apps/ompi/5.0.x/bin/mpirun --mca btl_tcp_if_include bond1.103 -np 2 -H gpu3-ib,gpu4-ib /usr/local/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency<br /> [gpu3:08818] No HIP capabale device found. Disabling component.<br /> # OSU MPI Latency Test v7.3<br /> # Size Latency (us)<br /> # Datatype: MPI_CHAR.<br /> 1 29.19<br /> 2 29.23<br /> 4 29.17<br /> 8 29.14<br /> 16 29.05<br /> 32 29.23<br /> 64 29.32<br /> 128 29.28<br /> 256 30.56<br /> 512 30.65<br /> 1024 30.90<br /> 2048 31.47<br /> 4096 33.62<br /> 8192 36.95<br /> 16384 46.44<br /> 32768 59.56<br /> 65536 163.48<br /> 131072 176.76<br /> 262144 260.66<br /> 524288 377.56<br /> 1048576 632.83<br /> 2097152 1135.50<br /> 4194304 2127.84<br /> ubuntu@gpu3:~$ /opt/apps/ompi/5.0.x/bin/mpirun --mca btl_tcp_if_include bond1.103 -np 2 -H gpu3-ib,gpu4-ib /usr/local/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_<br /> osu_bibw osu_bw osu_latency osu_latency_mp osu_latency_mt osu_mbw_mr osu_multi_lat<br /> ubuntu@gpu3:~$ /opt/apps/ompi/5.0.x/bin/mpirun --mca btl_tcp_if_include bond1.103 -np 2 -H gpu3-ib,gpu4-ib /usr/local/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw<br /> [gpu3:08906] No HIP capabale device found. Disabling component.<br /> # OSU MPI Bandwidth Test v7.3<br /> # Size Bandwidth (MB/s)<br /> # Datatype: MPI_CHAR.<br /> 1 0.33<br /> 2 0.66<br /> 4 1.31<br /> 8 2.64<br /> 16 5.33<br /> 32 8.42<br /> 64 19.40<br /> 128 38.31<br /> 256 74.01<br /> 512 127.42<br /> 1024 249.89<br /> 2048 455.06<br /> 4096 789.85<br /> 8192 1276.01<br /> 16384 1839.29<br /> 32768 2287.82<br /> 65536 2351.62<br /> 131072 3006.95<br /> 262144 3881.76<br /> 524288 4573.31<br /> 1048576 4889.55<br /> 2097152 4917.25<br /> 4194304 4944.65<br /> <br /> # checking rdma<br /> # server: rping -s -d -C 1<br /> # client: rping -c -a 192.168.17.2 -d -C 1<br /> <br /> <br /> # ib_send_lat -d bnxt_re3<br /> # ib_send_lat -d bnxt_re3 192.168.17.1<br /> <br /> rdma link show<br /> link bnxt_re0/1 state DOWN physical_state DISABLED netdev enp37s0np0<br /> link bnxt_re1/1 state DOWN physical_state DISABLED netdev enp12s0np0<br /> link bnxt_re2/1 state ACTIVE physical_state LINK_UP netdev enp180s0np0<br /> link bnxt_re3/1 state ACTIVE physical_state LINK_UP netdev enp139s0np0:<br /> &lt;/pre&gt;</div> David