Install and configure Intel Omni Path OPA Fabric

• Software can be downloaded from here: https://downloadcenter.intel.com/download/26064/Intel-Omni-Path-Fabric-Software-Including-Intel-Omni-Path-Host-Fabric-Interface-Driver- (As of Aug16)
• Docs etc: http://www.intel.com/content/www/us/en/support/network-and-i-o/fabric-products/000016242.html
• Two packages:
  • Basic: Just the compute nodes drivers and software (no subnet manager)
  • IFS: Includes the fabric manager / subnet manager

Install the OPA Fabric Software

# tested on an OpenHPC compute node - other deps may be required on vanilla centos nodes
yum install expect sysfsutils kernel-devel libibmad libibumad 
tar zxvf IntelOPA-IFS.RHEL72-x86_64.10.1.1.0.9.tgz 
cd IntelOPA-IFS.RHEL72-x86_64.10.1.1.0.9/
./INSTALL \
          -i opa_stack -i opa_stack_dev -i intel_hfi \
          -i delta_ipoib -i ibacm -i fastfabric \
          -i mvapich2_gcc_hfi -i mvapich2_intel_hfi \
          -i openmpi_gcc_hfi  -i openmpi_intel_hfi \
          -i opafm -i oftools -D opafm

# once installed its recommended that you reboot - NOTE OpenHPC nodes/make sure re-install isnt set. 
systemctl disable srpd
reboot

Verify the Fabric/Adaptor

Make sure the subnet manager is running

systemctl status opafm

Example output if the fabric manager is not running

[root@node001 IntelOPA-IFS.RHEL72-x86_64.10.1.1.0.9]# opainfo 
hfi1_0:1                           PortGUID:0x00117501017bfb57
   PortState:     Init (LinkUp)
   LinkSpeed      Act: 25Gb         En: 25Gb        
   LinkWidth      Act: 4            En: 4           
   LinkWidthDnGrd ActTx: 4  Rx: 4   En: 1,2,3,4     
   LCRC           Act: 14-bit       En: 14-bit,16-bit,48-bit       Mgmt: True 
   QSFP: PassiveCu, 2m   Hitachi Metals    P/N IQSFP26C-20       Rev 02
   Xmit Data:                  0 MB Pkts:                    0
   Recv Data:                  0 MB Pkts:                    0
   Link Quality: 5 (Excellent)

With the subnet manager running, link goes from INIT to ACTIVE

[root@node001 IntelOPA-IFS.RHEL72-x86_64.10.1.1.0.9]# opainfo 
hfi1_0:1                           PortGID:0xfe80000000000000:00117501017bfb57
   PortState:     Active
   LinkSpeed      Act: 25Gb         En: 25Gb        
   LinkWidth      Act: 4            En: 4           
   LinkWidthDnGrd ActTx: 4  Rx: 4   En: 3,4         
   LCRC           Act: 14-bit       En: 14-bit,16-bit,48-bit       Mgmt: True 
   LID: 0x00000001-0x00000001       SM LID: 0x00000001 SL: 0 
   QSFP: PassiveCu, 2m   Hitachi Metals    P/N IQSFP26C-20       Rev 02
   Xmit Data:                  1 MB Pkts:                 4355
   Recv Data:                  1 MB Pkts:                 4472
   Link Quality: 5 (Excellent)

Check the fabric details

[root@node001 ~]# opafabricinfo 
Fabric 0:0 Information:
SM: node001 hfi1_0 Guid: 0x00117501017bfb57 State: Master
Number of HFIs: 51
Number of Switches: 2
Number of Links: 67
Number of HFI Links: 51             (Internal: 0   External: 51)
Number of ISLs: 16                  (Internal: 0   External: 16)
Number of Degraded Links: 1         (HFI Links: 0   ISLs: 1)
Number of Omitted Links: 0          (HFI Links: 0   ISLs: 0)
-------------------------------------------------------------------------------