View source for Setup GPU Passthrough in OpenStack (Liberty) with Tesla K80

As part of the work we are doing with the OpenStack Community / Scientific Working Group (https://wiki.openstack.org/wiki/Scientific_working_group), the vScaler team are investigating the implementation of GPUs within an OpenStack environment for the purpose of HPC computing. 

This blog post will be added to added to as we progress our investigation. For now it covers the basic setup and configuration of GPUs (Tesla K80s) within an OpenStack environment (Liberty/Centos 7)

== Enable Memory Management ==

To ensure the devices perform in a virtualised / passthrough environment we need to enable IOMMU within the GPU server. IOMMU (I/O Memory Management Unit) is a feature supported by motherboard chipsets that provides enhanced virtual-to-physical memory mapping capabilities, including the ability to map large portions of non-contiguous memory. IOMMU can be enabled in the motherboard's BIOS, please refer to your server provider for instructions on how to ensure this is set. Once this is applied you can pass a boot parameter (<tt>intel_iommu=on</tt>) to the kernel at boot time to ensure its enabled and turned on within the OS. 

<syntaxhighlight>
# Ensure Grub is configured
[root@gpu ~]# grep intel_iommu=on /boot/grub2/grub.cfg | head -n 1 
	linux16 /vmlinuz-3.10.0-327.18.2.el7.x86_64 root=UUID=33011dab-c75a-45d0-b7a2-ae23545c850f ro quiet rdblacklist=nouveau intel_iommu=on 
</syntaxhighlight>

Then verify IOMMU is enabled when the system is booted up in <tt>dmesg</tt>

<syntaxhighlight>
[root@gpu ~]# dmesg | grep -iE "dmar|iommu" | grep -i enabled 
[    0.000000] Intel-IOMMU: enabled
</syntaxhighlight>

== Get the GPU IDs for the Nova configuration ==
The first thing we will need to capture is the vendor ID and device ID from the host system of the GPUs we want to pass through. In the example below we are using 1x NVIDIA Tesla K80 and 1x NVIDIA GRID K2

<syntaxhighlight>
# check for the nvidia GPUs
[root@gpu ~]# lspci | grep NVIDIA
04:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)
05:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)
83:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K2] (rev a1)
84:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K2] (rev a1)
# get their IDs with the -n flag
[root@gpu ~]# lspci -nn | grep NVIDIA
04:00.0 3D controller [0302]: NVIDIA Corporation GK210GL [Tesla K80] [10de:102d] (rev a1)
05:00.0 3D controller [0302]: NVIDIA Corporation GK210GL [Tesla K80] [10de:102d] (rev a1)
83:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104GL [GRID K2] [10de:11bf] (rev a1)
84:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104GL [GRID K2] [10de:11bf] (rev a1)
</syntaxhighlight>

Based on the above we now have: 
* The <b>vendor id</b> is: <tt>10de</tt> (Which corresponds to NVIDIA)
* And the <b>product id</b> is: <tt>102d</tt> and <tt>11bf</tt> (Where 102d is the K80 and 11bf is the GRID K2)

== Configure the GPU Nova Service
The next step is to configure the GPU server Nova Service with the appropriate passthrough flags: 
<syntaxhighlight>
# file: /etc/nova/nova.conf / On the GPU server
[DEFAULT]
...
pci_passthrough_whitelist={"vendor_id":"10de","product_id":"102d"}
</syntaxhighlight>

Restart the nova service 
<syntaxhighlight>
[root@gpu ~]# systemctl restart openstack-nova-compute
[root@gpu ~]# 
</syntaxhighlight>

== Configure the Controller(s) to be GPU aware ==
We then need to add the PCI configuration and resource scheduling parameters to the nova.conf on the controller(s)
<syntaxhighlight>
# In the file: /etc/nova/nova.conf on the controller
[DEFAULT]
...
pci_alias={"name":"K80_Tesla","vendor_id":"10de","product_id":"102d"}
scheduler_available_filters=nova.scheduler.filters.all_filters
scheduler_available_filters=nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter
scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,PciPassthroughFilter
scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
scheduler_max_attempts=5
</syntaxhighlight>

Restart all the nova services on the controller(s) to enable the configuration. 

== Setup the GPU Flavor in OpenStack ==
<syntaxhighlight>
# Replace with whatever CPU/RAM requirements are appropriate for your environment
openstack flavor create --public --ram 2048 --disk 20 --vcpus 2 m1.large.2xK80 

# Add a passthrough property for the flavor
openstack flavor set m1.large.2xK80 --property pci_passthrough:alias='K80_Tesla:2' 

# Then once created you should see something like this: 
openstack flavor show m1.large.1xK80 
+----------------------------+--------------------------------------+
| Field                      | Value                                |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                |
| OS-FLV-EXT-DATA:ephemeral  | 0                                    |
| disk                       | 20                                   |
| id                         | 481b0dd4-1148-4714-8f0b-fceb8aed884f |
| name                       | m1.large.2xK80                       |
| os-flavor-access:is_public | True                                 |
| properties                 | pci_passthrough:alias='K80_Tesla:2'  |
| ram                        | 2048                                 |
| rxtx_factor                | 1.0                                  |
| swap                       |                                      |
| vcpus                      | 2                                    |
+----------------------------+--------------------------------------+
</syntaxhighlight>

== Spin up an Instance with your new GPU flavor ==
Spin up an instance with CLI/GUI and ssh in, you'll hopefully see
<syntaxhighlight>
# yum install pciutils if a barebone centos 7
[root@gpupass1 ~]# lspci 
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device
00:04.0 SCSI storage controller: Red Hat, Inc Virtio block device
00:05.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1) # <----- BOOM!
00:06.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon
</syntaxhighlight>

== Next Steps ==
* Lots still to do 
** Performance Analysis
** GPU to GPU performance within a VM 
** GPU to GPU performance across nodes (SR-IOV on Mellanox Fabric)
** P100 cards to be added to lab environment shortly. 
** Mail david.power@vscaler.com for anything else you'd like to see!