Difference between revisions of "VScaler: Advantech PCIe passthrough"

From Define Wiki
Jump to navigation Jump to search
 
(6 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
== The setup ==
 +
 +
* Intel(R) Xeon(R) CPU E3-1225 v5 @ 3.30GHz
 +
* 16 GB RAM
 +
* Advantech Co. Ltd Device M30 card and fpga.
 +
* Base OS: centos 7.6
 +
* OPenstack version queens
 +
* backend storage ceph
 +
 +
== vScaler setup and passthrough ==
 +
 +
We have setup an ALL in one scaler setup to passthrough the advantech card. The globals files for the installation is attached.
 +
 +
<syntaxhighlight>
 +
[root@localhost ~]# cat /etc/kolla/globals.yml | grep -v '#'
 +
---
 +
kolla_base_distro: "centos"
 +
kolla_install_type: "binary"
 +
openstack_release: "6.0.1"
 +
node_custom_config: "/etc/kolla/config"
 +
kolla_internal_vip_address: "185.93.28.189"
 +
kolla_external_vip_address: "{{ kolla_internal_vip_address }}"
 +
docker_registry: "registry.vscaler.com:5000"
 +
network_interface: "eno1"
 +
neutron_external_interface: "eno2"
 +
enable_ceph: "yes"
 +
enable_cinder: "yes"
 +
enable_haproxy: "no"
 +
 +
</syntaxhighlight>
 +
 +
 +
The advantech card has 2 parts M30 codec and fpga part . check in lscpi -s
 +
 +
<syntaxhighlight>
 +
[root@localhost ~]#lspci -s 09:00.0
 +
Multimedia controller: Advantech Co. Ltd Device 004b (rev 01)
 +
 +
[root@localhost ~]#lspci -s 0b:00.0
 +
0b:00.0 Multimedia controller: Device 50c1:0003 (rev 01)
 +
</syntaxhighlight>
 +
 +
Advantech cards pass-through works with the following set of QEMU* version of rpms. Download the same rpm to make the passthrough work
 +
 +
Note : the other Qemu* rpm will end the system in Hang mode and restart after creating virtual machine. 
 +
 +
<syntaxhighlight>
 +
[root@localhost ~]# ls qemu/
 +
qemu-img-ev-2.6.0-28.el7.centos_3.6.1.x86_64.rpm        qemu-kvm-ev-2.6.0-28.el7.centos_3.6.1.x86_64.rpm
 +
qemu-kvm-common-ev-2.6.0-28.el7.centos_3.6.1.x86_64.rpm  qemu-kvm-tools-ev-2.6.0-28.el7.centos_3.6.1.x86_64.rpm
 +
 +
root@localhost ~]# docker exec -it -u root nova_libvirt rpm  -qa | grep qemu
 +
qemu-kvm-common-ev-2.10.0-21.el7_5.4.1.x86_64
 +
qemu-kvm-ev-2.10.0-21.el7_5.4.1.x86_64
 +
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
 +
qemu-img-ev-2.10.0-21.el7_5.4.1.x86_64
 +
libvirt-daemon-driver-qemu-3.9.0-14.el7_5.6.x86_64
 +
centos-release-qemu-ev-1.0-3.el7.centos.noarch
 +
 +
[root@localhost ~]# docker exec -it -u root nova_compute rpm  -qa | grep qemu
 +
centos-release-qemu-ev-1.0-3.el7.centos.noarch
 +
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
 +
qemu-kvm-common-ev-2.10.0-21.el7_5.4.1.x86_64
 +
libvirt-daemon-driver-qemu-3.9.0-14.el7_5.6.x86_64
 +
qemu-kvm-ev-2.10.0-21.el7_5.4.1.x86_64
 +
qemu-img-ev-2.10.0-21.el7_5.4.1.x86_64
 +
 +
 +
[root@localhost ~]# docker exec -it -u root glance_api rpm  -qa | grep qemu
 +
qemu-img-ev-2.10.0-21.el7_5.4.1.x86_64
 +
centos-release-qemu-ev-1.0-3.el7.centos.noarch
 +
 +
 +
</syntaxhighlight>
 +
 +
The containers with Qemu* and QEMU-kvm* rpms are nova_libvirt , nova_compute , glance_api .
 +
The changes were not needed in qemu-img-ev* in cinder* and ceph containers.
 +
 +
== Configure nova-api (Controller) ==
 +
 +
Specify the PCI alias for the device.
 +
 +
Change the nova.conf libvirt are mandatory. Add the pci_white list information to while list the
 +
 +
 
<syntaxhighlight>
 
<syntaxhighlight>
 
[root@localhost ~]# vi /etc/kolla/config/nova.conf
 
[root@localhost ~]# vi /etc/kolla/config/nova.conf
Line 20: Line 105:
  
 
<syntaxhighlight>
 
<syntaxhighlight>
 +
 
#check nova-scheduler , nova-api , nova-compute all have the setting
 
#check nova-scheduler , nova-api , nova-compute all have the setting
  
Line 50: Line 136:
 
</syntaxhighlight>
 
</syntaxhighlight>
  
−
docker
+
 
 +
== Reconfgigure the Kolla ansible setup to mreflect the changes
 +
 
 +
<syntaxhighlight>
 +
[root@localhost ~]# kolla-ansible -i all_in_one reconfigure
 +
</syntaxhighlight>
  
  
Line 166: Line 257:
 
[12751.678451] vfio-pci 0000:05:00.0: irq 66 for MSI/MSI-X          #IRQ will be given Once the PCIe device is assigned in VM
 
[12751.678451] vfio-pci 0000:05:00.0: irq 66 for MSI/MSI-X          #IRQ will be given Once the PCIe device is assigned in VM
 
[179033.494736] vfio-pci 0000:05:00.0: irq 66 for MSI/MSI-X
 
[179033.494736] vfio-pci 0000:05:00.0: irq 66 for MSI/MSI-X
 +
 +
</syntaxhighlight>
 +
 +
<syntaxhighlight>
 +
[root@mediaflow-bioslab scratch]# tar -xvf vega_bqb_release_1_4_2_14270_x64.tar.bz2
 +
[root@mediaflow-bioslab scratch]# cd vega_bqb_release_1_4_2/vega3311_fpga_x64/
 +
driver/      example/      install.log  install.sh    Makefile      sdk/          tools/        uninstall.sh
 +
[root@mediaflow-bioslab vega3311_fpga_x64]# ./install.sh
 +
Installation start.
 +
CentOS Linux release 7.4.1708 (Core)
 +
kernel-headers-3.10.0-693.el7.x86_64
 +
kernel-devel-3.10.0-693.el7.x86_64
 +
gcc-4.8.5-39.el7.x86_64
 +
Making sample applications...
 +
done.
 +
Installing sample applications...
 +
cp -f ./example/vega3311_capture_app/bin/vega3311_cap_app /usr/local/bin
 +
cp -f ./example/vega3311_capture_config/bin/vega3311_cap_config /usr/local/bin
 +
cp -f ./example/vega3311_transmit_config/bin/vega3311_trans_config /usr/local/bin
 +
cp -f ./tools/vega3311_fpgautil/bin/vega3311_fpgautil /usr/local/bin
 +
Installing header file and library...
 +
mkdir -p /usr/local/include/libvega_capture_api
 +
mkdir -p /usr/local/include/libvega_transmit_api
 +
cp -f ./driver/lib/libvega3311_fpga_drv.a /usr/local/lib
 +
cp -f ./driver/lib/libvega3311_fpga_drv.so /usr/local/lib
 +
cp -f ./sdk/vega3311_capture_api/lib/libvega3311_capture.a /usr/local/lib
 +
cp -f ./sdk/vega3311_capture_api/lib/libvega3311_capture.so /usr/local/lib
 +
cp -f ./sdk/vega3311_transmit_api/lib/libvega3311_transmit.a /usr/local/lib
 +
cp -f ./sdk/vega3311_transmit_api/lib/libvega3311_transmit.so /usr/local/lib
 +
cp -f ./driver/include/vega3311_fpga_drv.h /usr/local/include/libvega_capture_api
 +
cp -f ./sdk/vega3311_capture_api/include/VEGA3311_capture.h /usr/local/include/libvega_capture_api
 +
cp -f ./sdk/vega3311_capture_api/include/VEGA3311_cap_types.h /usr/local/include/libvega_capture_api
 +
cp -f ./sdk/vega3311_transmit_api/include/VEGA3311_transmit.h /usr/local/include/libvega_transmit_api
 +
cp -f ./sdk/vega3311_transmit_api/include/VEGA3311_trans_types.h /usr/local/include/libvega_transmit_api
 +
Clean the package...
 +
done.
 +
Making and loading device driver...done.
 +
Installation completed.
 +
 +
[root@mediaflow-bioslab scratch]# lspci -vs 09:00.0
 +
09:00.0 Multimedia controller: Advantech Co. Ltd Device 004b (rev 01)
 +
Subsystem: Advantech Co. Ltd Device 004b
 +
Physical Slot: 1
 +
Flags: bus master, fast devsel, latency 0, IRQ 18
 +
Memory at de800000 (32-bit, non-prefetchable) [size=128K]
 +
Memory at de400000 (32-bit, non-prefetchable) [size=4M]
 +
Memory at 3fff800000 (64-bit, prefetchable) [size=4M]
 +
Capabilities: [80] Power Management version 3
 +
Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit+
 +
Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-
 +
Capabilities: [c0] Express Endpoint, MSI 00
 +
Capabilities: [100] Advanced Error Reporting
 +
Capabilities: [340] Vendor Specific Information: ID=0001 Rev=0 Len=02c <?>
 +
Kernel driver in use: vega3311_fpga
 +
Kernel modules: vega3311_fpga
 +
 +
 +
</syntaxhighlight>
 +
 +
To verify the drivers are installed correctly chyeck the following command
 +
 +
<syntaxhighlight>
 +
 +
[root@mediaflow-bioslab scratch]# vega3311_cap_app list_dev
 +
DRV INF: SF: Got idcode 0xc2 0x20 0x1a
 +
Detected 1 chip(s).
 +
==============================================
 +
Chip: 0
 +
Device node name: /dev/vega3311_fpga_0
 +
PCI Bridge 0x08
 +
PCI Bus:Slot.Func 09:00.0
 +
Vendor ID: 0x13fe Device ID: 0x004b
 +
Subsystem VendorID: 0x13fe Subsystem DevID: 0x004b
 +
Serial Number: LKD0114896 Board ID: 9792A33111E
 +
Irq: 18
 +
BAR Configuration:
 +
Start              | Length
 +
0x00000000de800000 | 000131072
 +
0x00000000de400000 | 004194304
 +
==============================================
 +
[root@mediaflow-bioslab scratch]#
 +
[root@mediaflow-bioslab scratch]# vega3311_fpgautil list
 +
[Card 0] Device id 0, PCIe bridge 8
 +
   
 +
 +
</syntaxhighlight>
 +
Compling drivers for the fgpa (history back up from Roy VM)
 +
 +
<syntaxhighlight>
 +
1  sudo su -
 +
    2  exit
 +
    3  lspci
 +
    4  lspci -vvs 05.0
 +
    5  lspci -vvvs 05.0
 +
    6  sudo lspci -vvvs 05.0
 +
    7  sudo lspci -vvvs 06.0
 +
    8  ls
 +
    9  tar -jxvf vega_bqb_trc_release_1_5_1_12817_x64.tar.bz2
 +
  10  sudo yum list bzip2
 +
  11  sudo yum install bzip2
 +
  12  tar -jxvf vega_bqb_trc_release_1_5_1_12817_x64.tar.bz2
 +
  13  ls
 +
  14  cd vega_bqb_trc_release_1_5_1
 +
  15  ls
 +
  16  cd vega_bqb_codec_x64/01_HOST/realtime_codec_soft
 +
  17  ls
 +
  18  sudo ./install.sh
 +
  19  uname -r
 +
  20  sudo yum install kernel-headers-3.10.0-862.2.3.el7.x86_64
 +
  21  sudo yum install kernel-headers-3.10.0-862.2.3
 +
  22  sudo yum list kernel-headers
 +
  23  sudo yum list kernel-headers --showduplicates
 +
  24  uname -r
 +
  25  sudo yum update
 +
  26  sudo systemctl reboot
 +
  27  uname -r
 +
  28  cd vega_bqb_trc_release_1_5_1
 +
  29  cd vega_bqb_codec_x64/01_HOST/realtime_codec_soft
 +
  30  sudo ./install.sh
 +
  31  sudo dmesg | grep m30trc
 +
  32  vega_bqb_vtrc version
 +
  33  ls /dev
 +
  34  lsmob
 +
  35  lsmod
 +
  36  ls
 +
  37  sudo ./uninstall.sh
 +
  38  ls
 +
  39  lsmod
 +
  40  cd
 +
  41  #cd ~
 +
  42  ls
 +
  43  rm -r vega_bqb_trc_release_1_5_1
 +
  44  ls
 +
  45  rm vega_bqb_trc_release_1_5_1_12817_x64.tar.bz2
 +
  46  ls
 +
  47  tar -jxvf vega_bqb_release_1_3_1_12729_x64.tar.bz2
 +
  48  ls
 +
  49  cd vega_bqb_release_1_3_1
 +
  50  cd vega_bqb_codec_x64/01_HOST/realtime_codec_soft/
 +
  51  sudo ./install.sh
 +
  52  sudo dmesg | grep "m30*"
 +
  53  cd ~
 +
  54  cd vega_bqb_release_1_3_1
 +
  55  cd vega3311_fpga_x64/
 +
  56  ls
 +
  57  sudo ./install.sh
 +
  58  sudo dmesg | grep "vega3311*"
 +
  59  vega_bqb_sysctl set_enc --device 0
 +
  60  vega_bqb_sysctl version
 +
  61  vega_bqb_sysctl info
 +
  62  vega_bqb_venc version
 +
  63  sudo apt-get install mercurial cmake cmake-curses-gui build-essential yasm
 +
  64  sudo yum install mercurial cmake cmake-gui
 +
  65  sudo yum groupinstall "Development Tools"
 +
  66  sudo yum install -y epel-release
 +
  67  sudo yum --disablerepo="*" --enablerepo="epel" install -y yasm
 +
  68  sudo yum install -y SDL-devel
 +
  69  cd ~
 +
  70  wget http://sourceforge.net/projects/boost/files/boost/1.54.0/boost_1_54_0.tar.gz
 +
  71  sudo yum install wget
 +
  72  wget http://sourceforge.net/projects/boost/files/boost/1.54.0/boost_1_54_0.tar.gz
 +
  73  tar -xzvf boost_1_54_0.tar.gz
 +
  74  cd ~/boost_1_54_0
 +
  75  ./bootstrap.sh
 +
  76  sudo ./bjam --layout=system --toolset=gcc install
 +
  77  cd ~/vega_bqb_release_1_3_1/ffmpeg-3.4.1/vega_bqb_avc/build/linux
 +
  78  PATH="/usr/local/bin:$PATH" cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX="/usr/local" -DENABLE_SHARED:bool=on -DENABLE_PIC:bool=on ../../source
 +
  79  make
 +
  80  sudo make install
 +
  81  cd ~/vega_bqb_release_1_3_1/ffmpeg-3.4.1/vega_bqb_hevc/build/linux
 +
  82  PATH="/usr/local/bin:$PATH" cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX="/usr/local" -DENABLE_SHARED:bool=on -DENABLE_PIC:bool=on ../../source
 +
  83  make
 +
  84  sudo make install
 +
  85  cd ~/vega_bqb_release_1_3_1/ffmpeg-3.4.1/vega3311_ff_capture/build/linux
 +
  86  PATH="/usr/local/bin:$PATH" cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX="/usr/local" -DENABLE_SHARED:bool=on -DENABLE_PIC:bool=on ../../source
 +
  87  make
 +
  88  sudo make install
 +
  89  cd ~/vega_bqb_release_1_3_1/ffmpeg-3.4.1/ffmpeg_source
 +
  90  ls /usr/local/lib
 +
  91  ls /usr/local/lib/pkgconfig
 +
  92  PATH="/usr/local/bin:$PATH" PKG_CONFIG_PATH="/usr/local/lib/pkgconfig"
 +
  93  ./configure --enable-shared --enable-libvegabqbavc --enable-libvegabqbhevc --enable-nonfree
 +
  94  echo $PKG_CONFIG_PATH
 +
  95  pkg-config --help
 +
  96  pkg-config --exists --print-errors libvegabqbavc
 +
  97  find /usr -name libvegabqbavc.pc
 +
  98  find /usr -name vega_bqb_hevc.pc
 +
  99  find ~/ -name libvegabqbavc.pc
 +
  100  find ~/ -name vega_bqb_hevc.pc
 +
  101  less /home/centos/vega_bqb_release_1_3_1/ffmpeg-3.4.1/vega_bqb_hevc/build/linux/vega_bqb_hevc.pc
 +
  102  make
 +
  103  sudo make install
 +
  104  cd ~
 +
  105  sudo ldconfig
 +
  106  ffmpeg
 +
  107  ls mnt
 +
  108  ls /mnt
 +
  109  ls /media
 +
  110  df -h
 +
  111  lspci -t
 +
  112  lspci -vmms 05.0
 +
  113  lspci -vmms 06.0
 +
  114  vega3311_cap_app qsrc 0 0
 +
  115  vega3311_cap_app qsrc 0 1
 +
  116  vega3311_cap_app qsrc 0 2
 +
  117  vega3311_cap_app qsrc 0 3
 +
  118  vega3311_cap_app qsrc 1 0
 +
  119  vega3311_cap_config 0 0 -input_mode 4ch_fhd
 +
  120  vega3311_cap_config 0 0 -input_mode 4ch
 +
  121  vega3311_cap_app qsrc 0 0
 +
  122  vega3311_cap_app qsrc 0 1
 +
  123  vega3311_cap_config 0 0 -image_format nv12
 +
  124  vega3311_cap_config -h
 +
  125  vega3311_cap_app -h
 +
  126  vega3311_cap_app reset 0 0
 +
  127  vega3311_cap_app list_dev
 +
  128  systemctl reboot
 +
  129  sudo systemctl reboot
 +
  130  vega3311_cap_config 0 0 -image_format nv16
 +
  131  ls
 +
  132  cd vega_bqb_release_1_3_1
 +
  133  ls
 +
  134  cd vega3311_fpga_x64/tools/fpga_fw/
 +
  135  ls
 +
  136  vega3311_cap_app qsrc 0 0
 +
  137  vega_bqb_venc version
 +
  138  vega_bqb_sysctl set_dec --device 0
 +
  139  vega_bqb_venc version
 +
  140  ls
 +
  141  cd ..
 +
  142  ls
 +
  143  cd vega_bqb_codec_x64
 +
  144  ls
 +
  145  cd 02_M30
 +
  146  ls
 +
  147  cd 02_tools
 +
  148  ls
 +
  149  mv config_system_dec.txt ../
 +
  150  ls
 +
  151  sudo ./flash_write_all.sh 0
 +
  152  sudo systemctl reboot
 +
  153  vega_bqb_venc version
 +
  154  sudo lspci -vvvs 05.0
 +
  155  sudo lspci -vvvvs 05.0
 +
  156  sudo lspci -vvvvs 06.0
 +
  157  ls
 +
  158  cd vega_bqb_release_1_3_1
 +
  159  ls
 +
  160  cd vega3311_fpga_x64/tools/fpga_fw
 +
  161  ls
 +
  162  cd ..
 +
  163  ls
 +
  164  cd vega3311_fpgautil
 +
  165  ls
 +
  166  cd ..
 +
  167  cd vega3311_fpga_x64/tools/fpga_fw
 +
  168  ls
 +
  169  vega3311_fpgautil fw_update 0 vega3311_0_6C_20181029_3G12G_wt.bin
 +
  170  lspci
 +
  171  lspci -m
 +
  172  lspci -mm
 +
  173  lspci -v
 +
  174  free -h
 +
  175  vega3311_cap_app qsrc 0 0
 +
  176  vega3311_cap_app list_dev
 +
  177  sudo dmesg | grep -e BAR
 +
  178  sudo dmesg | grep -e vega3311_fpga
 +
  179  sudo lspci -vvvs 05.0
 +
  180  ls /sys
 +
  181  ls /sys/dev
 +
  182  ls /sys/device
 +
  183  sudo ls /sys/device
 +
  184  sudo ls /sys/devices
 +
  185  sudo ls /sys/bus
 +
  186  ls /sys/bus
 +
  187  ls /sys/bus/pci
 +
  188  ls /sys/bus/pci/devices
 +
  189  ls /sys/bus/pci/devices/0000:00:05.0
 +
  190  ls /sys/bus/pci/devices/0000:00:05.0/vega3311_fpga
 +
  191  ls /sys/bus/pci/devices/0000:00:05.0/vega3311_fpga/vega3311_fpga_0
 +
  192  ls /sys/bus/pci/devices/0000:00:05.0
 +
  193  ls -l /sys/bus/pci/devices/0000:00:05.0
 +
  194  ls -l /sys/bus/pci/devices/0000:00:05.0/
 +
  195  sudo less -l /sys/bus/pci/devices/0000:00:05.0/max_link_width
 +
  196  sudo less -l /sys/bus/pci/devices/0000:00:05.0/max_link_speed
 +
  197  sudo less -l /sys/bus/pci/devices/0000:00:05.0/irq
 +
  198  ls -l /sys/bus/pci/devices/0000:00:05.0/firmware_node
 +
  199  ls -l /sys/bus/pci/devices/0000:00:05.0/firmware_node/
 +
  200  sudo less /sys/bus/pci/devices/0000:00:05.0/firmware_node/adr
 +
  201  cd /~
 +
  202  cd ~
 +
  203  ls
 +
  204  vega_bqb_sysctl set_enc –device 0
 +
  205  vega_bqb_sysctl set_enc --device 0
 +
  206  vega_bqb_venc version
 +
  207  vega_bqb_venc reset
 +
  208  ls /dev
 +
  209  sudo systemctl reboot
 +
  210  vega_bqb_venc version
 +
  211  sudo systemctl poweroff
 +
  212  vega_bqb_venc version
 +
  213  vega_bqb_sysctl set_enc --device 0
 +
  214  vega_bqb_venc version
 +
  215  vega_bqb_sysctl set_dec --device 0
 +
  216  vega_bqb_venc version
 +
  217  ls
 +
  218  cd vega_bqb_release_1_3_1
 +
  219  cd vega_bqb_codec_x64/02_M30
 +
  220  ls
 +
  221  mv config_system_dec.txt ./01_M30_firm  02_tools
 +
  222  ls
 +
  223  ls 02_tools
 +
  224  cd 02_tools
 +
  225  ls
 +
  226  mv 01_M30_firm ../
 +
  227  ls
 +
  228  ls ../
 +
  229  mv config_system.txt ../
 +
  230  ls
 +
  231  sudo ./flash_write_all.sh 0
 +
  232  ls
 +
  233  less flash_write_mcpu_fw.sh
 +
  234  sudo systemctl reboot
 +
  235  vega_bqb_sysctl set_enc --device 0
 +
  236  vega_bqb_venc version
 +
  237  vega_bqb_sysctl set_dec --device 0
 +
  238  vega_bqb_venc version
 +
  239  ls
 +
  240  cd vega_bqb_release_1_3_1
 +
  241  ls
 +
  242  cd vega_bqb_codec_x64/02_M30
 +
  243  ls
 +
  244  mv config_system.txt ./02_tools
 +
  245  ls
 +
  246  cd 02_tools
 +
  247  ls
 +
  248  sudo ./flash_write_all.sh 0
 +
  249  sudo poweroff
 +
  250  vega_bqb_sysctl set_enc --device 0
 +
  251  vega_bqb_venc version
 +
  252  vega3311_cap_app list_dev
 +
  253  logout
 +
  254  yum list tmux
 +
  255  yum list screen
 +
  256  vega3311_cap_app list_dev
 +
  257  sudo vega3311_cap_app list_dev
 +
  258  sudo /usr/local/bin/vega3311_cap_app list_dev
 +
  259  lsmod
 +
  260  free -h
 +
  261  df -h
 +
  262  ls
 +
  263  ffmpeg -vsync vfr -f mpegts -i ~/Videos/bbb.ts -c:v libvegabqbavc -b:v 4000k -vegabqbavc-params profile=high:level-idc=51:open-gop=2:keyint=30:scenecut=1:bframes=2:deblock=1:rate-ctrl=cbr:bitrate=4000:sar=1:device=0:channel=0:input-mode=0 -c:a aac ~/Videos/bbb.mp4
 +
  264  alias
 +
  265  logout
 +
  266  lspci
 +
  267  sudo lspci -vvv -s 05.0
 +
  268  sudo poweroff
 +
  269  lspci
 +
  270  sudo lspci -vvv -s 05.0
 +
  271  vega3311_cap_app qsrc 0 0
 +
  272  vega_bqb_sysctl set_enc --device 0
 +
  273  vega_bqb_venc version
 +
  274  vega_bqb_sysctl set_dec --device 0
 +
  275  vega3311_cap_app list_dev
 +
  276  sudo lspci -vvv -s 05.0
 +
  277  ls
 +
  278  cd vega_bqb_release_1_3_1
 +
  279  ls
 +
  280  cd vega3311_fpga_x64
 +
  281  ls
 +
  282  cd ../vega3310_fpga_x64
 +
  283  ls
 +
  284  cd ..
 +
  285  cd vega3311_fpga_x64
 +
  286  less install.log
 +
  287  modinfo vega3311_fpga
 +
  288  lspci
 +
  289  lspci -n -s 05.0
 +
  290  sudo lspci -vvvv -s 5.0
 +
  291  ls /sys/bus/pci
 +
  292  ls /sys/bus/pci/devices
 +
  293  ls /sys/bus/pci/devices/0000:00:05.0
 +
  294  ls -l
 +
  295  ls -l /sys/bus/pci/devices/0000:00:05.0/
 +
  296  ls -l /sys/bus/pci/devices/0000:00:06.0/
 +
  297  ls -l /sys/bus/pci/devices/0000:00:05.0/firmware_node/
 +
  298  less /sys/bus/pci/devices/0000:00:05.0/irq
 +
  299  less /sys/bus/pci/devices/0000:00:06.0/irq
 +
  300  sudo cat /sys/bus/pci/devices/0000:00:05.0/irq
 +
  301  sudo cat /sys/bus/pci/devices/0000:00:06.0/irq
 +
  302  vega_bqb_sysctl set_enc --device 0
 +
  303  vega_bqb_venc version
 +
  304  vega_bqb_venc version
 +
  305  stty -ixon
 +
  306  sudo lspci -vvv -s 05.0
 +
  307  sudo cat /proc/interrupts
 +
  308  sudo lspci -vvv -s 06.0
 +
  309  logout
 +
  310  sudo cat /proc/interrupts
 +
  311  vega3311_cap_app qsrc 0 0
 +
  312  sudo cat /proc/interrupts
 +
  313  sudo journalctl -k | grep remap
 +
  314  sudo poweroff
 +
  315  sudo cat /proc/interrupts
 +
  316  vega3311_cap_app qsrc 0 0
 +
  317  sudo cat /proc/interrupts
 +
  318  sudo dmesg | grep less
 +
  319  sudo dmesg | less
 +
  320  sudo cat /proc/interrupts
 +
  321  sudo dmesg | less
 +
  322  sudo poweroff
 +
  323  sudo dmesg | less
 +
  324  ls
 +
  325  cd vega_bqb_release_1_3_1
 +
  326  ls
 +
  327  cd vega3311_fpga_x64
 +
  328  ls
 +
  329  less install.log
 +
  330  sudo ./uninstall.sh
 +
  331  lsmod
 +
  332  sudo dmesg | less
 +
  333  sudo cat /proc/interrupts
 +
  334  sudo lspci -vvv -s 05.0
 +
  335  ls
 +
  336  sudo ./uninstall.sh
 +
  337  sudo ./install.sh
 +
  338  sudo dmesg | less
 +
  339  sudo cat /proc/interrupts
 +
  340  vega3311_cap_app list_dev
 +
  341  sudo cat /proc/interrupts
 +
  342  sudo dmesg | less
 +
  343  sudo cat /proc/interrupts
 +
  344  vega3311_cap_app list_dev
 +
  345  sudo cat /proc/interrupts
 +
  346  vega3311_cap_app list_dev
 +
  347  sudo cat /proc/interrupts
 +
  348  vega_bqb_sysctl set_enc --device 0
 +
  349  sudo cat /proc/interrupts
 +
  350  vega_bqb_venc version
 +
  351  sudo cat /proc/interrupts
 +
  352  ls -l /sys/bus/pci/devices/0000:00:05.0/
 +
  353  cat /sys/bus/pci/devices/0000:00:05.0/enable
 +
  354  cat /sys/bus/pci/devices/0000:00:05.0/msi_bus
 +
  355  ls -l /sys/bus/pci/devices/0000:00:05.0/msi_irqs/
 +
  356  cat /sys/bus/pci/devices/0000:00:05.0/msi_irqs/29
 +
  357  cat /sys/bus/pci/devices/0000:00:05.0/msi_irqs/30
 +
  358  cat /sys/bus/pci/devices/0000:00:05.0/msi_irqs/37
 +
  359  ls
 +
  360  ffmpeg
 +
  361  logout
 +
  362  ls
 +
  363  cd vega_bqb_release_1_3_1
 +
  364  ls
 +
  365  cd vega3311_ff_capture/build/linux
 +
  366  ls
 +
  367  cd ffmpeg-3.4.1
 +
  368  cd vega3311_ff_capture/build/linux
 +
  369  ls
 +
  370  PATH="/usr/local/bin:$PATH" cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX="/usr/local" -DENABLE_SHARED:bool=on -DENABLE_PIC:bool=on ../../source
 +
  371  make
 +
  372  sudo make install
 +
  373  cd ~/vega_bqb_release_1_3_1
 +
  374  cd ffmpeg-3.4.1
 +
  375  cd ffmpeg_source
 +
  376  ls
 +
  377  PATH="/usr/local/bin:$PATH" PKG_CONFIG_PATH="/usr/local/lib/pkgconfig"
 +
  378  ./configure --enable-shared --enable-libvegabqbavc --enable-libvegabqbhevc --enable-libvega3311-capture --enable-nonfree
 +
  379  logout
 +
  380  dmesg | grep -i 05.0
 +
  381  sudo cat /proc/interrupts
 +
  382  sudo poweroff
 +
  383  sudo cat /proc/interrupts
 +
  384  lspci -vs 05.0
 +
  385  sudolspci -vs 05.0
 +
  386  sudo lspci -vs 05.0
 +
  387  sudo lspci -vv | grep -i "IRQ 10"
 +
  388  sudo lspci -vs 06.0
 +
  389  sudo lspci -vv | grep -i "IRQ 50"
 +
  390  logout
 +
  391  sudo su -
 +
  392  ls
 +
  393  vi Videos/
 +
  394  cd Videos/
 +
  395  ls
 +
  396  cd ..
 +
  397  ls
 +
  398  history
  
 
</syntaxhighlight>
 
</syntaxhighlight>

Latest revision as of 17:41, 21 February 2020

The setup

  • Intel(R) Xeon(R) CPU E3-1225 v5 @ 3.30GHz
  • 16 GB RAM
  • Advantech Co. Ltd Device M30 card and fpga.
  • Base OS: centos 7.6
  • OPenstack version queens
  • backend storage ceph

vScaler setup and passthrough

We have setup an ALL in one scaler setup to passthrough the advantech card. The globals files for the installation is attached. 

[root@localhost ~]# cat /etc/kolla/globals.yml | grep -v '#'
---
kolla_base_distro: "centos"
kolla_install_type: "binary"
openstack_release: "6.0.1"
node_custom_config: "/etc/kolla/config"
kolla_internal_vip_address: "185.93.28.189"
kolla_external_vip_address: "{{ kolla_internal_vip_address }}"
docker_registry: "registry.vscaler.com:5000"
network_interface: "eno1"
neutron_external_interface: "eno2"
enable_ceph: "yes"
enable_cinder: "yes"
enable_haproxy: "no"


The advantech card has 2 parts M30 codec and fpga part . check in lscpi -s 

[root@localhost ~]#lspci -s 09:00.0 
Multimedia controller: Advantech Co. Ltd Device 004b (rev 01)

[root@localhost ~]#lspci -s 0b:00.0 
0b:00.0 Multimedia controller: Device 50c1:0003 (rev 01)

Advantech cards pass-through works with the following set of QEMU* version of rpms. Download the same rpm to make the passthrough work

Note : the other Qemu* rpm will end the system in Hang mode and restart after creating virtual machine. 

[root@localhost ~]# ls qemu/
qemu-img-ev-2.6.0-28.el7.centos_3.6.1.x86_64.rpm         qemu-kvm-ev-2.6.0-28.el7.centos_3.6.1.x86_64.rpm
qemu-kvm-common-ev-2.6.0-28.el7.centos_3.6.1.x86_64.rpm  qemu-kvm-tools-ev-2.6.0-28.el7.centos_3.6.1.x86_64.rpm

root@localhost ~]# docker exec -it -u root nova_libvirt rpm  -qa | grep qemu
qemu-kvm-common-ev-2.10.0-21.el7_5.4.1.x86_64
qemu-kvm-ev-2.10.0-21.el7_5.4.1.x86_64
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
qemu-img-ev-2.10.0-21.el7_5.4.1.x86_64
libvirt-daemon-driver-qemu-3.9.0-14.el7_5.6.x86_64
centos-release-qemu-ev-1.0-3.el7.centos.noarch

[root@localhost ~]# docker exec -it -u root nova_compute rpm  -qa | grep qemu
centos-release-qemu-ev-1.0-3.el7.centos.noarch
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
qemu-kvm-common-ev-2.10.0-21.el7_5.4.1.x86_64
libvirt-daemon-driver-qemu-3.9.0-14.el7_5.6.x86_64
qemu-kvm-ev-2.10.0-21.el7_5.4.1.x86_64
qemu-img-ev-2.10.0-21.el7_5.4.1.x86_64


[root@localhost ~]# docker exec -it -u root glance_api rpm  -qa | grep qemu
qemu-img-ev-2.10.0-21.el7_5.4.1.x86_64
centos-release-qemu-ev-1.0-3.el7.centos.noarch

The containers with Qemu* and QEMU-kvm* rpms are nova_libvirt , nova_compute , glance_api . The changes were not needed in qemu-img-ev* in cinder* and ceph containers.

Configure nova-api (Controller)

Specify the PCI alias for the device.

Change the nova.conf libvirt are mandatory. Add the pci_white list information to while list the 


[root@localhost ~]# vi /etc/kolla/config/nova.conf
[root@localhost ~]# cat /etc/kolla/config/nova.conf
[filter_scheduler]
enabled_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, PciPassthroughFilter
available_filters=nova.scheduler.filters.all_filters

[libvirt]
cpu_mode = custom
cpu_model = Skylake-Client

[pci]
passthrough_whitelist = { "address": "0000:05:00.0" }
alias={"vendor_id":"50c1","product_id":"0003","device_type":"type-PCI","name":"a1"}

[root@localhost ~]# kolla-ansible -i all-in-one reconfigure
#check nova-scheduler , nova-api , nova-compute all have the setting

docker exec -it nova_compute cat /etc/nova/nova.conf
....
[filter_scheduler]
enabled_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, PciPassthroughFilter
available_filters = nova.scheduler.filters.all_filters

[pci]
passthrough_whitelist = { "address": "0000:05:00.0" }
alias = {"vendor_id":"50c1","product_id":"0003","device_type":"type-PCI","name":"a1"}

docker exec -it nova_api cat /etc/nova/nova.conf

....
[filter_scheduler]
enabled_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, PciPassthroughFilter
available_filters = nova.scheduler.filters.all_filters

[pci]
passthrough_whitelist = { "address": "0000:05:00.0" }
alias = {"vendor_id":"50c1","product_id":"0003","device_type":"type-PCI","name":"a1"}

docker exec -it nova_compute cat /etc/nova/nova.conf


== Reconfgigure the Kolla ansible setup to mreflect the changes 

[root@localhost ~]# kolla-ansible -i all_in_one reconfigure


[root@localhost ~]# openstack image create --disk-format raw --container-format bare --public  --file CentOS-7-x86_64-Everything-1804.iso Centos7.5_everything


# Make sure enough RAM is allocated 8G worked at time of testing 

[root@localhost ~]# openstack flavor create m1.pass_8G --ram 8024 --disk 0 --vcpu 2
[root@localhost ~]# openstack flavor set m1.pass_8G --property "pci_passthrough:alias"="a1:2"
[root@localhost ~]# openstack flavor show  m1.pass_8G
+----------------------------+--------------------------------------+
| Field                      | Value                                |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                |
| OS-FLV-EXT-DATA:ephemeral  | 0                                    |
| access_project_ids         | None                                 |
| disk                       | 0                                    |
| id                         | d476fa2d-71d6-4686-b363-2b84baac3a3d |
| name                       | m1.pass_8G                           |
| os-flavor-access:is_public | True                                 |
| properties                 | pci_passthrough:alias='a1:1'         |
| ram                        | 8024                                 |
| rxtx_factor                | 1.0                                  |
| swap                       |                                      |
| vcpus                      | 2                                    |
+----------------------------+--------------------------------------+
[root@localhost ~]#
# to enable vfio drivers 

root@localhost ~]# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=50c1:0003
root@localhost ~]# echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf
root@localhost ~]# reboot

# In base OS 
root@localhost ~]# lspci -vvvvvvns 05:00
[root@localhost ~]# lspci -vvns 05:00
05:00.0 0480: 50c1:0003 (rev 01)
        Subsystem: 50c1:0001
        Physical Slot: 41
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin ? routed to IRQ 66
        Region 0: Memory at df400000 (32-bit, non-prefetchable) [size=128K]
        Region 2: Memory at df200000 (32-bit, non-prefetchable) [size=2M]
        Region 4: Memory at df430000 (32-bit, non-prefetchable) [size=32K]
        Region 5: Memory at df420000 (32-bit, non-prefetchable) [size=64K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [c0] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000fee004d8  Data: 0000
        Capabilities: [80] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [200 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=01
                        Status: NegoPending- InProgress-
        Capabilities: [3a0 v1] Device Serial Number 00-00-00-00-00-00-00-01
        Capabilities: [400 v1] Vendor Specific Information: ID=0020 Rev=3 Len=100 <?>
        '''Kernel driver in use: vfio-pci'''


[root@localhost ~]# dmesg | grep 05:00
[    0.223047] pci 0000:05:00.0: [50c1:0003] type 00 class 0x048000
[    0.223063] pci 0000:05:00.0: reg 0x10: [mem 0xdf400000-0xdf41ffff]
[    0.223077] pci 0000:05:00.0: reg 0x18: [mem 0xdf200000-0xdf3fffff]
[    0.223091] pci 0000:05:00.0: reg 0x20: [mem 0xdf430000-0xdf437fff]
[    0.223099] pci 0000:05:00.0: reg 0x24: [mem 0xdf420000-0xdf42ffff]
[    0.223154] pci 0000:05:00.0: supports D1 D2
[    0.223155] pci 0000:05:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.515870] iommu: Adding device 0000:05:00.0 to group 1
[    0.548264] pci 0000:05:00.0: Signaling PME through PCIe PME interrupt
[ 2036.852908] vfio-pci 0000:05:00.0: enabling device (0400 -> 0402)
[12533.193714] vfio-pci 0000:05:00.0: enabling device (0400 -> 0402)
[12751.678451] vfio-pci 0000:05:00.0: irq 66 for MSI/MSI-X           #IRQ will be given Once the PCIe device is assigned in VM
[179033.494736] vfio-pci 0000:05:00.0: irq 66 for MSI/MSI-X
[root@mediaflow-bioslab scratch]# tar -xvf vega_bqb_release_1_4_2_14270_x64.tar.bz2
[root@mediaflow-bioslab scratch]# cd vega_bqb_release_1_4_2/vega3311_fpga_x64/
driver/       example/      install.log   install.sh    Makefile      sdk/          tools/        uninstall.sh
[root@mediaflow-bioslab vega3311_fpga_x64]# ./install.sh
Installation start.
CentOS Linux release 7.4.1708 (Core)
kernel-headers-3.10.0-693.el7.x86_64
kernel-devel-3.10.0-693.el7.x86_64
gcc-4.8.5-39.el7.x86_64
Making sample applications...
done.
Installing sample applications...
cp -f ./example/vega3311_capture_app/bin/vega3311_cap_app /usr/local/bin
cp -f ./example/vega3311_capture_config/bin/vega3311_cap_config /usr/local/bin
cp -f ./example/vega3311_transmit_config/bin/vega3311_trans_config /usr/local/bin
cp -f ./tools/vega3311_fpgautil/bin/vega3311_fpgautil /usr/local/bin
Installing header file and library...
mkdir -p /usr/local/include/libvega_capture_api
mkdir -p /usr/local/include/libvega_transmit_api
cp -f ./driver/lib/libvega3311_fpga_drv.a /usr/local/lib
cp -f ./driver/lib/libvega3311_fpga_drv.so /usr/local/lib
cp -f ./sdk/vega3311_capture_api/lib/libvega3311_capture.a /usr/local/lib
cp -f ./sdk/vega3311_capture_api/lib/libvega3311_capture.so /usr/local/lib
cp -f ./sdk/vega3311_transmit_api/lib/libvega3311_transmit.a /usr/local/lib
cp -f ./sdk/vega3311_transmit_api/lib/libvega3311_transmit.so /usr/local/lib
cp -f ./driver/include/vega3311_fpga_drv.h /usr/local/include/libvega_capture_api
cp -f ./sdk/vega3311_capture_api/include/VEGA3311_capture.h /usr/local/include/libvega_capture_api
cp -f ./sdk/vega3311_capture_api/include/VEGA3311_cap_types.h /usr/local/include/libvega_capture_api
cp -f ./sdk/vega3311_transmit_api/include/VEGA3311_transmit.h /usr/local/include/libvega_transmit_api
cp -f ./sdk/vega3311_transmit_api/include/VEGA3311_trans_types.h /usr/local/include/libvega_transmit_api
Clean the package...
done.
Making and loading device driver...done.
Installation completed.

[root@mediaflow-bioslab scratch]# lspci -vs 09:00.0
09:00.0 Multimedia controller: Advantech Co. Ltd Device 004b (rev 01)
	Subsystem: Advantech Co. Ltd Device 004b
	Physical Slot: 1
	Flags: bus master, fast devsel, latency 0, IRQ 18
	Memory at de800000 (32-bit, non-prefetchable) [size=128K]
	Memory at de400000 (32-bit, non-prefetchable) [size=4M]
	Memory at 3fff800000 (64-bit, prefetchable) [size=4M]
	Capabilities: [80] Power Management version 3
	Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-
	Capabilities: [c0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [340] Vendor Specific Information: ID=0001 Rev=0 Len=02c <?>
	Kernel driver in use: vega3311_fpga
	Kernel modules: vega3311_fpga

To verify the drivers are installed correctly chyeck the following command 

[root@mediaflow-bioslab scratch]# vega3311_cap_app list_dev
DRV INF: SF: Got idcode 0xc2 0x20 0x1a
Detected 1 chip(s).
==============================================
Chip: 0
	Device node name: /dev/vega3311_fpga_0
	PCI Bridge 0x08
	PCI Bus:Slot.Func 09:00.0
	Vendor ID: 0x13fe	Device ID: 0x004b
	Subsystem VendorID: 0x13fe	Subsystem DevID: 0x004b
	Serial Number: LKD0114896	Board ID: 9792A33111E
	Irq: 18
	BAR Configuration:
	Start              | Length
	0x00000000de800000 | 000131072
	0x00000000de400000 | 004194304
==============================================
[root@mediaflow-bioslab scratch]# 
[root@mediaflow-bioslab scratch]# vega3311_fpgautil list
	[Card 0] Device id 0, PCIe bridge 8

Compling drivers for the fgpa (history back up from Roy VM) 

1  sudo su -
    2  exit
    3  lspci
    4  lspci -vvs 05.0
    5  lspci -vvvs 05.0
    6  sudo lspci -vvvs 05.0
    7  sudo lspci -vvvs 06.0
    8  ls
    9  tar -jxvf vega_bqb_trc_release_1_5_1_12817_x64.tar.bz2
   10  sudo yum list bzip2
   11  sudo yum install bzip2
   12  tar -jxvf vega_bqb_trc_release_1_5_1_12817_x64.tar.bz2
   13  ls
   14  cd vega_bqb_trc_release_1_5_1
   15  ls
   16  cd vega_bqb_codec_x64/01_HOST/realtime_codec_soft
   17  ls
   18  sudo ./install.sh
   19  uname -r
   20  sudo yum install kernel-headers-3.10.0-862.2.3.el7.x86_64
   21  sudo yum install kernel-headers-3.10.0-862.2.3
   22  sudo yum list kernel-headers
   23  sudo yum list kernel-headers --showduplicates
   24  uname -r
   25  sudo yum update
   26  sudo systemctl reboot
   27  uname -r
   28  cd vega_bqb_trc_release_1_5_1
   29  cd vega_bqb_codec_x64/01_HOST/realtime_codec_soft
   30  sudo ./install.sh
   31  sudo dmesg | grep m30trc
   32  vega_bqb_vtrc version
   33  ls /dev
   34  lsmob
   35  lsmod
   36  ls
   37  sudo ./uninstall.sh
   38  ls
   39  lsmod
   40  cd 
   41  #cd ~
   42  ls
   43  rm -r vega_bqb_trc_release_1_5_1
   44  ls
   45  rm vega_bqb_trc_release_1_5_1_12817_x64.tar.bz2
   46  ls
   47  tar -jxvf vega_bqb_release_1_3_1_12729_x64.tar.bz2
   48  ls
   49  cd vega_bqb_release_1_3_1
   50  cd vega_bqb_codec_x64/01_HOST/realtime_codec_soft/
   51  sudo ./install.sh
   52  sudo dmesg | grep "m30*"
   53  cd ~
   54  cd vega_bqb_release_1_3_1
   55  cd vega3311_fpga_x64/
   56  ls
   57  sudo ./install.sh
   58  sudo dmesg | grep "vega3311*"
   59  vega_bqb_sysctl set_enc --device 0
   60  vega_bqb_sysctl version
   61  vega_bqb_sysctl info
   62  vega_bqb_venc version
   63  sudo apt-get install mercurial cmake cmake-curses-gui build-essential yasm
   64  sudo yum install mercurial cmake cmake-gui
   65  sudo yum groupinstall "Development Tools"
   66  sudo yum install -y epel-release
   67  sudo yum --disablerepo="*" --enablerepo="epel" install -y yasm
   68  sudo yum install -y SDL-devel
   69  cd ~
   70  wget http://sourceforge.net/projects/boost/files/boost/1.54.0/boost_1_54_0.tar.gz
   71  sudo yum install wget
   72  wget http://sourceforge.net/projects/boost/files/boost/1.54.0/boost_1_54_0.tar.gz
   73  tar -xzvf boost_1_54_0.tar.gz
   74  cd ~/boost_1_54_0
   75  ./bootstrap.sh
   76  sudo ./bjam --layout=system --toolset=gcc install
   77  cd ~/vega_bqb_release_1_3_1/ffmpeg-3.4.1/vega_bqb_avc/build/linux
   78  PATH="/usr/local/bin:$PATH" cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX="/usr/local" -DENABLE_SHARED:bool=on -DENABLE_PIC:bool=on ../../source
   79  make
   80  sudo make install
   81  cd ~/vega_bqb_release_1_3_1/ffmpeg-3.4.1/vega_bqb_hevc/build/linux
   82  PATH="/usr/local/bin:$PATH" cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX="/usr/local" -DENABLE_SHARED:bool=on -DENABLE_PIC:bool=on ../../source
   83  make
   84  sudo make install
   85  cd ~/vega_bqb_release_1_3_1/ffmpeg-3.4.1/vega3311_ff_capture/build/linux
   86  PATH="/usr/local/bin:$PATH" cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX="/usr/local" -DENABLE_SHARED:bool=on -DENABLE_PIC:bool=on ../../source
   87  make
   88  sudo make install
   89  cd ~/vega_bqb_release_1_3_1/ffmpeg-3.4.1/ffmpeg_source
   90  ls /usr/local/lib
   91  ls /usr/local/lib/pkgconfig
   92  PATH="/usr/local/bin:$PATH" PKG_CONFIG_PATH="/usr/local/lib/pkgconfig"
   93  ./configure --enable-shared --enable-libvegabqbavc --enable-libvegabqbhevc --enable-nonfree
   94  echo $PKG_CONFIG_PATH
   95  pkg-config --help
   96  pkg-config --exists --print-errors libvegabqbavc
   97  find /usr -name libvegabqbavc.pc
   98  find /usr -name vega_bqb_hevc.pc
   99  find ~/ -name libvegabqbavc.pc
  100  find ~/ -name vega_bqb_hevc.pc
  101  less /home/centos/vega_bqb_release_1_3_1/ffmpeg-3.4.1/vega_bqb_hevc/build/linux/vega_bqb_hevc.pc
  102  make
  103  sudo make install
  104  cd ~
  105  sudo ldconfig
  106  ffmpeg
  107  ls mnt
  108  ls /mnt
  109  ls /media
  110  df -h
  111  lspci -t
  112  lspci -vmms 05.0
  113  lspci -vmms 06.0
  114  vega3311_cap_app qsrc 0 0
  115  vega3311_cap_app qsrc 0 1
  116  vega3311_cap_app qsrc 0 2
  117  vega3311_cap_app qsrc 0 3
  118  vega3311_cap_app qsrc 1 0
  119  vega3311_cap_config 0 0 -input_mode 4ch_fhd
  120  vega3311_cap_config 0 0 -input_mode 4ch
  121  vega3311_cap_app qsrc 0 0
  122  vega3311_cap_app qsrc 0 1
  123  vega3311_cap_config 0 0 -image_format nv12
  124  vega3311_cap_config -h
  125  vega3311_cap_app -h
  126  vega3311_cap_app reset 0 0
  127  vega3311_cap_app list_dev
  128  systemctl reboot
  129  sudo systemctl reboot
  130  vega3311_cap_config 0 0 -image_format nv16
  131  ls
  132  cd vega_bqb_release_1_3_1
  133  ls
  134  cd vega3311_fpga_x64/tools/fpga_fw/
  135  ls
  136  vega3311_cap_app qsrc 0 0
  137  vega_bqb_venc version
  138  vega_bqb_sysctl set_dec --device 0
  139  vega_bqb_venc version
  140  ls
  141  cd ..
  142  ls
  143  cd vega_bqb_codec_x64
  144  ls
  145  cd 02_M30
  146  ls
  147  cd 02_tools
  148  ls
  149  mv config_system_dec.txt ../
  150  ls
  151  sudo ./flash_write_all.sh 0
  152  sudo systemctl reboot
  153  vega_bqb_venc version
  154  sudo lspci -vvvs 05.0
  155  sudo lspci -vvvvs 05.0
  156  sudo lspci -vvvvs 06.0
  157  ls
  158  cd vega_bqb_release_1_3_1
  159  ls
  160  cd vega3311_fpga_x64/tools/fpga_fw
  161  ls
  162  cd ..
  163  ls
  164  cd vega3311_fpgautil
  165  ls
  166  cd ..
  167  cd vega3311_fpga_x64/tools/fpga_fw
  168  ls
  169  vega3311_fpgautil fw_update 0 vega3311_0_6C_20181029_3G12G_wt.bin
  170  lspci
  171  lspci -m
  172  lspci -mm
  173  lspci -v
  174  free -h
  175  vega3311_cap_app qsrc 0 0
  176  vega3311_cap_app list_dev
  177  sudo dmesg | grep -e BAR
  178  sudo dmesg | grep -e vega3311_fpga
  179  sudo lspci -vvvs 05.0
  180  ls /sys
  181  ls /sys/dev
  182  ls /sys/device
  183  sudo ls /sys/device
  184  sudo ls /sys/devices
  185  sudo ls /sys/bus
  186  ls /sys/bus
  187  ls /sys/bus/pci
  188  ls /sys/bus/pci/devices
  189  ls /sys/bus/pci/devices/0000:00:05.0
  190  ls /sys/bus/pci/devices/0000:00:05.0/vega3311_fpga
  191  ls /sys/bus/pci/devices/0000:00:05.0/vega3311_fpga/vega3311_fpga_0
  192  ls /sys/bus/pci/devices/0000:00:05.0
  193  ls -l /sys/bus/pci/devices/0000:00:05.0
  194  ls -l /sys/bus/pci/devices/0000:00:05.0/
  195  sudo less -l /sys/bus/pci/devices/0000:00:05.0/max_link_width
  196  sudo less -l /sys/bus/pci/devices/0000:00:05.0/max_link_speed
  197  sudo less -l /sys/bus/pci/devices/0000:00:05.0/irq
  198  ls -l /sys/bus/pci/devices/0000:00:05.0/firmware_node
  199  ls -l /sys/bus/pci/devices/0000:00:05.0/firmware_node/
  200  sudo less /sys/bus/pci/devices/0000:00:05.0/firmware_node/adr
  201  cd /~
  202  cd ~
  203  ls
  204  vega_bqb_sysctl set_enc –device 0
  205  vega_bqb_sysctl set_enc --device 0
  206  vega_bqb_venc version
  207  vega_bqb_venc reset
  208  ls /dev
  209  sudo systemctl reboot
  210  vega_bqb_venc version
  211  sudo systemctl poweroff
  212  vega_bqb_venc version
  213  vega_bqb_sysctl set_enc --device 0
  214  vega_bqb_venc version
  215  vega_bqb_sysctl set_dec --device 0
  216  vega_bqb_venc version
  217  ls
  218  cd vega_bqb_release_1_3_1
  219  cd vega_bqb_codec_x64/02_M30
  220  ls
  221  mv config_system_dec.txt ./01_M30_firm  02_tools
  222  ls
  223  ls 02_tools
  224  cd 02_tools
  225  ls
  226  mv 01_M30_firm ../
  227  ls
  228  ls ../
  229  mv config_system.txt ../
  230  ls
  231  sudo ./flash_write_all.sh 0
  232  ls
  233  less flash_write_mcpu_fw.sh
  234  sudo systemctl reboot
  235  vega_bqb_sysctl set_enc --device 0
  236  vega_bqb_venc version
  237  vega_bqb_sysctl set_dec --device 0
  238  vega_bqb_venc version
  239  ls
  240  cd vega_bqb_release_1_3_1
  241  ls
  242  cd vega_bqb_codec_x64/02_M30
  243  ls
  244  mv config_system.txt ./02_tools
  245  ls
  246  cd 02_tools
  247  ls
  248  sudo ./flash_write_all.sh 0
  249  sudo poweroff
  250  vega_bqb_sysctl set_enc --device 0
  251  vega_bqb_venc version
  252  vega3311_cap_app list_dev
  253  logout
  254  yum list tmux
  255  yum list screen
  256  vega3311_cap_app list_dev
  257  sudo vega3311_cap_app list_dev
  258  sudo /usr/local/bin/vega3311_cap_app list_dev
  259  lsmod
  260  free -h
  261  df -h
  262  ls
  263  ffmpeg -vsync vfr -f mpegts -i ~/Videos/bbb.ts -c:v libvegabqbavc -b:v 4000k -vegabqbavc-params profile=high:level-idc=51:open-gop=2:keyint=30:scenecut=1:bframes=2:deblock=1:rate-ctrl=cbr:bitrate=4000:sar=1:device=0:channel=0:input-mode=0 -c:a aac ~/Videos/bbb.mp4
  264  alias
  265  logout
  266  lspci
  267  sudo lspci -vvv -s 05.0
  268  sudo poweroff
  269  lspci
  270  sudo lspci -vvv -s 05.0
  271  vega3311_cap_app qsrc 0 0
  272  vega_bqb_sysctl set_enc --device 0
  273  vega_bqb_venc version
  274  vega_bqb_sysctl set_dec --device 0
  275  vega3311_cap_app list_dev
  276  sudo lspci -vvv -s 05.0
  277  ls
  278  cd vega_bqb_release_1_3_1
  279  ls
  280  cd vega3311_fpga_x64
  281  ls
  282  cd ../vega3310_fpga_x64
  283  ls
  284  cd ..
  285  cd vega3311_fpga_x64
  286  less install.log
  287  modinfo vega3311_fpga
  288  lspci
  289  lspci -n -s 05.0
  290  sudo lspci -vvvv -s 5.0
  291  ls /sys/bus/pci
  292  ls /sys/bus/pci/devices
  293  ls /sys/bus/pci/devices/0000:00:05.0
  294  ls -l
  295  ls -l /sys/bus/pci/devices/0000:00:05.0/
  296  ls -l /sys/bus/pci/devices/0000:00:06.0/
  297  ls -l /sys/bus/pci/devices/0000:00:05.0/firmware_node/
  298  less /sys/bus/pci/devices/0000:00:05.0/irq
  299  less /sys/bus/pci/devices/0000:00:06.0/irq
  300  sudo cat /sys/bus/pci/devices/0000:00:05.0/irq
  301  sudo cat /sys/bus/pci/devices/0000:00:06.0/irq
  302  vega_bqb_sysctl set_enc --device 0
  303  vega_bqb_venc version
  304  vega_bqb_venc version
  305  stty -ixon
  306  sudo lspci -vvv -s 05.0
  307  sudo cat /proc/interrupts
  308  sudo lspci -vvv -s 06.0
  309  logout
  310  sudo cat /proc/interrupts
  311  vega3311_cap_app qsrc 0 0
  312  sudo cat /proc/interrupts
  313  sudo journalctl -k | grep remap
  314  sudo poweroff
  315  sudo cat /proc/interrupts
  316  vega3311_cap_app qsrc 0 0
  317  sudo cat /proc/interrupts
  318  sudo dmesg | grep less
  319  sudo dmesg | less
  320  sudo cat /proc/interrupts
  321  sudo dmesg | less
  322  sudo poweroff
  323  sudo dmesg | less
  324  ls
  325  cd vega_bqb_release_1_3_1
  326  ls
  327  cd vega3311_fpga_x64
  328  ls
  329  less install.log
  330  sudo ./uninstall.sh
  331  lsmod
  332  sudo dmesg | less
  333  sudo cat /proc/interrupts
  334  sudo lspci -vvv -s 05.0
  335  ls
  336  sudo ./uninstall.sh
  337  sudo ./install.sh
  338  sudo dmesg | less
  339  sudo cat /proc/interrupts
  340  vega3311_cap_app list_dev
  341  sudo cat /proc/interrupts
  342  sudo dmesg | less
  343  sudo cat /proc/interrupts
  344  vega3311_cap_app list_dev
  345  sudo cat /proc/interrupts
  346  vega3311_cap_app list_dev
  347  sudo cat /proc/interrupts
  348  vega_bqb_sysctl set_enc --device 0
  349  sudo cat /proc/interrupts
  350  vega_bqb_venc version
  351  sudo cat /proc/interrupts
  352  ls -l /sys/bus/pci/devices/0000:00:05.0/
  353  cat /sys/bus/pci/devices/0000:00:05.0/enable
  354  cat /sys/bus/pci/devices/0000:00:05.0/msi_bus
  355  ls -l /sys/bus/pci/devices/0000:00:05.0/msi_irqs/
  356  cat /sys/bus/pci/devices/0000:00:05.0/msi_irqs/29
  357  cat /sys/bus/pci/devices/0000:00:05.0/msi_irqs/30
  358  cat /sys/bus/pci/devices/0000:00:05.0/msi_irqs/37
  359  ls
  360  ffmpeg
  361  logout
  362  ls
  363  cd vega_bqb_release_1_3_1
  364  ls
  365  cd vega3311_ff_capture/build/linux
  366  ls
  367  cd ffmpeg-3.4.1
  368  cd vega3311_ff_capture/build/linux
  369  ls
  370  PATH="/usr/local/bin:$PATH" cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX="/usr/local" -DENABLE_SHARED:bool=on -DENABLE_PIC:bool=on ../../source
  371  make
  372  sudo make install
  373  cd ~/vega_bqb_release_1_3_1
  374  cd ffmpeg-3.4.1
  375  cd ffmpeg_source
  376  ls
  377  PATH="/usr/local/bin:$PATH" PKG_CONFIG_PATH="/usr/local/lib/pkgconfig"
  378  ./configure --enable-shared --enable-libvegabqbavc --enable-libvegabqbhevc --enable-libvega3311-capture --enable-nonfree
  379  logout
  380  dmesg | grep -i 05.0
  381  sudo cat /proc/interrupts
  382  sudo poweroff
  383  sudo cat /proc/interrupts
  384  lspci -vs 05.0
  385  sudolspci -vs 05.0
  386  sudo lspci -vs 05.0
  387  sudo lspci -vv | grep -i "IRQ 10"
  388  sudo lspci -vs 06.0
  389  sudo lspci -vv | grep -i "IRQ 50"
  390  logout
  391  sudo su -
  392  ls
  393  vi Videos/
  394  cd Videos/
  395  ls
  396  cd ..
  397  ls
  398  history
Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=VScaler:_Advantech_PCIe_passthrough&oldid=29804"