KVM: Installation and operation

Check CPU supports KVM

egrep '(vmx|svm)' --color=always /proc/cpuinfo

Install KVM

# on RHEL 5
yum install virt-viewer virt-manager
yum install kvm kvm-qemu-img
OR
yum groupinstall KVM

# on RHEL 6
yum groupinstall  Virtualisation 'Virtualization Client' 'Virtualization Platform' 'Virtualization Tools'

Install the modules

modprobe kvm-intel

Start libvirtd

/etc/init.d/libvirtd restart

Load up virtual manager

virt-manager

Setup Bridged Networking

By Default KVM uses NAT (hosts can see outside world but world cannot see the hosts). To setup bridged networking:

• Disable NetworkManager

 # chkconfig NetworkManager off
 # chkconfig network on
 # service NetworkManager stop
 # service network start

• Edit the file: /etc/sysconfig/network-scripts/ifcfg-eth0

 DEVICE="eth0"
 ONBOOT="yes"
 BRIDGE=br0

• Create the bridge script/file: /etc/sysconfig/network-scripts/ifcfg-br0

 DEVICE=br0
 TYPE=Bridge
 IPADDR=172.28.10.58
 NETMASK=255.255.0.0
 NETWORK=172.28.0.0
 DNS1=172.28.1.3
 ONBOOT=yes
 DELAY=0

• Configure iptables to allow traffic be forwarded across the bridge

 # iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT
 # service iptables save
 # service iptables restart

• Reload libvirtd

 # service libvirtd reload

• Verify it's working fine:

 # brctl show 
 bridge name	bridge id		STP enabled	interfaces
 br0		8000.0025902efb56	no		eth0
                                                       vnet2
 virbr0	8000.525400098976	yes		virbr0-nic
                                                       vnet0

Create the image file

In RHEL6, use virt-install: (dont use any files in /root)

[root@kvm kvms]$ virt-install \
                 --name=guest1-sl6-64 \
                 --disk path=/vm/guest1-rhel6-64,size=10,sparse=false \
                 --vnc \
                 --vcpus=2 \
                 --ram=2048 \
                 --cdrom=/vm/SL-61-x86_64-2011-07-27-Everything-DVD1.iso \
                 --network bridge=br0 \
                 --os-type=linux \
                 --os-variant=rhel6

Starting install...
Creating storage file guest1-rhel6-64    8% [==-                           ]  17 MB/s | 873 MB     09:13 ETA

Notes from RHEL5 on creating a qcow filesystem (virt-install runs this under the covers)

dd if=/dev/zero of=/home/viglen/vm_winxp.img bs=1024k count=12000
OR
qemu-img create -f qcow windows.img 12G

Creating files in /root resulted in the following error:

Starting install...
ERROR    internal error process exited while connecting to monitor: char device redirected to /dev/pts/1
qemu: could not open disk image /root/scratch/SL-61-x86_64-2011-07-27-Everything-DVD1.iso: Permission denied

• Change CDs using virt-manger -> Open Guest -> Show Virtual Hardware -> CDROM

Start qemu

/usr/libexec/qemu-kvm -hda vig_xp.img  -cdrom vig_xp.iso -m 512 -boot d

Other Handy Bits

VNC / Args

CLI Args:
-alt-grab - makes ctrl+alt+shift the key escape rather than ctrl+alt (which makes it impossible for windows logins)
on vncviewer, you can also hit f8 to bring up a menu which will allow
-smp n (number of cores to allocate)
-m 8192 (amount of memory to give the machine)
-net nic -net nic (give the system two nics)

Mounting NFS in VM

ERROR: mountd: refused mount nfs illegal port

In case you’re getting error while mounting a nfs share, and logs on server showing:

Mar 24 10:44:57 localhost mountd[8344]: refused mount request from 192.168.1.2 for /dump (/dump): illegal port 52213

Add insecure option in your exports file:

    /dump  *(ro,sync,no_root_squash,insecure)

    # And do 
    exportfs -r  # or reload/restart nfs service.

Grow a KVM Disk Image (RAW)

E.G to grow a KVM image by 10G (Perhaps this could be a hell of a lot easier with LVM)

• Create additional space as a raw qemu-img
• cat the two files together
• Boot gparted-live to resize partitions/filesystem

[root@kvm vm]$ qemu-img create -f raw guest1-rhel6-64-largerdisk 10G
Formatting 'guest1-rhel6-64-largerdisk', fmt=raw size=10737418240 
[root@kvm vm]$ ll
total 17611676
-rwxrwxrwx. 1 root root 10737418240 Sep 27 14:34 guest1-rhel6-64
-rwxr-xr-x  1 root root 10737418240 Sep 27 13:14 guest1-rhel6-64-clone
'-rw-r--r--  1 root root 10737418240 Sep 27 14:37 guest1-rhel6-64-largerdisk'
[root@kvm vm]$ cat guest1-rhel6-64 guest1-rhel6-64-largerdisk > sl6-30g-puppetcobbler.kvm
[root@kvm vm]$ ll
total 38583200
-rwxrwxrwx. 1 root root 10737418240 Sep 27 14:34 guest1-rhel6-64
-rwxr-xr-x  1 root root 10737418240 Sep 27 13:14 guest1-rhel6-64-clone
-rw-r--r--  1 root root 10737418240 Sep 27 14:37 guest1-rhel6-64-largerdisk
'-rw-r--r--  1 root root 21474836480 Sep 27 14:42 sl6-30g-puppetcobbler.kvm'

Just as easy to convert a raw image to qcow2 and then resize that(qemu-img resize [vmname.qcow2] +100G). Use LVM as below to grow the FS.

Now boot up a ubuntu (or other) live cd (in virt-manager: add hardware -> storage -> select managed.. -> device type -> IDE CDrom)

 Install lvm tools: apt-get install lvm2 
 Fdisk create new partition: 
 fdisk /dev/sda
 delete lvm partition (d -> 2 in this instance)
 create lvm partition (same cylinder start, new end pick the default which will be the last cylinder) (n -> p -> start / finish)
 change type of new partition to lvm (t -> 8e)

Use LVM to grow the volume

root@ubuntu:~# pvdisplay /dev/sda2
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               vg_puppetmaster
  PV Size               9.51 GiB / not usable 3.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              2434
  Free PE               0
  Allocated PE          2434
  PV UUID               LBxPCN-RL3d-rd7d-qEcr-M5Fq-GfMF-ttiPPd
   
root@ubuntu:~# partprobe
Warning: Unable to open /dev/sr0 read-write (Read-only file system).  /dev/sr0 has been opened read-only.
root@ubuntu:~# pvresize /dev/sda2 
  Physical volume "/dev/sda2" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized
root@ubuntu:~# pvdisplay /dev/sda2
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               vg_puppetmaster
  PV Size               19.50 GiB / not usable 3.46 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              4992
  Free PE               2558
  Allocated PE          2434
  PV UUID               LBxPCN-RL3d-rd7d-qEcr-M5Fq-GfMF-ttiPPd
   

t@ubuntu:~# vgchange -a y vg_puppetmaster
  2 logical volume(s) in volume group "vg_puppetmaster" now active
root@ubuntu:~# vgdisplay vg_puppetmaster
  --- Volume group ---
  VG Name               vg_puppetmaster
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               19.50 GiB
  PE Size               4.00 MiB
  Total PE              4992
  Alloc PE / Size       2434 / 9.51 GiB
  Free  PE / Size       2558 / 9.99 GiB
  VG UUID               Bu7xaM-OQrj-aZYV-gwY8-MvXg-Az1m-8wZh6p
   
root@ubuntu:~# lvextend -L +10G /dev/vg_puppetmaster/lv_root 
  Extending logical volume lv_root to 15.57 GiB
  Insufficient free space: 2560 extents needed, but only 2558 available

root@ubuntu:~# lvextend -L +9.9G /dev/vg_puppetmaster/lv_root 
  Rounding up size to full physical extent 9.90 GiB
  Extending logical volume lv_root to 15.47 GiB
  Logical volume lv_root successfully resized

root@ubuntu:~# resize2fs /dev/vg_puppetmaster/lv_root 
resize2fs 1.41.14 (22-Dec-2010)
Please run 'e2fsck -f /dev/vg_puppetmaster/lv_root' first.

root@ubuntu:~# e2fsck -f /dev/vg_puppetmaster/lv_root 
e2fsck 1.41.14 (22-Dec-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/vg_puppetmaster/lv_root: 50681/365760 files (0.1% non-contiguous), 448889/1460224 blocks
root@ubuntu:~# resize2fs /dev/vg_puppetmaster/lv_root 
resize2fs 1.41.14 (22-Dec-2010)
Resizing the filesystem on /dev/vg_puppetmaster/lv_root to 4056064 (4k) blocks.
The filesystem on /dev/vg_puppetmaster/lv_root is now 4056064 blocks long.

root@ubuntu:~# lvscan 
  ACTIVE            '/dev/vg_puppetmaster/lv_root' [15.47 GiB] inherit
  ACTIVE            '/dev/vg_puppetmaster/lv_swap' [3.94 GiB] inherit

Use old VM img with virt-manager

You'll need to use virt-manager to import an old file. Point this towards an existing raw VM and import. Import didnt add a CD drive, so used the command:

    
[root@kvm vm]$ /usr/libexec/qemu-kvm -hda ./sl6-30g-puppetcobbler.kvm \
      -cdrom /mnt/vstor/gparted-live/gparted-live-0.7.1-1.iso -m 512 -boot d
Using CPU model "cpu64-rhel6"
VNC server running on `::1:5900'

# Then vmcviewer on another terminal
vncviewer localhost:5900

Copying VMs udev renames network

• udev seems to be adding lines in /etc/udev/70-persistent-net.rules
• Remove entries and set MAC correctly
• Also had to update the files in /etc/sysconfig/network-scripts/ifcfg-eth0 to remove/change the MAC address
• All this could probably have been avoided if i set the MAC correctly when setting up the machine? (not tested)

Disable DHCP on KVM Server

In this example i wanted to run a virtual DHCP server on the KVM system. The KVM server was getting in the way as it run a DHCP service also. To disable, remove the <dhcp> tags from the file /var/lib/libvirt/network/default.xml

Change from:

<network>
  <name>default</name>
  <uuid>319d4ae5-4b6e-433c-832f-ad20269ed495</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0' />
  <mac address='52:54:00:09:89:76'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254' />
    </dhcp>
  </ip>
</network>

Change to:

<network>
  <name>default</name>
  <uuid>319d4ae5-4b6e-433c-832f-ad20269ed495</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0' />
  <mac address='52:54:00:09:89:76'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
  </ip>
</network>

And then restart libvirtd and dnsmasq:

  service libvirtd restart
  service dnsmasq restart

Important Config Files

Important Configuration And Log Files (Directories) Location

The following files are required to manage and debug KVM problems:

  1. /etc/libvirt/ - Main configuration directory.
  2. /etc/libvirt/qemu/ - Virtual machine configuration directory. All xml files regarding VMs are stored here. You can edit them manually or via virt-manager.
  3. /etc/libvirt/qemu/networks/ - Networking for your KVM including default NAT. NAT is only recommended for small setup or desktops. I strongly suggest you use bridged based networking for performance.
  4. /etc/libvirt/qemu/networks/default.xml - The default NAT configuration used by NAT device virbr0.
  5. /var/log/libvirt/ - The default log file directory. All VM specific logs files are stored here.
  6. /etc/libvirt/libvirtd.conf - Master libvirtd configuration file.
  7. /etc/libvirt/qemu.conf - Master configuration file for the QEMU driver.

Editing the VM XML

libvirt stores it's configuration as xml in '/etc/libvirt/qemu'. The xml is easy to understand, and is similar to VMware *.vmx files. While it is possible to edit these files in place and restart libvirt-bin for the changes to take affect, the recommended method for modifying the attributes of a virtual machine is via virsh (or virt-manager, if it supports changing the hardware you want to change). The concept is simple:

  1. export (aka 'dump') the xml of the virtual machine you want to edit
  2. edit the xml
  3. import (aka 'define') the xml 

For example, to edit the machine named 'foo' (you can get a list of your machines with 'virsh list --all'), do:

  $ virsh dumpxml foo > /tmp/foo.xml
  (edit /tmp/foo.xml as needed)
  $ virsh define /tmp/foo.xml