qemu-kvm-pcie-passthrough

Install KVM

• https://wiki.archlinux.org/title/KVM
• https://sysguides.com/install-kvm-on-linux

Looking Glass

Module

https://looking-glass.io/docs/B7/ivshmem_kvmfr/

Client

https://looking-glass.io/docs/B7/install_client/

cgroup_device_acl

uncomment cgroup_device_acl block in /etc/libvirt/qemu.conf and add /dev/kmvfr0

PCIE Passthrough

• https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF

CPU Isolation

• Check CPU: lscpu -e

❯ lscpu -e
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ   MINMHZ       MHZ
  0    0      0    0 0:0:0:0          yes 4853.5850 555.5310 3790.8679
  1    0      0    1 1:1:1:0          yes 4853.5850 555.5310 1754.3080
  2    0      0    2 2:2:2:0          yes 4853.5850 555.5310 3359.5859
  3    0      0    3 3:3:3:0          yes 4853.5850 555.5310 1754.3080
  4    0      0    4 4:4:4:0          yes 4853.5850 555.5310 1754.3080
  5    0      0    5 5:5:5:0          yes 4853.5850 555.5310 1754.3080
  6    0      0    6 6:6:6:0          yes 4853.5850 555.5310 1754.3080
  7    0      0    7 7:7:7:0          yes 4853.5850 555.5310 3599.6721
  8    0      0    0 0:0:0:0          yes 4853.5850 555.5310 3356.3181
  9    0      0    1 1:1:1:0          yes 4853.5850 555.5310 1754.3080
 10    0      0    2 2:2:2:0          yes 4853.5850 555.5310 1754.3080
 11    0      0    3 3:3:3:0          yes 4853.5850 555.5310 4217.1719
 12    0      0    4 4:4:4:0          yes 4853.5850 555.5310 1754.3080
 13    0      0    5 5:5:5:0          yes 4853.5850 555.5310 1754.3080
 14    0      0    6 6:6:6:0          yes 4853.5850 555.5310 1754.3080
 15    0      0    7 7:7:7:0          yes 4853.5850 555.5310 3534.9700

• create hook script under /etc/libvirt/hooks/qemu.d
• example nvim vm-win

#!/bin/sh

command=$2

if [ "$command" = "started" ]; then
    systemctl set-property --runtime -- system.slice AllowedCPUs=0,1,2,3,8,9,10,11
    systemctl set-property --runtime -- user.slice AllowedCPUs=0,1,2,3,8,9,10,11
    systemctl set-property --runtime -- init.scope AllowedCPUs=0,1,2,3,8,9,10,11
elif [ "$command" = "release" ]; then
    systemctl set-property --runtime -- system.slice AllowedCPUs=0-15
    systemctl set-property --runtime -- user.slice AllowedCPUs=0-15
    systemctl set-property --runtime -- init.scope AllowedCPUs=0-15
fi

• make executable: chmod +x vm-win
• check if it worked after launching the vm: systemctl show user.slice | grep AllowedCPUs

Links

• https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Isolating_pinned_CPUs
• https://www.reddit.com/r/VFIO/comments/nv0cgo/help_with_moving_from_isolcpus_to_vfioisolate/
• https://github.com/spheenik/vfio-isolate
• https://pypi.org/project/vfio-isolate/

CPU Pinning

• edit xml: virsh edit vm-win

  <vcpu placement='static'>8</vcpu>
  <iothreads>1</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='12'/>
    <vcpupin vcpu='2' cpuset='5'/>
    <vcpupin vcpu='3' cpuset='13'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='16'/>
    <vcpupin vcpu='6' cpuset='17'/>
    <vcpupin vcpu='7' cpuset='15'/>
    <emulatorpin cpuset='0,8'/>
    <iothreadpin iothread='1' cpuset='0,8'/>
  </cputune>

• Windwos expects vcpu0 and vcpu 1 is a pair, so vcpu0 and vcpu1 need to be on the same core
• emulatorpin: pin emulation to cpus not in vm
• iothreadpin: offload disk/networking from vm
  • make sure to add iothread='1' to disk driver:

<disk type='file' device='disk'>
<driver name='qemu' type='raw' iothread='1'/>

• Core 0 rule: never pin to vm, default core in linux for e.g. kernel tasks

Links

• https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Performance_tuning

Hugepages

• edit vm:

<memoryBacking>
  <hugepages/>
  <locked/>
</memoryBacking>

• edit /etc/libvirt/hooks/qemu.d/vm-win

#!/bin/sh

command=$2

if [ "$command" = "started" ]; then
    systemctl set-property --runtime -- system.slice AllowedCPUs=0,1,2,3,8,9,10,11
    systemctl set-property --runtime -- user.slice AllowedCPUs=0,1,2,3,8,9,10,11
    systemctl set-property --runtime -- init.scope AllowedCPUs=0,1,2,3,8,9,10,11
    echo 6144 > /proc/sys/vm/nr_hugepages
elif [ "$command" = "release" ]; then
    systemctl set-property --runtime -- system.slice AllowedCPUs=0-15
    systemctl set-property --runtime -- user.slice AllowedCPUs=0-15
    systemctl set-property --runtime -- init.scope AllowedCPUs=0-15
    echo 0 > /proc/sys/vm/nr_hugepages
fi

Links

• https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Static_huge_pages

Windows 11 VM Tuning

Nvidia Control Panel

• Modus für geringe Latenz: Ultra
• Energieverwaltungsmodus: Maximale Leistung bevorzugen
• Nvidia Audio Treiber in Device Manager deaktivieren

Taming Wdf01000.sys (USB/Framework)

This driver usually spikes because a USB device (like your audio interface) or the USB controller is being "polled" too aggressively or entering a low-power state.

• Disable USB Selective Suspend: Go to Control Panel > Power Options > Change plan settings > Change advanced power settings. Under USB settings, set "USB selective suspend setting" to Disabled.
• Power Management Tweak: In Device Manager, go to USB Serial Bus controllers. Right-click every "USB Root Hub" and "Generic USB Hub," go to Properties > Power Management, and uncheck "Allow the computer to turn off this device to save power".

Tweaks

• msi utility v3 (message signaled interrupts)
  • https://github.com/spddl/GoInterruptPolicy
• hyper-v tuning
  • https://www.androidsdream.net/posts/kvm-windows11-virtual-machine-fine-tuning/
  • https://sysguides.com/install-windows-11-on-kvm#111-enable-hyperv-enlightenments-for-better-performance
• windows 11 vm tuning
  • https://sysguides.com/install-windows-11-on-kvm#section-4-optimize-windows-11-for-better-performance

Images

images in var/lib/libvirt/images need to be owned by libvirt-qemu