Nvidia GPU passthrough fail with code 43

Question

I'm currently trying to pass a nvidia GPU to Windows 10 guest using qemu 2.5 and libvirt 1.3.5.

I see "Error 43" on Nvidia GPU in Device Manager.

I had tried to hide the hypervisor by adding "kvm=off" and "hv_vendor_id=123456780ab", but it does not work for me. I searched in google and people solved the problem in this way.

And I also saw Virtual Machine : Yes in task manager.

1. Did I use in the wrong way? I can pass a AMD gpu to windows guest(AMD does not check the kvm virtualization).

2. Can I spoof nvidia in other way?

My system information:

#uname -a
Linux ns.mqcache.net 4.2.0-1.el7.elrepo.x86_64 #1 SMP Sun Aug 30 21:25:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

#/root/qemu25/qemu/x86_64-softmmu/qemu-system-x86_64 --version
QEMU emulator version 2.5.1.1, Copyright (c) 2003-2008 Fabrice Bellard

GPU:

02:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 620 OEM] (rev a1)
02:00.1 Audio device: NVIDIA Corporation GF119 HDMI Audio Controller (rev a1)

libvirt.xml

<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">  
  <name>win10</name>  
  <os> 
    <type machine="q35">hvm</type>  
    <boot dev="hd"/>  
    <boot dev="cdrom"/> 
  </os>  
  <features> 
    <acpi/>  
    <apic/>  
    <hyperv>
      <vendor_id state='on' value='1234567890ab'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>   
  <clock offset="localtime"> 
    <timer name="rtc" tickpolicy="catchup"/>  
    <timer name="pit" tickpolicy="delay"/>  
    <timer name="hpet" present="no"/> 
    <timer name='hypervclock' present='yes'/>
  </clock>  
  <on_poweroff>destroy</on_poweroff>  
  <on_reboot>restart</on_reboot>  
  <on_crash>destroy</on_crash>  
  <vcpu current="4">4</vcpu>  
  <cpu mode="host-passthrough"> 
    <topology sockets="1" cores="4" threads="1"/> 
  </cpu>  
  <memory>8388608</memory>  
  <currentMemory>8388608</currentMemory>  
  <devices> 
    <emulator>/root/qemu25/qemu/x86_64-softmmu/qemu-system-x86_64</emulator>  
    <disk device="disk" type="file"> 
      <driver name="qemu" type="qcow2"/>  
      <source file="/root/vm/win10/image.qcow2"/>  
      <target bus="virtio" dev="vda"/> 
    </disk>  
    <sound model="ac97"/>  
    <interface type="bridge"> 
      <mac address="fa:16:3e:81:00:03"/>  
      <source bridge="eucabr"/>  
      <model type="virtio"/>  
      <driver name="qemu"/>  
      <alias name="net0"/> 
    </interface>  
    <hostdev mode="subsystem" type="pci" managed="yes"> 
      <source> 
        <address domain="0x0000" bus="0x02" slot="0x00" function="0x1"/>
      </source> 
    </hostdev> 
  </devices>  
  <qemu:commandline> 
    <qemu:arg value="-machine"/>  
    <qemu:arg value="smm=off"/>  
    <qemu:arg value="-device"/>  
    <qemu:arg value="ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1"/>  
    <qemu:arg value="-device"/>  
    <qemu:arg value="vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on"/>  
    <qemu:arg value="-vga"/>  
    <qemu:arg value="none"/>  
  </qemu:commandline> 
</domain>

qemu command

/root/qemu25/qemu/x86_64-softmmu/qemu-system-x86_64 \
-name win10 \
-machine q35,accel=kvm,usb=off \
-cpu host,kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_vendor_id=blah \
-m 2048 \
-realtime mlock=off \
-smp 2,sockets=1,cores=2,threads=1 \
-no-user-config \
-nodefaults  \
-rtc base=localtime \
-no-shutdown \
-boot strict=on \
-device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e \
-device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 \
-drive file=/root/vm/win10/snap.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 \
-device virtio-blk-pci,scsi=off,bus=pci.2,addr=0x2,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-k en-us \
-device virtio-balloon-pci,id=balloon0,bus=pci.2,addr=0x4 \
-machine smm=off \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 \
-msg timestamp=on \
-vga none

Look forward to your help!

Amplify

Answer 1

You need to pass copy of unmodified videocard ROM to VM.

• You need a secondary GPU that you can use as the primary for this
  process. You cannot dump a clean copy of the BIOS without having the passthrough GPU as a secondary card
• Put the extra card in the primary slot and the intended passthrough card in another pci-e port and bootup.
• Find your intended GPU again via lspci -v. In my case it had about the same address.

• Now you can dump the ROM to a file:

  # echo "0000:05:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
  # cd /sys/bus/pci/devices/0000\:05\:00.0
  # echo 1 > rom 
  # cat rom > /home/username/KVM/evga_gtx970.dump
  # echo 0 > rom
  # echo "0000:05:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
  

  In this case, 0000:05:00.0 is my PCI card address. You don't really need the bind step at the bottom since you'll be rebooting anyways.

• You can check the integrity of the ROM dump with this handy utility at https://github.com/awilliam/rom-parser. My rom looks like:

  # ./rom-parser evga_gtx970.dump
  Valid ROM signature found @0h, PCIR offset 1a0h
          PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 13c2, class: 030000
          PCIR: revision 0, vendor revision: 1
  Valid ROM signature found @f400h, PCIR offset 1ch
          PCIR: type 3 (EFI), vendor: 10de, device: 13c2, class: 030000
          PCIR: revision 3, vendor revision: 0
                  EFI: Signature Valid, Subsystem: Boot, Machine: X64
  Last image
  

  You should have both an EFI and a non-EFI x86 ROM in the dump ( I think most cards have both)

• Turn off the machine and put your GTX 1070 back in the primary slot.

• After booting, edit your VM xml and in the section for your GPU (if you have already assigned the GPU to the VM) there should be a section. Add a file='path/to/dump/here' statement to it. My full section looks like:

  <hostdev mode='subsystem' type='pci' managed='yes'>
    <source>
      <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </source>
    <rom bar='on' file='/home/username/KVM/evga_gtx970.dump'/>
    <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
  </hostdev>
  

  This will have the VM start the card with that BIOS instead of whatever the kernel gives it.

source

Please note that you have to use OVMF (EFI) because SeaBIOS does not use card ROM properly.

Drakensberg

For me, dumping the BIOS this way worked, but dumping through GPU-Z produced a BIOS that didn't work. Yet rom-parser says both are valid. Working BIOS is 126K, non-working BIOS is 253K. – Gunwale

The card I am passing through is my secondary card. I already have my host machine ignoring it with vfio. There is nothing to dump when I start up my host machine unless I have booted into the host computer, then I can dump the rom from that GPU. Is this different than what you described? – Republic

Not sure I understand your case. Could you explain with a bit more details, please? – Drakensberg

Answer 2

If you're on OVMF or some other UEFI, make sure to triple-check that your card is UEFI-ready, especially on stuff that is older than ~2014.

I was under false impression that mine was (GTX 770), while, in fact, it wasn't (looked at the wrong version of ROM online) and wasted almost 2 days ripping my hair out. Look up UEFI support like so and look for ROM updates here.

I flashed my card, but I think you can put an UEFI-enabled ROM as romfile=. It appears that other manufacturers' ROMs could work too, if yours doesn't have an UEFI fix for you.

Stace

Indeed this "false impression" was the problem in my case. I went to that site, downloaded the latest firmware for my card model and, BAM!, it worked. – Councilman