kvm hypercall with multiple args

Question

I'm currently trying to build a small hypervisor and kernel using kvm and I struggle to get hypercalls with multiple args working correctly.

Here is what I've tried:

// guest.c

#define KVM_HYPERCALL vmcall
// #define KVM_HYPERCALL vmmcall
// #define KVM_HYPERCALL ".byte 0x0f,0x01,0xd9"
// #define KVM_HYPERCALL .byte 0x0f,0x01,0xc1"

static inline long kvm_hypercall4(int nr, unsigned long p1,
                  unsigned long p2, unsigned long p3,
                  unsigned long p4) {
    long ret;
    asm volatile(KVM_HYPERCALL
             : "=a"(ret)
             : "a"(nr), "b"(p1), "c"(p2), "d"(p3), "S"(p4)
             : "memory");
    return ret;
}

Any of those Hypercalls lead to vcpu->kvm_run->exit_reason to equal 6 which is to my surprise KVM_EXIT_MMIO instead of KVM_EXIT_HYPERCALL

switch (vcpu->kvm_run->exit_reason) {
  case KVM_EXIT_MMIO:
    printf("syscall: %lld\n", vcpu->kvm_run->hypercall.nr); // prints 0
    printf("arg 1: %lld\n",  vcpu->kvm_run->hypercall.args[1]); // prints 0
    printf("arg 2: %lld\n", vcpu->kvm_run->hypercall.args[2]); // prints 0
    printf("arg 3: %lld\n",  vcpu->kvm_run->hypercall.args[3]); // prints 0

    if(ioctl(vcpu->fd, KVM_GET_REGS, &regs)<0) exit 1;

    printf("rax: %lld\n", regs.rax); // prints 0
    printf("rbx: %lld\n", regs.rbx); // prints 0
    printf("rcx: %lld\n", regs.rcx); // prints 0

Aside of the exit reason being KVM_EXIT_MMIO why are the regs not set? What is the right way to trigger a KVM_EXIT_HYPERCALL with multple arguments?

Thanks in advance

EDIT: In case it matters: I'm using 9th generation intel i7 cpu running debian with linux kernel 5.4

Haberman

Answer 1

KVM_EXIT_HYPERCALL is no longer used, according to the documentation:

/* KVM_EXIT_HYPERCALL */
      struct {
          __u64 nr;
          __u64 args[6];
          __u64 ret;
          __u32 longmode;
          __u32 pad;
      } hypercall;

Unused. This was once used for 'hypercall to userspace'. To implement such functionality, use KVM_EXIT_IO (x86) or KVM_EXIT_MMIO (all except s390). Note KVM_EXIT_IO is significantly faster than KVM_EXIT_MMIO.

And it seems to me that KVM_EXIT_HYPERCALL is also not implemented. Quick and dirty search with grep. It's defined, but will never assigned as exit_reason:

user@host:~/Linux/src> grep -R KVM_EXIT_HYPERCALL
include/uapi/linux/kvm.h:#define KVM_EXIT_HYPERCALL        3
include/uapi/linux/kvm.h:               /* KVM_EXIT_HYPERCALL */
Documentation/virt/kvm/api.rst:         /* KVM_EXIT_HYPERCALL */
tools/include/uapi/linux/kvm.h:#define KVM_EXIT_HYPERCALL        3
tools/include/uapi/linux/kvm.h:         /* KVM_EXIT_HYPERCALL */
tools/testing/selftests/kvm/lib/kvm_util.c:     {KVM_EXIT_HYPERCALL, "HYPERCALL"},
user@host:~/Linux/src>

Linux version:

user@host:~/Linux/src> git-describe --tags
v5.6-10895-g4c205c84e249
user@host:~/Linux/src>

There is an older question how to implement custom VMCALL's with two answers on this site. Have you tried them?

Misnomer

The answers only demonstrated how to create hypercalls with 0 arguments using kvm_hypercall0 or assembly. However, I'm trying to get it working for multiple arguments and the kvm_hypercallN provided of my system don't seem to behave as expected. For instance I'd expected regs.rax would have the values I assigned in my guest program – Haberman

Answer 2

From the kernel documentation kvm/api

If exit_reason is KVM_EXIT_MMIO, then the vcpu has executed a memory-mapped I/O instruction which could not be satisfied by kvm. The 'data' member contains the written data if 'is_write' is true, and should be filled by application code otherwise.

The hypercall you trigged raised such failure.
This depend on the code of the hypercall you called.

Masteratarms

1) I performed a vmcall not IO 2) How do I setup the registers and VMM correctly so that I can pass multiple args through the registers to the VMM? - I'd expect that I would be able to read the registers via ioctl(KVM_GET_REGS, &regs) but all regs seem to be zero – Haberman

@Haberman I guess the vmcall is raising KVM_EXIT_MMIO as an exception, still not undersood why you are redefinng KVM_HYPERCALL instead of using the definition in kvm_para.h or the one in your corresponding architechture – Masteratarms

thanks for your reply. I'm using kvm_para.h but for troubleshooting purposes I've copy & pasted it into my project and here to potentially enable others with a non kvm background to join the discussion. In the end it's about using VT-X / AMD-V correctly – Haberman

Answer 3

I found this in the KVM API documentation:

/* KVM_EXIT_HYPERCALL */
struct {
__u64 nr;
__u64 args[6]; __u64 ret;
__u32 longmode;
__u32 pad;
} hypercall;

Unused. This was once used for 'hypercall to userspace'. To implement such functionality, use KVM_EXIT_IO (x86) or KVM_EXIT_MMIO (all except s390). Note KVM_EXIT_IO is significantly faster than KVM_EXIT_MMIO.

So, perhaps you should try implementing the Hypercall using either the KVM_EXIT_IO or KVM_EXIT_MMIO.

Concurrence