🍻 Relaxed RMRR Mapping for Linux 3.17+

🐧💨 Now you can use PCI passthrough on broken platforms

TL;DR

When you try to use PCI/PCIe passthrough in KVM/QEMU/Proxmox you get:

vfio-pci 0000:01:00.1: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.

followed by vfio: failed to set iommu for container: Operation not permitted.

This kernel patch fixes the problem on kernels v3.17 and up (tested up to 5.9.1). You can skip to "Installation" section if you don't care about the rest. Reading of "Disclaimers" section to understand the risks, and "Solutions & hacks" to get the idea of different alternatives is highly recommended.

---

Table of Contents

1. Installation
   • Proxmox - premade packages
   • Proxmox - building from sources
   • Other distros
2. Configuration
3. Deep Dive - a throughout research on the problem written for mortals
   • Technical details
     • How virtual machines use memory?
     • Why do we need VT-d / AMD-Vi?
     • How PCI/PCIe actually work?
     • RMRR - the monster in a closet
     • What vendors did wrong?
   • Other solutions & hacks
     • Contact your platform vendor
     • Use OS which ignores RMRRs
     • Attempt HPE's pseudofix (if you use HP)
     • The comment-the-error-out hack (v3.17 - 5.3)
     • Long-term solution - utilizing relaxable reservation regions (>=3.17)
       • Why commenting-out the error is a bad idea
       • The kernel moves on quickly
       • What this patch actually does
       • Why kernel patch and not a loadable module?
     • The future
4. Disclaimers
5. Acknowledgments & References
6. License

---

Installation

Proxmox - premade packages

As I believe in eating your own dog food I run the kernel described here. Thus, I publish precompiled packages.

1. Go to the releases tab and pick appropriate packages
2. Download all *.debs packages to the server (you can copy links and use wget https://... on the server itself)
3. Install all using dpkg -i *.deb in the folder where you downloaded the debs
4. (OPTIONAL) Verify the kernel works with the patch disabled by rebooting and checking if uname -r shows a version ending with -pve-relaxablermrr
5. Configure the kernel

---

Proxmox - building from sources

If you're running a version of Proxmox with no packages available you can compile the kernel yourself using patches provided.

1. Prepare at least 60GB of free disk space
2. Install required packages:

   apt update
   apt install git nano screen patch fakeroot build-essential devscripts libncurses5 libncurses5-dev libssl-dev bc flex bison libelf-dev libaudit-dev libgtk2.0-dev libperl-dev asciidoc xmlto gnupg gnupg2 rsync lintian debhelper libdw-dev libnuma-dev libslang2-dev sphinx-common asciidoc-base automake cpio dh-python file gcc kmod libiberty-dev libpve-common-perl libtool perl-modules python-minimal sed tar zlib1g-dev lz4

3. Download everything:

   mkdir new-kernel ; cd new-kernel
   git clone --depth=1 git://git.proxmox.com/git/pve-kernel.git
   git clone --depth=1 https://github.com/kiler129/relax-intel-rmrr.git

4. Add kernel patch & patch the toolchain

   cd pve-kernel
   cp ../relax-intel-rmrr/patches/add-relaxable-rmrr-below-5_8.patch ./patches/kernel/CUSTOM-add-relaxable-rmrr.patch
   patch -p1 < ../relax-intel-rmrr/patches/proxmox.patch

5. Compile the kernel

   make

This step will take a lot of time (30m-3h depending on your machine).

6. Install new kernel:

   dpkg -i *.deb

7. (OPTIONAL) Verify the kernel works with the patch disabled by rebooting and checking if uname -r shows a version ending with -pve-relaxablermrr
8. Configure the kernel

---

Other distros

1. Download kernel sources appropriate for your distribution
2. Apply an appropriate patch to the source tree
   • Go to the folder with your kernel source
   • For Linux 3.17 - 5.7: patch -p1 < ../patches/add-relaxable-rmrr-below-5_8.patch
   • For Linux >=5.8: patch -p1 < ../patches/add-relaxable-rmrr-5_8_and_up.patch
3. Follow your distro kernel compilation & installation instruction

TODO: Add automation script

---

Configuration

By default, after the kernel is installed, the patch will be inactive (i.e. the kernel will behave like this patch was never applied). To activate it you have to add intel_iommu=relax_rmrr to your Linux boot args.

In most distros (including Proxmox) you do this by:

1. Opening /etc/default/grub (e.g. using nano /etc/default/grub)
2. Editing the GRUB_CMDLINE_LINUX_DEFAULT to include the option:
   • Example of old line:

     GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt intremap=no_x2apic_optout"
     

   • Example of new line:

     GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on,relax_rmrr iommu=pt intremap=no_x2apic_optout"
     

   • Side note: these are actually options which will make your PCI passthrough work and do so efficiently
3. Running update-grub
4. Rebooting

To verify if the the patch is active execute dmesg | grep 'Intel-IOMMU' after reboot. You should see a result similar to this:

root@sandbox:~# dmesg | grep 'Intel-IOMMU'
[    0.050195] DMAR: Intel-IOMMU: assuming all RMRRs are relaxable. This can lead to instability or data loss
root@sandbox:~# 

---

Disclaimers

• I'm not a kernel programmer by any means, so if I got something horribly wrong correct me please :)
• This path should be safe, as long as you don't try to remap devices which are used by the IPMI/BIOS, e.g.
  • Network port shared between your IPMI and OS
  • RAID card in non-HBA mode with its driver loaded on the host
  • Network card with monitoring system installed on the host (e.g. Intel Active Health System Agent)
• This is not a supported solution by any of the vendors. In fact this is a direct violation of Intel's VT-d specs (which Linux already violates anyway, but this is increasing the scope). It may cause crashes or major instabilities. You've been warned.

---

Acknowledgments & References

• Comment-out hack research by dschense
• Proxmox kernel compilation & patching by Feni
• Linux IOMMU Support
• RedHat RMRR EXCLUSION Whitepaper
• Intel® Virtualization Technology for Directed I/O (VT-d)
• Intel® Virtualization Technology for Directed I/O Architecture Specification

---

License

This work (patches & docs) is dual-licensed under MIT and GPL 2.0 (or any later version), which should be treated as an equivalent of Linux Dual MIT/GPL (i.e. pick a license you prefer).