► OpenWRT installation as Xen domU with PCI passthrough ◄

La version Française est disponible ici: Installation d'OpenWRT en domU Xen avec activation du PCI passthrough

Introduction

Of of the main potential security issue with Xen is dom0. As he controls all devices, and network cards among the others, he can be hacked with hardware attack against poorly firmwares. So, when dom0 is supposed to be pretty much protected against unauthorized and/or malicious access, in fact it is not. The idea here is to use PCI passthrough to transfert network card control to a domU so that dom0 will be isolated from the Internet.

Qubes-os project do use PCI passthrough to isolate dom0 from network. Its architecture spécifications document perfectly describe and explain the why and how.

So, let's try to isolate as well our Debian dom0 behind a dedicated domU. Basically, any distribution more or less security oriented can be used here. In my case, I've choosen OpenWRT because it's small, I know it and I like technicals challenges.

Compilation environment setup

We're going to compile OpenWRT from source. For that, we need to setup a build environment. Current OpenWRT stable release, called Backfire, provides kernel 2.6.32. This kernel miss of important thing for us: Xen PCI Frontend driver. If you really want / need to use this kernel, you'll have to backport patches. Here, we'll use the newly released kernel 2.6.37 because needed driver is now merged into mainstream.

For build setup, no problem, everything is fully described here. For those how don't want to go away from here:

Build environment setup Install needed package to get OpenWRT source tree

aptitude install subversion subversion-tools

BuildRoot packages (32bits)

aptitude install install build-essential asciidoc autoconf binutils bison bzip2 \
flex gawk gettext libncurses5-dev libz-dev patch unzip zlib1g-dev

BuildRoot packages (64bits)

aptitude install install build-essential asciidoc autoconf binutils bison bzip2 \
  flex gawk gettext libncurses5-dev libz-dev patch unzip zlib1g-dev \
  ia32-libs lib32gcc1 libc6-dev-i386

Get SVN source tree

svn co svn://svn.openwrt.org/openwrt/trunk trunk
cd trunk
./scripts/feeds update
./scripts/feeds install -a

Check

make prereq

Specify kernel version to use

vi target/linux/x86/Makefile
[.....]
LINUX_VERSION:=2.6.37
[.....]

Fix xen kernel module name

vi package/kernel/modules/virtual.mk
[.....]
define KernelPackage/xen-evtchn
  SUBMENU:=$(VIRTUAL_MENU)
  TITLE:=Xen event channels
  DEPENDS:=@TARGET_x86_xen_domu
  KCONFIG:=CONFIG_XEN_DEV_EVTCHN
  FILES:=$(LINUX_DIR)/drivers/xen/xen-evtchn.ko
  AUTOLOAD:=$(call AutoLoad,06,evtchn)
endef
[.....]

The 2 last modifications allow you to specify the kernel version you want to use and fix a little bug introduced with one kernel module filname change (line FILES). If you forgot the last one, build will fail. You've been warned.

OpenWRT configuration & compilation

OpenWRT configuration & compilationOpenWRT configuration

make menuconfig
[.....]
Target System
  [x] x86
Subtarget
  [x] Xen Paravirt Guest
Global build settings
  [ ] Compile the kernel with Debug Filesystem enabled
Kernel modules
  Xen paravirtualized guest support
      [x] kmod-xen-evtchn
      [ ] kmod-xen-fbdev
      [x] kmod-xen-fs
      [x] kmod-xen-kbddev
      [x] kmod-xen-netdev
[.....]

pciutils package activation (not mandatory, but it will help you to check that PCI passthrough is working.)

[.....]
Utilities
  [x] pciutils
[.....]

OpenWRT compilation and disk image generation

make

Once build is done, you can deploy kernel and disk image on your dom0:

Deployment

scp bin/x86/openwrt-x86-xen_domu-combined-ext4.img.gz bin/x86/openwrt-x86-xen_domu-vmlinuz dom0:/path_to_your_image/

Now, you can configure the Xen part.

OpenWRT domU configuration

We'll use Paravirt domU. That's why kernel bzimage is not included into disk image. But all the module you'll use are because they must be reachable from whithin the domU.

OpenWRT domU Xen configuration Prepare image

cd /path_to_your_image
gunzip -d openwrt-x86-xen_domu-combined-ext4.img.gz

Configuration file /etc/xen/auto/openwrt.cfg

kernel       = '/path_to_your_image/openwrt-x86-xen_domu-vmlinuz'
root         = '/dev/xvda2 rw'

memory       = '256'
vcpus        = '1'
cpus         = '1'
localtime    = 0
serial       = 'pty'

disk         = [ 'file:/path_to_your_image/openwrt-x86-xen_domu-combined-ext4.img,xvda,w' ]

on_poweroff  = 'destroy'
on_reboot    = 'restart'
on_crash     = 'restart'

extra = "console=hvc0 xencons=tty"
#extra = "iommu=soft swiotlb=force console=hvc0 xencons=tty"
#pci = [ '04:00.0' ]

name         = 'openwrt'
hostname     = 'openwrt.yourdomain.tld'

Let's first test domU without any PCI passthrough. If it works, you'll just have to comment line extra = "console=hvc0 xencons=tty" and uncomment the 2 next in order to activate PCI passthrough.

OpenWRT domU first start

xm create /etc/xen/auto/openwrt.cfg -c
[.....]
blkfront: xvda: barriers enabled
 xvda: xvda1 xvda2
xvda: p2 size 98721 extends beyond EOD, truncated
XENBUS: Device with no driver: device/console/0
EXT4-fs (xvda2): mounted filesystem without journal. Opts: (null)
VFS: Mounted root (ext2 filesystem) on device 202:2.
Freeing unused kernel memory: 304k freed
- preinit -
Press the [f] key and hit [enter] to enter failsafe mode
- regular preinit -
EXT4-fs (xvda2): re-mounted. Opts: (null)
- init -

Please press Enter to activate this console.

BusyBox v1.17.3 (2011-01-05 16:42:08 CET) built-in shell (ash)
Enter 'help' for a list of built-in commands.

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 KAMIKAZE (bleeding edge, r24906) ------------------
  * 10 oz Vodka       Shake well with ice and strain
  * 10 oz Triple sec  mixture into 10 shot glasses.
  * 10 oz lime juice  Salute!
 ---------------------------------------------------
root@OpenWrt:/#

If you can see something like this, more or less, you're done. You OpenWRT domU is properly working. Let's activate PCI passthrough.

Xen PCI passthrough

I'll give here only a quick summary, time for me to write a real documentation about that topic. You'll have to:

Identify which devices you want to export to domU
Configure dom0 so that I won't take control over the devices and use instead Xen PCI backend driver (Reboot mandatory !)
Adapt domU xen configuration file.
Start domU and check

This will change a bit the Xen network architecture:

Default Xen network Architecture
Xen network architecture with PCI passthrough

Of course, because I work with Debian, we'll follow the Debian way.

PCI passthrough setup Identify device

lspci | grep Ethernet
01:05.0 Ethernet controller: D-Link System Inc DGE-528T Gigabit Ethernet Adapter (rev 10)
03:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
03:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 12)

Grub configuration /etc/default/grub

[.....]
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX="xen-pciback.permissive xen-pciback.hide=(04:00.0) pci=resource_alignment=04:00.0"
[.....]
GRUB_CMDLINE_XEN_DEFAULT=""
GRUB_CMDLINE_XEN="dom0_mem=256M dom0_max_vcpus=1 dom0_vcpus_pin"

update-grub
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.32-5-xen-amd64
Found initrd image: /boot/initrd.img-2.6.32-5-xen-amd64
Found linux image: /boot/vmlinuz-2.6.32-5-xen-amd64
Found initrd image: /boot/initrd.img-2.6.32-5-xen-amd64
done

Now you can reboot. But be sure to have an easy access to you dom0, either physical, console, KVM or even network (never know). If you don't use bonding and /or don't have spare network device, you'll loose network access until you configured both OpenWRT and dom0 internal network.

In my case, I use bonding so it's easy to remove one card without loosing everything. I also have RS232 console access from another server. Just said "never know", hu ? :)

After reboot check

Check that PCI passthrough do works on dom0 List of PCI devices

lspci | grep Ethernet
01:05.0 Ethernet controller: D-Link System Inc DGE-528T Gigabit Ethernet Adapter (rev 10)
03:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
03:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 12)

lspci -v -s 0000:04:00
04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 12)
        Subsystem: ASUSTeK Computer Inc. Device 81f8
        Flags: fast devsel, IRQ 18
        Memory at feb00000 (64-bit, non-prefetchable) [disabled] [size=16K]
        I/O ports at e800 [disabled] [size=256]
        Expansion ROM at 10100000 [disabled] [size=128K]
        Capabilities: [48] Power Management version 3
        Capabilities: [50] Vital Product Data
        Capabilities: [5c] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [e0] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: pciback
        Kernel modules: sky2

Don"t be suprised to see PCI device into dom0. It's not hidden but now under control of pciback driver. Kernel module which should have had control is sky2.

Check that PCI passthrough do work from domU PCI devices list

lspci | grep Ethernet
04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 12)

Yeah !, you're done. You can now configure OpenWRT so that it'll act as gatekeeper for you dom0 and other domU as well.

Have fun !

Sources

OpenWRT

How build OpenWRT

http://wiki.openwrt.org/doc/howto/build

Xen

Xen PCI passthrough

http://wiki.xensource.com/xenwiki/XenPCIpassthrough

I'm a Linux system engineer, mainly on Debian and RedHat/CentOS. Among many other things, I also work on virtualisation (Xen, VMware) and web technologies.

OpenWRT installation as Xen domU with PCI passthrough

Jean Baptiste FAVRE