Our team is working on the best approach to secure our customers’ systems against the recently reported Spectre and Meltdown vulnerabilities. Our first step is to understand the problem and its mitigations. This post provides a roundup of discussions and work on the topic with a focus on mitigation for the Xen hypervisor.
The vulnerabilities
https://access.redhat.com/security/vulnerabilities/speculativeexecution
Spectre (aka “Branch target injection”) includes:
SP1) speculative execution to perform bounds-check bypass (CVE-2017-5753)
SP2) utilizing branch target injection (CVE-2017-5715) to cause kernel code at an address under attacker control to execute speculatively
Meltdown:
SP3) third variant (CVE-2017-5754) rogue data cache load. Relies on the fact that, on impacted microprocessors, during speculative execution of instruction permission faults, exception generation triggered by a faulting access is suppressed until the retirement of the whole instruction block. Subsequent memory accesses may cause an allocation into the L1 data cache even when they reference otherwise inaccessible memory locations. As a result, an unprivileged local attacker could read privileged (kernel space) memory (including arbitrary physical memory locations on a host) by conducting targeted cache side-channel attacks.
Meltdown: A CPU predicts you will walk into a bar, but you do not. Your credit card information has been stolen from the chair you were going to sit in. [reddit]
Kernel Side-Channel Attacks is the catch all for the three variants.
Xen FAQ
https://blog.xenproject.org/2018/01/04/xen-project-spectremeltdown-faq/:
- Intel is only one affected by Meltdown. AMD has (some?) vulnerability to Spectre.
- Both guest and hypervisor affected by Spectre.
- On Intel processors, only 64-bit PV mode guests can attack Xen.
- Guests running in 32-bit PV mode, HVM mode, and PVH mode cannot attack the hypervisor using SP3.
- However, in 32-bit PV mode, HVM mode, and PVH mode, guest userspaces can attack guest kernels [same VM?] using SP3; so updating guest kernels is advisable.
- Guest kernels running in 64-bit PV mode are not vulnerable to attack using SP3, because 64-bit PV guests already run in a KPTI-like mode
https://lists.xenproject.org/archives/html/xen-devel/2018-01/ ongoing Xen development, much related to Meltdown/Spectre.
Linux Kernel Spectre mitigations
Retpoline: a software construct for preventing branch-target-injection
https://support.google.com/faqs/answer/7625886
“retpoline” is a portmanteau of “return” and “trampoline.”
https://lkml.org/lkml/2018/1/4/419 config RETPOLINE bool “Avoid speculative indirect branches in kernel”
https://access.redhat.com/articles/3311301 indicates microcode update from hardware vendor required to address SP2/Spectre.
- SP1 is both hardly exploitable and hard to patch but with a limited impact.
- SP2 Will be fixed using a microcode update on the short term (the exact delay depends of Intel and the server manufacturers) with a performance impact. We currently have no confirmation on the exact Kernel version needed to work in conjunction with the microcode update. Our current assumption is that the fixes needed are not yet in the main kernel tree and that it will be merged in 4.15
- SP2 also fixed by Retpoline (longer term)
Xen Spectre mitigations
https://blog.xenproject.org/2018/01/04/xen-project-spectremeltdown-faq/
- SP2 protection “requires the hypervisor to be recompiled with a compiler that contains special new features”.
- For SP2 the hypervisor can flush the branch prediction logic on entry into the hypervisor ([nosource] no patch for this as at 2018-01-05).
- SP1 is 'more difficult' to fix and Xen are only at the design stage (as at 2018-01-05).
Andrew Cooper’s mitigations for SP2 vesion 6.5 https://lists.xenproject.org/archives/html/xen-devel/2018-01/msg00110.html needs compiler with “-mindirect-branch=thunk-external and -mindirect-branch-register“ and microcode for Intel. “should be functionally correct, if a little rough around the edges”
https://wiki.xenproject.org/wiki/Xen_Project_Meltdown_and_Spectre_Technical_FAQ
For Meltdown/SP3 says to use Wei Liu patches for Run PV guest in PVH container. For SP2 use Andrew Cooper’s SP2 Mitigation patches per https://lists.xenproject.org/archives/html/xen-devel/2018-01/msg00110.html Ensure Supervisor Mode Execution Protection (SMEP) is enabled in the BIOS.
Meltdown mitigations
Kernel Page Table Isolation (was KAISER, FUCKWIT, etc).
https://access.redhat.com/articles/3311301
Indicates PTI can be disabled at runtime.
https://lkml.org/lkml/2017/12/4/1019 “PTI breaks PV”
Xen PV uses its own KPTI like feature. So the new KPTI feature is disabled on paravirt guest VMs per https://news.ycombinator.com/item?id=16065647::
if (hypervisor_is_type(X86_HYPER_XEN_PV)) {
pti_print_if_insecure(“disabled on XEN PV.”);
return;
}
Xen Meltdown mitigations
https://xenbits.xen.org/xsa/advisory-254.html
- SP3 can be mitigated by running guests in HVM or PVH mode.
- SP3: For guests with legacy PV kernels which cannot be run in HVM mode, we have developed a “shim” hypervisor that allows PV guests to run in PVH mode. Unfortunately, due to the accelerated schedule, this is not yet ready to release. We expect to have it ready for 4.10, as well as PVH backports to 4.9 and 4.8, available over the next few days [as at 4 Jan]. See Wei Liu “Run PV guest in PVH container” below.
https://lists.xenproject.org/archives/html/xen-devel/2018-01/msg00274.html
44 commit patch set from Andrew Cooper to address KPTI/Kaiser/SP3/Meltdown in Xen. RFC at Thu, 04 Jan 2018 20:22. Creates per-pcpu isolated region. But will either break the ABI or be slow per ‘If we want to continue down this route, we either need someone to have a clever alternative to the shadowing algorithm I came up with, or change the PV ABI to require VMs not to share L4 pagetables.’ [opinion fixes coming for PV Xen. But looks like PV is not long for this world. And options like PVHv2 or HVM are going to be the necessary way to go. Else Run PV guest in PVH container per Wei Liu below]
https://lists.xenproject.org/archives/html/xen-devel/2018-01/msg00188.html
77 commit patch set from Wei Liu “Run PV guest in PVH container”. This series makes it possible to continue running untrusted PV guests. Slideshow at https://www.slideshare.net/xen_com_mgr/xpdds17-keynote-towards-a-configurable-and-slimmer-x86-hypervisor-wei-liu-citrix Per slide 15 the of a ‘shim’ refers to a some code ‘inside’ the guest VM between the guest kernel and the hypervisor.
Performance impacts
https://www.phoronix.com/scan.php?page=article&item=linux-415-x86pti&num=2 Performance with/without KPTI patches. Performance for some things is significantly worse (synthetic file I/O, postgres). Other things are more marginal (kernel compile, video processing). [opinion] Performance takes a hit, but it shouldn’t be as bad as the early 30% hit people were talking about.
https://www.phoronix.com/scan.php?page=article&item=linux-kpti-kvm&num=3 Performance with/without KPTI for KVM. KVM performs marginally slower.
https://access.redhat.com/articles/3307751 Measureable: 8-19% – Highly cached random memory, with buffered I/O, OLTP database workloads, and benchmarks with high kernel-to-user space transitions are impacted between 8-19%. Examples include OLTP Workloads (tpc), sysbench, pgbench, netperf (< 256 byte), and fio (random I/O to NvME).
Other tasks (particularly things like CPU intensive workloads like HPC) have less to minimal overhead from KPTI patches.