Linux live kernel patching with kpatch on CentOS 7

Live or dynamic kernel patching allows you to patch a running kernel with no impact to running applications and without rebooting the system. Since the announcement and release of Linux kernel version 4, live kernel patching got got a lot of extra attention. Most probably this is because there weren’t a lot of big changes that one would expect with an increase in the version number. While v4 of the kernel does introduce some new stuff related to live or dynamic kernel patching, Live kernel patching was and is already available for earlier kernel releases. In this post, I’ll try to explain the differences between the new and old method and more important how to do live patching of a v3 kernel on CentOS 7.

There is a lot of confusion around live kernel patching. First of all, there is still a lot of work to do and as far as I know, there is no certified, production-ready, solution for live patching available.

Ksplice, kpatch and kGraft

There once was Ksplice but since Oracle took over the project, it’s no longer freely usable. Both Redhat and Suse started working on an alternative: kpatch and kGraft respectively. Both are function-based so either the old or new (patched) version of a function is used. kGraft does this on an task by task basis while kpatch idles all CPU’s and patches the function for all tasks in one time. While kpatch technically introduces a small latency it’s much faster and more robust than kGraft.

Linux kernel v4 and live patching

What does all of the above has to do with Linux kernel version 4? Well, since v4, the common part of the code that is required by kpatch and kGraft got integrated in the kernel source. This doesn’t include the necessary tools to do live patching so with only the addition to the source, you can’t do live patching. To give an example: Fedora (upstream for RHEL and CentOS) announced that they will disable live patching functionality on the v4-kernel that ships with F22 since they don’t feel ready and don’t see the advantage (yet). Basically the changes in the kernel prepare for serious live patching in the future but it isn’t there yet.

Live kernel patching today

What you probably would like to know if you ended up here is how to patch a running kernel. Like I said, it doesn’t matter if it’s a kernel v3 or v4, you can do live patching. When you have a v4-kernel with the live patching feature enable during kernel compilation, the added functionality in the kernel will be used for patching.

To show you that it’s perfectly possible to patch a kernel without rebooting, I’ll guide you trough the necessary steps. I did mention but will do it again that live patching is not production ready and you could end up with an unstable system.

As with most of my posts, I’m starting with a minimal install of CentOS 7. At the time of writing, I installed the latest CentOS-supported kernel: 3.10.0-229.7.2.el7.x86_64.

Install kpatch and kpatch-build

The first step is to get kpatch and kpatch-build and compile/install them on your system.

Install the necessary dependencies for kpatch and kpatch-build:

Clone the kpatch source from Github:

Compile kpatch:

Install the compiled kpatch:

Install other required components for live patching

After installing kpatch, it’s time to install the rest of the requirements for live kernel patching. In order to do the actual patching, the kernel needs to be recompiled into an original and patched version. This means that the requirements are quite equal to what you need for normal kernel compiling.

Install required packages for kernel compiling:

Install the debuginfo packages for the kernel:

Install and configure the compiler cache (ccache) to speed up the compilation process (especially when doing multiple compilation of the same components):

Create a kernel patch for testing

In order to test live patching of the kernel, I’ll create a small patch for the kernel and activate it without rebooting. The patch will be to show a different value for /proc/version (I know, very mission-critical :)).

Before we do anything, we’ll check the contents of /proc/version:

Now we’ll create the patch for the changed version. The first step is to copy the kernel source in two versions, the original one (linux.orig) and the one that will be changed (linux.kpatch):

Edit the file responsible for /proc/version in ~/linux.kpatch/fs/proc/version.c and add something to the version_proc_show-function:

Create the patch:

Live patch the kernel

In the previous step, we got ourselves a small patch for the kernel. Now it’s finally time to apply the patch to the running system.

Create the patch module with kpatch:

This process will take a while since it involves compiling the kernel twice. Fortunately, ccache helps to speed-up the process a lot. After building, you should end up with a patch module:

Now that we have a module suited for live patching, we can load it immediately using kpatch:

Let’s see if that worked by asking for /proc/version:

As you can see, besides installing all prerequisites, live patching is very easy and fast and more important without any interruption :)

More actions

Besides dynamically loading a kernel patch, you can also unload it in case you’re not happy with it. This will bring you intermediately back to the situation as it was before the patch:

To permanently install a patch:

To list applied and installed patches:

2 thoughts on “Linux live kernel patching with kpatch on CentOS 7

  1. Pingback: cron.weekly issue #69: SHA-1, Kernel expoit, pssh, securitybot, nscan, Kernel 4.10, live kernel patching & more

  2. Pingback: How to live patch Kernel on CentOS Linux 7

Leave a Reply

Your email address will not be published. Required fields are marked *