Live or dynamic kernel patching allows you to patch a running kernel with no impact to running applications and without rebooting the system. Since the announcement and release of Linux kernel version 4, live kernel patching got got a lot of extra attention. Most probably this is because there weren’t a lot of big changes that one would expect with an increase in the version number. While v4 of the kernel does introduce some new stuff related to live or dynamic kernel patching, Live kernel patching was and is already available for earlier kernel releases. In this post, I’ll try to explain the differences between the new and old method and more important how to do live patching of a v3 kernel on CentOS 7.

There is a lot of confusion around live kernel patching. First of all, there is still a lot of work to do and as far as I know, there is no certified, production-ready, solution for live patching available.

Ksplice, kpatch and kGraft

There once was Ksplice but since Oracle took over the project, it’s no longer freely usable. Both Redhat and Suse started working on an alternative: kpatch and kGraft respectively. Both are function-based so either the old or new (patched) version of a function is used. kGraft does this on an task by task basis while kpatch idles all CPU’s and patches the function for all tasks in one time. While kpatch technically introduces a small latency it’s much faster and more robust than kGraft.

Linux kernel v4 and live patching

What does all of the above has to do with Linux kernel version 4? Well, since v4, the common part of the code that is required by kpatch and kGraft got integrated in the kernel source. This doesn’t include the necessary tools to do live patching so with only the addition to the source, you can’t do live patching. To give an example: Fedora (upstream for RHEL and CentOS) announced that they will disable live patching functionality on the v4-kernel that ships with F22 since they don’t feel ready and don’t see the advantage (yet). Basically the changes in the kernel prepare for serious live patching in the future but it isn’t there yet.

Live kernel patching today

What you probably would like to know if you ended up here is how to patch a running kernel. Like I said, it doesn’t matter if it’s a kernel v3 or v4, you can do live patching. When you have a v4-kernel with the live patching feature enable during kernel compilation, the added functionality in the kernel will be used for patching.

To show you that it’s perfectly possible to patch a kernel without rebooting, I’ll guide you trough the necessary steps. I did mention but will do it again that live patching is not production ready and you could end up with an unstable system.

As with most of my posts, I’m starting with a minimal install of CentOS 7. At the time of writing, I installed the latest CentOS-supported kernel: 3.10.0-229.7.2.el7.x86_64.

Install kpatch and kpatch-build

The first step is to get kpatch and kpatch-build and compile/install them on your system.

Install the necessary dependencies for kpatch and kpatch-build:

[jensd@cen ~]$ sudo yum install git gcc kernel-devel elfutils elfutils-devel rpmdevtools pesign yum-utils zlib-devel binutils-devel newt-devel python-devel perl-ExtUtils-Embed audit-libs-devel numactl-devel pciutils-devel bison -y
...
Complete!

Clone the kpatch source from Github:

[jensd@cen ~]$ git clone https://github.com/dynup/kpatch.git
Cloning into 'kpatch'...
remote: Counting objects: 3689, done.
remote: Total 3689 (delta 0), reused 0 (delta 0), pack-reused 3689
Receiving objects: 100% (3689/3689), 1.11 MiB | 99.00 KiB/s, done.
Resolving deltas: 100% (2145/2145), done.

Compile kpatch:

[jensd@cen ~]$ cd kpatch
[jensd@cen kpatch]$ make
make -C kpatch-build
make[1]: Entering directory `/home/jensd/kpatch/kpatch-build'
...
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/home/jensd/kpatch/contrib'

Install the compiled kpatch:

[jensd@cen kpatch]$ sudo make install
make -C kpatch-build install
make[1]: Entering directory `/home/jensd/kpatch/kpatch-build'
...
make[1]: Leaving directory `/home/jensd/kpatch/contrib'

Install other required components for live patching

After installing kpatch, it’s time to install the rest of the requirements for live kernel patching. In order to do the actual patching, the kernel needs to be recompiled into an original and patched version. This means that the requirements are quite equal to what you need for normal kernel compiling.

Install required packages for kernel compiling:

[jensd@cen ~]$ sudo yum install asciidoc bc hmaccalc net-tools xmlto ncurses-devel -y
...
Complete!

Install the debuginfo packages for the kernel:

[jensd@cen ~]$ sudo yum --enablerepo base-debuginfo install kernel-debuginfo -y
...
Complete!

Install and configure the compiler cache (ccache) to speed up the compilation process (especially when doing multiple compilation of the same components):

[jensd@cen ~]$ sudo yum install epel-release -y
...
Complete!
[jensd@cen ~]$ sudo yum install ccache -y
...
Complete!
[jensd@cen ~]$ ccache --max-size=5G
Set cache size limit to 5.0 Gbytes

Create a kernel patch for testing

In order to test live patching of the kernel, I’ll create a small patch for the kernel and activate it without rebooting. The patch will be to show a different value for /proc/version (I know, very mission-critical :)).

Before we do anything, we’ll check the contents of /proc/version:

[jensd@cen ~]$ cat /proc/version
Linux version 3.10.0-229.7.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Tue Jun 23 22:06:11 UTC 2015

Now we’ll create the patch for the changed version. The first step is to copy the kernel source in two versions, the original one (linux.orig) and the one that will be changed (linux.kpatch):

[jensd@cen ~]$ mkdir ~/linux.orig
[jensd@cen ~]$ cp -r /usr/src/debug/kernel-3.10.0-229.7.2.el7/linux-3.10.0-229.7.2.el7.x86_64/* ~/linux.orig
[jensd@cen ~]$ cp -r ~/linux.orig/ ~/linux.kpatch

Edit the file responsible for /proc/version in ~/linux.kpatch/fs/proc/version.c and add something to the version_proc_show-function:

#include <linux/fs.h>
#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/proc_fs.h>
#include <linux/seq_file.h>
#include <linux/utsname.h>

static int version_proc_show(struct seq_file *m, void *v)
{
        seq_printf(m, linux_proc_banner,
                "JENSD.BE",
                utsname()->sysname,
                utsname()->release,
                utsname()->version);
        return 0;
}

static int version_proc_open(struct inode *inode, struct file *file)
{
        return single_open(file, version_proc_show, NULL);
}

static const struct file_operations version_proc_fops = {
        .open           = version_proc_open,
        .read           = seq_read,
        .llseek         = seq_lseek,
        .release        = single_release,
};

static int __init proc_version_init(void)
{
        proc_create("version", 0, NULL, &version_proc_fops);
        return 0;
}
module_init(proc_version_init);

Create the patch:

[jensd@cen ~]$ diff -u linux.orig/fs/proc/version.c linux.kpatch/fs/proc/version.c>version.patch
[jensd@cen ~]$ cat version.patch
--- linux.orig/fs/proc/version.c        2015-07-28 14:31:31.254894476 +0200
+++ linux.kpatch/fs/proc/version.c      2015-07-28 15:34:45.270740061 +0200
@@ -8,6 +8,7 @@
 static int version_proc_show(struct seq_file *m, void *v)
 {
        seq_printf(m, linux_proc_banner,
+               "JENSD.BE",
                utsname()->sysname,
                utsname()->release,
                utsname()->version);

Live patch the kernel

In the previous step, we got ourselves a small patch for the kernel. Now it’s finally time to apply the patch to the running system.

Create the patch module with kpatch:

[jensd@cen ~]$ kpatch-build version.patch
Fedora/Red Hat distribution detected
Downloading kernel source for 3.10.0-229.7.2.el7.x86_64
Unpacking kernel source
Testing patch file
checking file fs/proc/version.c
Building original kernel
Building patched kernel
Extracting new and modified ELF sections
version.o: changed function: version_proc_show
Patched objects: vmlinux
Building patch module: kpatch-version.ko
SUCCESS

This process will take a while since it involves compiling the kernel twice. Fortunately, ccache helps to speed-up the process a lot. After building, you should end up with a patch module:

[jensd@cen ~]$ ls -l kpatch-version.ko
-rw-rw-r--. 1 jensd jensd 281529 Jul 28 15:39 kpatch-version.ko

Now that we have a module suited for live patching, we can load it immediately using kpatch:

[jensd@cen ~]$ sudo /usr/local/sbin/kpatch load kpatch-version.ko
loading patch module: kpatch-version.ko

Let’s see if that worked by asking for /proc/version:

[jensd@cen ~]$ cat /proc/version
JENSD.BE version Linux (builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) 3.10.0-229.7.2.el7.x86_64

As you can see, besides installing all prerequisites, live patching is very easy and fast and more important without any interruption :)

More actions

Besides dynamically loading a kernel patch, you can also unload it in case you’re not happy with it. This will bring you intermediately back to the situation as it was before the patch:

[jensd@cen ~]$ sudo /usr/local/sbin/kpatch unload kpatch_version
disabling patch module: kpatch_version
unloading patch module: kpatch_version
[jensd@cen ~]$ cat /proc/version
Linux version 3.10.0-229.7.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Tue Jun 23 22:06:11 UTC 2015

To permanently install a patch:

[jensd@cen ~]$ sudo /usr/local/sbin/kpatch install kpatch-version.ko
installing kpatch-version.ko (3.10.0-229.7.2.el7.x86_64)
ln -s '/usr/local/lib/systemd/system/kpatch.service' '/etc/systemd/system/multi-user.target.wants/kpatch.service'

To list applied and installed patches:

[jensd@cen ~]$ sudo /usr/local/sbin/kpatch list
Loaded patch modules:
kpatch_version

Installed patch modules: