Somehow, I expected to have little to no work when I wanted to monitor an oVirt host over SNMP. One would expect this since oVirt is the upstream project for Red Hat’s Enterprise Virtualization (RHEV) which seems to be quite strong in the market. While it isn’t that hard to get information about using SNMP for oVirt or libvirt the outcome can be disappointing. There is some documentation about it but here you can find a more hands-on explanation.

Personally, I have a love-hate relationship with SNMP. While most products support SNMP and it gives you an quick way to monitor something, it can be a real pain in the ass the get the information which you want and how you want out of something. Especially in combination with non-existing Zabbix templates…

It’s clear that an oVirt (or libvirt) host needs some monitoring. By monitoring the underlying Linux OS, as you would do with any other Linux installation, you can already get a lot of useful information but there is just that little bit of hypervisor specific information that’s missing. Unfortunately, the possibilities to monitor libvirt over SNMP are rather poor but let’s stay positive and take what we can get.

Official documentation for libvirt-SNMP can be found here: http://wiki.libvirt.org/page/Libvirt-snmp

Access to libvirt on an oVirt host

By default, users do not have access to libvirt, for example with virsh, on an oVirt machine. This is a good thing since you want to control your VM’s and everything around it in a single, consistent, place. In order for us to get the correct information over SNMP, the SNMP subagent for libvirt will require access.

As you can see, by default, libvirt is asking us to authenticate before we can get some information via virsh. Even when we run the command as root.

[jensd@ovirt ~]$ sudo virsh -c qemu:///system 'list'
Please enter your authentication name:

The SNMP subagent which we will install later, will run as root, that’s a requirement for it to correctly communicate with the “normal” snmpd. Logically, the subagent will require access to libvirt to get information and entering a username and password each time we start the subagent isn’t really an option.

To allow access, edit /etc/libvirt/libvirtd.conf, don’t forget to take a backup before you do so :)

# Set an authentication scheme for UNIX read-write sockets
# By default socket permissions only allow root. If PolicyKit
# support was compiled into libvirt, the default will be to
# use 'polkit' auth.
#
# If the unix_sock_rw_perms are changed you may wish to enable
# an authentication mechanism here
auth_unix_rw = "none"

Uncomment the the auth_unix_rw = “none” parameter in the file.

After saving the changes, restart libvirtd (it seems that a reload is not working). In my experience this didn’t cause my virtual machines to stop or pause so it could be used (with care) in a production environment.

[jensd@ovirt ~]$ sudo /etc/init.d/libvirtd restart
Stopping libvirtd daemon:                                  [  OK  ]
Starting libvirtd daemon:                                  [  OK  ]

After restarting, let’s test to see if root has proper access to libvirt:

[jensd@ovirt ~]$ sudo virsh -c qemu:///system 'list'
 Id    Name                           State
----------------------------------------------------
 6     testvm1                        running
 16    testvm2                        running
 17    testvm3                        running

The communication with libvirt will be done using unix-sockets, which is basically a file. As we can see, root will have access after our changes. If we would like to give access to other users, we can add them to the group-owner (qemu).

[jensd@ovirt ~]$ ll /var/run/libvirt/
total 4
srwxrwx---. 1 root qemu    0 Jan 22 15:58 libvirt-sock
srwxrwxrwx. 1 root qemu    0 Jan 22 15:58 libvirt-sock-ro
drwxr-xr-x. 2 root root 4096 Feb  5 15:29 qemu

To test this with group ownership, let’s add the current non-superuser to the group:

[jensd@ovirt ~]$ sudo usermod -a -G qemu jensd

In order for the group membership to become active, we need to reloging or just su to ourselves:

[jensd@ovirt ~]$ su jensd
Password:
[jensd@ovirt ~]$ virsh -c qemu:///system 'list'
 Id    Name                           State
----------------------------------------------------
 6     testvm1                        running
 16    testvm2                        running
 17    testvm3                        running

As you can see, the user jensd has access to libvirt now too.

Install and configure SNMPd

Since we want to monitor libvirt over SNMP, we need to install a basic SNMP agent on our host in order to have SNMP working.

Install the required packages from the repositories:

[jensd@ovirt ~]$ sudo yum -y install net-snmp snmpwalk
...
Complete !

Configure the SNMP daemon for read-only access to our system: edit  /etc/snmp/snmpd.conf to look like this:

com2sec snmpuser default      public
group   snmpgroup v1          snmpuser
group   snmpgroup v2c         snmpuser
view    all          included        .1
access  snmpgroup ""      any       noauth    exact  all none none
syslocation Datacenter
syscontact  Root <root@localhost>
dontLogTCPWrappersConnects yes
includeAllDisks 10%
disk /dev 0
disk /dev/shm 0
master agentx

As you can see, I’ve chosen to use the communitystring “public” for read-only access. The rest of the snmpd.conf is rather basic except for the last line. Since we want to monitor libvirt via an SNMP subagent, we need AgentX master support in the “normal” snmpd daemon.

After saving the config, let’s start snmpd:

[jensd@ovirt ~]$ sudo /etc/init.d/snmpd start
Starting snmpd:                                            [  OK  ]

As you can see in the syslog, AgentX master support was turned on.

Feb  6 08:25:08 ovirt snmpd[28323]: Turning on AgentX master support.
Feb  6 08:25:08 ovirt snmpd[28323]: Created directory: /var/agentx

To get information over SNMP from another machine, we need to open UDP port 161 in our firewall:

[jensd@ovirt ~]$ sudo iptables -I INPUT -p udp -m udp --dport 161 -j ACCEPT
[jensd@ovirt ~]$ sudo iptables-save | sudo tee /etc/sysconfig/iptables

Install and configure libvirt-snmp

There are two options to install libvirt-snmp. Libvirt-snmp can be found in the Base repository of CentOS and RHEL but there is a newer version which claims to fix some bugs available.

I’ll continue with the version from the repositories but added a walk trough on how to use the newer version in the end of this post. So if you feel like you would like to use the newer version, execute the steps listed in the end and come back to this point to continue.

Let’s start by installing the libvirt-snmp package from the repositories (skip this if you want the newer version):

[jensd@ovirt ~]$ sudo yum -y install libvirt-snmp
...
Complete !

The package is rather basic and doesn’t provide an init-script so we’ll create one ourselves. Since oVirt is still recommending to run on CentOS 6, I didn’t choose to create a systemd-service file.

Create /etc/init.d/ovirt-snmp with the following contents:

#!/bin/bash
# chkconfig: 345 95 20
# description: Starts libvirtMib SNMP subagent
# Make sure that snmpd is running with the correct agentX-options
# processname: libvirtMib_subagent

. /etc/init.d/functions

start() {
        echo -n "Starting ovirt-snmp: "
        echo -e "\n"
        LIBVIRT_DEFAULT_URI="qemu:///system"
        /usr/bin/libvirtMib_subagent
        touch /var/lock/subsys/ovirt-snmp
        return 0
}

stop() {
        echo -n "Shutting down ovirt-snmp: "
        killproc libvirtMib_subagent
        echo -e "\n"
        rm -f /var/lock/subsys/ovirt-snmp
        return 0
}

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    status)
        status
        ;;
    restart)
        stop
        start
        ;;
    reload)
        reload
        ;;
    *)
        echo "Usage: ovirt-snmp {start|stop|status|restart"
        exit 1
        ;;
esac
exit $?

After saving the script, which is quite basic (and dirty), let’s install and activate the service:

[jensd@ovirt ~]$ sudo chmod +x /etc/init.d/ovirt-snmp
[jensd@ovirt ~]$ sudo chkconfig --add /etc/init.d/ovirt-snmp
[jensd@ovirt ~]$ sudo /etc/init.d/ovirt-snmp start

Test libvirt-snmp

No that we got all components installed and running, we can test getting libvirt or ovirt information over SNMP using snmpwalk:

[jensd@ovirt ~]$ snmpwalk -m ALL -v 2c -c public -OX 127.0.0.1 libvirtMIB
LIBVIRT-MIB::libvirtGuestName[STRING: 689e920b-315b-4ec2-bb95-af7e2b516ec5] = STRING: "testvm1"
LIBVIRT-MIB::libvirtGuestName[STRING: cedaf8c6-419-41e1-8e43-569bc893126a] = STRING: "testvm2"
LIBVIRT-MIB::libvirtGuestName[STRING: dbee745a-b2fc-493e-8bfb-bf2d0b7b242a] = STRING: "testvm3"
LIBVIRT-MIB::libvirtGuestState[STRING: 689e920b-315b-4ec2-bb95-af7e2b516ec5] = INTEGER: running(1)
LIBVIRT-MIB::libvirtGuestState[STRING: cedaf8c6-419-41e1-8e43-569bc893126a] = INTEGER: running(1)
LIBVIRT-MIB::libvirtGuestState[STRING: dbee745a-b2fc-493e-8bfb-bf2d0b7b242a] = INTEGER: running(1)
LIBVIRT-MIB::libvirtGuestCpuCount[STRING: 689e920b-315b-4ec2-bb95-af7e2b516ec5] = Gauge32: 1
LIBVIRT-MIB::libvirtGuestCpuCount[STRING: cedaf8c6-419-41e1-8e43-569bc893126a] = Gauge32: 1
LIBVIRT-MIB::libvirtGuestCpuCount[STRING: dbee745a-b2fc-493e-8bfb-bf2d0b7b242a] = Gauge32: 4
LIBVIRT-MIB::libvirtGuestMemoryCurrent[STRING: 689e920b-315b-4ec2-bb95-af7e2b516ec5] = Gauge32: 2048
LIBVIRT-MIB::libvirtGuestMemoryCurrent[STRING: cedaf8c6-419-41e1-8e43-569bc893126a] = Gauge32: 2048
LIBVIRT-MIB::libvirtGuestMemoryCurrent[STRING: dbee745a-b2fc-493e-8bfb-bf2d0b7b242a] = Gauge32: 32768
LIBVIRT-MIB::libvirtGuestMemoryLimit[STRING: 689e920b-315b-4ec2-bb95-af7e2b516ec5] = Gauge32: 2048
LIBVIRT-MIB::libvirtGuestMemoryLimit[STRING: cedaf8c6-419-41e1-8e43-569bc893126a] = Gauge32: 2048
LIBVIRT-MIB::libvirtGuestMemoryLimit[STRING: dbee745a-b2fc-493e-8bfb-bf2d0b7b242a] = Gauge32: 32768
LIBVIRT-MIB::libvirtGuestCpuTime[STRING: 689e920b-315b-4ec2-bb95-af7e2b516ec5] = Counter64: 1071200000000
LIBVIRT-MIB::libvirtGuestCpuTime[STRING: cedaf8c6-419-41e1-8e43-569bc893126a] = Counter64: 799630000000
LIBVIRT-MIB::libvirtGuestCpuTime[STRING: dbee745a-b2fc-493e-8bfb-bf2d0b7b242a] = Counter64: 20829850000000
LIBVIRT-MIB::libvirtGuestRowStatus[STRING: 689e920b-315b-4ec2-bb95-af7e2b516ec5] = INTEGER: active(1)
LIBVIRT-MIB::libvirtGuestRowStatus[STRING: cedaf8c6-419-41e1-8e43-569bc893126a] = INTEGER: active(1)
LIBVIRT-MIB::libvirtGuestRowStatus[STRING: dbee745a-b2fc-493e-8bfb-bf2d0b7b242a] = INTEGER: active(1)

To get a more nice output for the above, we van use snmptable:

[jensd@ovirt ~]$ snmptable -m ALL -v 2c -c public -Cb 127.0.0.1 libvirtGuestTable
SNMP table: LIBVIRT-MIB::libvirtGuestTable

              Name   State CpuCount MemoryCurrent MemoryLimit        CpuTime RowStatus
 "testvm1" running        1          2048        2048  1072560000000    active
 "testvm2" running        1          2048        2048   801200000000    active
 "testvm3" running        4         32768       32768 20832120000000    active

In case you want to get the information from another host that doesn’t have the libvirtMIB installed, you can download it here: http://jensd.be/download/LIBVIRT-MIB.txt or use the OID .1.3.6.1.4.1.12345

Monitor oVirt or libvirt with Zabbix

To monitor this information from within Zabbix, I created a Zabbix-template. You can download it from the Zabbix templates website or directly from: http://jensd.be/download/Zabbix_ovirt_template.zip Don’t forget to read the readme and create the value mapping before importing the template.

Compile and install the latest version of libvirt-snmp

As mentioned earlier in the post, there is a newer version of libvirt-snmp available that isn’t in the official repositories yet. You can follow these steps to download, build and install it. The build-script generates an RPM together with the binaries so it’s recommended to build the RPM on another system to keep your oVirt-host clean.

Install the required packages in order to build libvirt-snmp:

[jensd@ovirt]$ sudo yum install net-snmp-perl net-snmp net-snmp-utils net-snmp-devel libvirt-devel autoconf automake gcc make
...
Complete !

Then, download the latest version, either from the GIT-repository of libvirt of from here: http://jensd.be/download/libvirt-snmp-latest.tar.gz. After downloading, untar the file:

[jensd@ovirt]$ wget http://jensd.be/download/libvirt-snmp-latest.tar.gz
--2015-02-06 08:58:22--  http://jensd.be/download/libvirt-snmp-latest.tar.gz
Resolving jensd.be... 46.30.212.1
Connecting to jensd.be|46.30.212.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 67728 (66K) [application/x-tar]
Saving to: “libvirt-snmp-latest.tar.gz”

100%[===========================================>] 67,728      --.-K/s   in 0.07s

2015-02-06 08:58:22 (950 KB/s) - “libvirt-snmp-latest.tar.gz” saved [67728/67728]
[jensd@ovirt]$ tar -xvzf libvirt-snmp-latest.tar.gz
...

Now, cd into the directory which was untarred and execute the autobuild.sh script:

[jensd@ovirt]$ cd libvirt-snmp-HEAD-9ed687e/
[jensd@ovirt libvirt-snmp-HEAD-9ed687e]$ ./autobuild.sh

test -n "$1" && RESULTS=$1 || RESULTS=results.log
: ${AUTOBUILD_INSTALL_ROOT=$HOME/builder}

# Make things clean.
test -f Makefile && make -k distclean || :

rm -rf build
mkdir build
cd build

MIBDIR=$AUTOBUILD_INSTALL_ROOT/share/snmp/mibs
mkdir -p $MIBDIR

../autogen.sh --prefix=$AUTOBUILD_INSTALL_ROOT \
  --with-mibdir=$MIBDIR
configure.ac:8: installing `./compile'
...
Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.hU3ZoB
+ umask 022
+ cd /home/jensd/rpmbuild/BUILD
+ rm -rf libvirt-snmp-0.0.3
+ exit 0
[jensd@ovirt libvirt-snmp-HEAD-9ed687e]$

When the build finishes successfully, you can find the binary files in build/srv:

[jensd@ovirt libvirt-snmp-HEAD-9ed687e]$ ls build/src/
libvirtMib_subagent
libvirtMib_subagent-libvirtGuestTable_data_access.o
libvirtMib_subagent-libvirtGuestTable_data_get.o
libvirtMib_subagent-libvirtGuestTable_data_set.o
libvirtMib_subagent-libvirtGuestTable_interface.o
libvirtMib_subagent-libvirtGuestTable.o
libvirtMib_subagent-libvirtMib_subagent.o
libvirtMib_subagent-libvirtNotifications.o
libvirtMib_subagent-libvirtSnmpError.o
libvirtMib_subagent-libvirtSnmp.o
Makefile

The autobuild.sh-script also create an RPM in ~/rpmbuild which is easier to install (and remove) on RHEL-based systems:

[jensd@ovirt libvirt-snmp-HEAD-9ed687e]$ ls ~/rpmbuild/RPMS/x86_64/
libvirt-snmp-0.0.3-1.el6.jendep1423209549.x86_64.rpm

To install the RPM on this or another system:

[jensd@ovirt libvirt-snmp-HEAD-9ed687e]$ sudo yum -y install ~/rpmbuild/RPMS/x86_64/libvirt-snmp-0.0.3-1.el6.jendep1423209549.x86_64.rpm