If you are building a script, application or anything else that collects measurements or data over time intervals, a time-series database might be a good match for you. In a TSDB you simply store the data and read it back for analysis. For these use cases, there is no need for updates or changes to the data as it’s time-based. The data can’t change back in the past. InfluxDB is the most popular of these time-series databases and this article describes how to get started with it, on Debian or Ubuntu, and how to use it from Python.Continue reading
Category Archives: Linux
Upgrade Debian 9 (Stretch) to Debian 10 (Buster)
When there is a need to upgrade an existing Linux installation that has been running for some time, there is always a certain fear of breaking things. Especially if the machine has been doing it’s job fine for a long time. Unfortunately we can’t escape these kind of things and in most cases it’s better to move to a new version before it gets unsupported. Fortunately the procedure and success of this process has been greatly improved with recent versions of (Debian) Linux. This post will take you through the required steps to upgrade an existing Debian 9 to the current stable Debian 10 installation.Continue reading
Read data from the Belgian digital meter through the P1 port
Like some other Belgians, unfortunately, my analog electricity meter was replaced by a digital one since a few months. Initially, I was quite happy with the change as it allows a much easier look into your consumption and solar panel injection to the net. But exactly that last turned out to be a bummer as the government decided to not keep their word regarding reversing consumption with solar panel injection. Nevertheless, I have the thing so I might as well take advantage of it. This post describes how to connect to and use the data from the P1 port on a digital electricity meter in Belgium.Continue reading
Data lezen van de P1 poort op de Belgische digitale elektriciteitsmeter
Zoals enkele andere Belgen werd, jammer genoeg, ook mijn analoge elektriciteitsmeter vervangen door een digitaal exemplaar. Initieel was ik daar best wel blij mee gezien het hiermee gemakkelijker is om het verbruik en opbrengst van de zonnepanelen bij te houden. Dat laatste is net he probleem, gezien de overheid heeft beslist dat de teller niet langer kan “terugdraaien”. Los daarvan, ik heb de meter nu dus kan ik er zowel gebruik van maken. Dit artikel beschrijft hoe je de meter kunt verbinden met je computer en hoe je de data van de P1 poort kunt gebruiken voor een Belgische digital elektriciteitsmeter.Continue reading
Cross compiling for arm or aarch64 on Debian or Ubuntu
ARM is gaining more and more traction and is growing a lot in popularity. It’s not always possible to build directly on these ARM-based devices, especially when they are limited in resources. The majority of build and developer machines are still on x86 and by using cross compiling, it is possible to build binaries or executables usable on another architecture. For example, to use your standard PC, most likely x86, to build something that is usable on another machine or device that’s on another architecture, like ARM. In this post, I’ll explain how to do cross compiling for 32bit ARM (arm) or 64bit ARM (aarch64) using Debian 10 or Ubuntu 20.04 LTS.Continue reading
Forward a TCP port to another IP or port using NAT with nftables
Besides using NAT for accessing the internet with multiple machines using a single IP address, there are many other uses of NAT. One of them is to forward all traffic that is sent to a certain TCP port to another host. In practice, this technique can be used to test a service on a new host without adjusting anything on the client. The users or the clients do not need to be pointed to a new machine in order to test it. When the test would be unsuccessful, removing the NAT-rule is all it takes to switch back.Continue reading
BIOS or SPI programming on Windows or Linux using a CH341a MiniProgrammer
Without a working BIOS, even the most powerful hardware can’t be brought to life. In some rare cases, the BIOS chip in your system can become corrupt, which will return your system completely unusable. To fix this situation but also for other reasons like simply a backup, tuning, modding or enabling hidden features, flashing your BIOS ROM is interesting. Recently I got in the situation where the BIOS chip in my Lenovo X1 Yoga got corrupt. To fix this, I purchased the CH341a MiniProgrammer. In this post you can find the documented steps I took to reflash the BIOS and bring my laptop back to life.Continue reading
Install a newer kernel in Debian 10 (buster) stable
When using the latest version of Debian 10 stable, also known as buster, even with all updates installed, by default, you can’t get the most recent kernel via the standard repositories in your package manager. While the idea of using Debian stable is to remain stable and rather conservative, there are several benefits with installing a newer kernel and in some cases it’s the only option to get the OS to support all your hardware. The risk and impact on stability is small and the process is rather simple.Continue reading
Build/configure a Linux-based NAS
There are several options when you would like a NAS or Network Attached Storage. One of them is, instead of using dedicated NAS hardware, to build and configure your own (Linux-based) NAS system, using a standard distro. Getting dedicated NAS hardware has an advantage in terms of ease of use. On the other hand, building your own system and configuring the OS yourself has the advantage of flexibility, potential additional features and might even save you some money. But what about the power consumption? This post covers my journey for a self-built, Linux-based, NAS for homeContinue reading
Build Docker images for deployment on non-x86 architecture
When building a Docker container image, in most cases, the machine where you build it on runs on the same CPU architecture as where you eventually want to use it. Today, more and more, there is a need for deploying containers on a different architecture as where it was built on. The most common scenario would be to build on your regular x86 desktop and to deploy to an ARM variant.Continue reading
Install and use CentOS 8 or RHEL 8 as KVM virtualization host
There are many possibilities to turn your machine into a host for virtual machines, emulating one or more separate and isolated computers. Among the popular commercial products from VMWare or Hyper-V on Windows, KVM is a great and mature alternative for full virtualization with great stability and performance. This guide will explain you step by step, from the start how to get this up and running.Continue reading
Install and use Zabbix 3.2 on CentOS 7 or RHEL 7
Zabbix is a monitoring tool which is widely used in all kinds of environments. Zabbix is very flexible, information can be retrieved using HTTP/SNMP or by installing a Zabbix agent on the machines to monitor, and allows a lot of customisation. Unfortunately this also means that the learning curve can be rather high. This post will go a little deeper into the installation process and the first steps in Zabbix.
Install a newer kernel in Debian 9 (stretch) stable
When using the latest version of Debian 9 stable, even with all updates installed, by default, you can’t get a very recent kernel via the standard repositories in your package manager. While the idea of using Debian stable is to remain stable and rather conservative, there are several benefits with installing a newer kernel and in some cases it’s the only option to get the OS to support all your hardware. The risk and impact on stability is small and the process is rather simple.
Install a newer kernel in CentOS 7
When using the latest version of CentOS 7, even with all updates installed, by default, you can’t get a very recent kernel via the standard repositories in your package manager. While the idea is to remain stable and rather conservative, there are several benefits with installing a newer kernel and in some cases it’s the only option to get the OS to support all your hardware. The risk and impact on stability is small and the process is rather simple.
Cross compiling for ARM with Ubuntu 16.04 LTS
The goal of cross compiling is to compile for one architecture on machine running another one. In this post, I’ll try to explain the steps required to be able to compile software that is executable on ARM-based hardware using a “normal” x64-based PC or virtual machine. ARM-based devices are usually limited in processing power and are mostly running stripped-down, embedded versions of Linux. This makes it sometimes difficult to compile on the target device directly.
Install the Zabbix 3.0.0 beta from source on CentOS or RHEL
It’s been a while since I was busy with Zabbix but I have been following their alpha releases for version 3 quite closely. Now that Zabbix released the first beta release for version 3, I thought that it was time to have a good look at this new and improved version. Zabbix is used a lot and more and more companies count on it as their primary monitoring solution. In this post, I’ll explain how to get started with the beta of the latest Zabbix release on CentOS 7 or RHEL 7.
Create a minimal SSH-accessible chroot jail with Dropbear and BusyBox on CentOS
Sometimes you just need some kind of setup which is not so standard. This was the case which lead to this post. For some testing, I needed a minimal environment containing only a BusyBox shell and that was accessible via Dropbear SSH-server. Probably this setup is quite useless for daily use but it can be used as a base for testing with a minimum set of libraries available. Using this setup gives you the flexibility to experiment with a minimal environment without rebuilding small Linux images.
Linux live kernel patching with kpatch on CentOS 7
Live or dynamic kernel patching allows you to patch a running kernel with no impact to running applications and without rebooting the system. Since the announcement and release of Linux kernel version 4, live kernel patching got got a lot of extra attention. Most probably this is because there weren’t a lot of big changes that one would expect with an increase in the version number. While v4 of the kernel does introduce some new stuff related to live or dynamic kernel patching, Live kernel patching was and is already available for earlier kernel releases. In this post, I’ll try to explain the differences between the new and old method and more important how to do live patching of a v3 kernel on CentOS 7.
SELinux in a practical way
SELinux is often seen as an evil, complex, unnecessary and especially annoying security component which exists in a lot of Linux distributions. Often you can hear something like: “Disable SELinux and try again” or , “The first thing I do on a new server is to disable SELinux”. The problem with SELinux is that it looks very complex and that it looks like you need to spend ages to understand it. In this post, I’ll try to explain a few basic SELinux principles and especially focus on daily, practical problems related to SELinux and their solutions. Don’t forget that there’s a very good reason for SELinux and it would be a shame to not use it.
Setup roundcube webmail as a proxy for Gmail
Recently, I was asked to find a solution or workaround for people using Gmail and are visiting countries where there’s limited or no access to Gmail. I’m not talking about bad internet connection quality but about an explicit (government) block on Gmail or related websites. An example is the block on Gmail by the Chinese Great Firewall. While there is a possibility to avoid the limitation by using a normal proxy or VPN connection, those methods are also actively blocked and monitored. A workaround is to setup your own webmail, as a proxy for Gmail.
Setup a redundant PostgreSQL database with repmgr and pgpool
Using a database to store information is very common. The more information gets stored in a database, the more critical it becomes. At a certain point, the need for a redundant, high(er) available database-setup raises. When you’re using PostgreSQL, you can setup streaming replication quite easily in order to have your data redundant on two or more nodes. But wouldn’t it be a shame if you wouldn’t use the copies of your data to balance the load of your primary database-server? In this post, I’ll try to explain how to setup a redundant PostgreSQL database with load balancing.
Tips & tricks for Ansible
Ansible is a very powerful configuration management tool. It can be used to deploy settings and software to one or more (virtual) machines. Using a CM-tool ensures that every deployment is done in exactly the same way. This doesn’t only make your job as a system engineer a whole lot easier, it also results in a more stable environment since every machine is consistent and more predictable. I’m using Ansible intensively for quite some time now and found that it was time to write a post about some small, but usually hard to find or annoying to solve, problems or challenges.
Complile and use a realtime kernel on CentOS 7 or RHEL 7
While the need for a realtime kernel or kernel-rt hasn’t been very high recently, there are these special cases where you really need one. A “standard” linux kernel is fast but also balanced in order to treat all workload fair and give each kind of task an equal share of the system resources. A realtime kernel allows you to squeeze that extra bit of performance out of the kernel. It basically allows you to tune it in a more aggressive way in regards to timing and priorities. Since I couldn’t find any clear instructions on how to get a realtime kernel for CentOS 7 and RHEL 7, I decided to write them myself.
Install and use Spacewalk 2.4 on CentOS 7 and RHEL 7
Spacewalk is the upstream-project for Redhat Satellite. It’s a tool which is mainly used to list, deploy and manage packages and updates to Fedora, CentOS, SLES and even Debian installations. Recently a new version, 2.4, came out, so let’s have a look at the installation procedure and how to perform some basic Spacewalk actions
Install oVirt on CentOS 7
Recently, as you could see in previous posts, oVirt got to my attention. oVirt is the upstream project of Red Hat Enterprise Visualization or RHEV. While oVirt it isn’t providing everything that competitor VMWare ESXi does, it comes close and is a very good alternative for smaller or home setups. For a long time, oVirt wasn’t supported on Enterprise Linux 7 (like RHEL 7 or CentOS 7) but since a short time it is. Here you can find how to install oVirt and get started.
Install the Cinnamon desktop environment on CentOS 7
As you might know, Linux is everything about diversity. With the numerous desktop environments available nothing is different in that area. Besides Mate and XFCE, Cinnamon is one of my favorite desktop environments on Linux. Honestly, I can’t really choose. Cinnamon is fast, (not too) fancy, customizable and familiar in some way. Here is how to install Cinnamon on a fresh minimal CentOS 7 installation.
Create a PXE bootserver to install multiple Linux distributions
When regularly installing Linux hosts or VM’s, it easily becomes annoying to constantly burn CD’s/DVD’s or mount ISO’s for all the Linux distributions that you want to deploy. Especially if you want to keep them current or customize them you’ll end up with a whole lot of discs. Booting your installations from the network, using a PXE boot server, makes life a lot easier and isn’t very hard to setup. In this post I’ll explain how to setup such a PXE boot server that is able to provide multiple Linux distribution installations for deployment over the network.
Setup an FTP-server with quotas on RHEL or CentOS 6 or 7 with proftpd
Using FTP actually should be avoided whenever that’s possible but sometimes it’s just the most handy and convenient way of transferring files. In most cases, your FTP-users will be able to upload files to the FTP-server. To avoid that some users would fill up the complete machine, you can use quotas. In this post, I’ll describe how to setup a basic proftpd FTP-server with quotas on RHEL or CentOS 6 and 7.
Move a guest from VMWare ESX to oVirt or RHEV
When working in an environment where you have both a VMWare hypervisor and an oVirt or RHEV hypservisor or you want to migrate away from VMWare ESX to oVirt or RHEV, you will need a way to move or copy your current virtual machines from one to another. Unfortunately this can’t be done without downtime but at least it can be done. In this post I’ll try to explain how to successfully move a VM running on ESX or vCenter to an oVirt or Red Hat Enterprise Virtualization host.
Create a new virtual machine in oVirt with Python using the API
Recently, I started to work or play, it’s a matter of definition, a little more with oVirt. oVirt is less known than VMWare but it’s the upstream project for Red Hat’s Enterprise Virtualization (RHEV) and based on libvirt. In an earlier post, I explained how to create VM’s on VMWare vSphere environments using Python and the VMWare API. In this post, I’ll explain how to acomplish the same using the oVirt API. It turned out to be easier than I expected.
Monitor oVirt or libvirt with SNMP and Zabbix
Somehow, I expected to have little to no work when I wanted to monitor an oVirt host over SNMP. One would expect this since oVirt is the upstream project for Red Hat’s Enterprise Virtualization (RHEV) which seems to be quite strong in the market. While it isn’t that hard to get information about using SNMP for oVirt or libvirt the outcome can be disappointing. There is some documentation about it but here you can find a more hands-on explanation.
Configure two network cards in a different subnet on RHEL 6, RHEL 7, CentOS 6 and CentOS 7
When configuring a Linux host running either Red Hat Linux 6, Red Hat Linux 7, CentOS 6 or CentOS7 with two network interface cards (NIC) that each have an IP address in a different network or subnet, you could end up in a situation where one of the IP addresses isn’t reachable outside it’s own network. Both IP’s will be responding to a ping from another host in the same network as those IP addresses but only one is responding to ping from another network. On most other distributions, like Debian, this issue, which is caused by asymmetric routing, doesn’t seem to exist.
Compile the Huawei ES3000 PCIe SSD driver for CentOS 7 or RHEL7
The Huawei ES3000 is an SSD accelerator card connected to the PCIe bus which delivers exceptional performance. Today, I found out that Huawei doesn’t provide any ready made drivers to use a Huawei ES3000 PCIe SSD on a system running a Linux kernel > 3.x. This means that it’s not possible to use it on RHEL 7 or CentOS 7 or any other modern Linux distribution that uses a kernel newer than version 220.127.116.11. Fortunately it’s not difficult to compile the driver, install it and start to use the SSD accelerator with a newer kernel.
Expand or grow a file system on a Linux VMWare VM without downtime
When your VM’s file systems are running out of space and you want to provide more space to your VM and you can’t afford any downtime, there are basically two options. Either you delete some files on the file system or you expand your current file system. Expanding your current file system can be accomplished by growing an existing virtual disk or adding a new virtual disk. Besides cleaning up, both other solutions work fine and I’ll try to explain how to get both of them active without rebooting the system or any downtime.
Create a new virtual machine in Vsphere with Python, Pysphere and the VMWare API
The VMWare API is very extensive and allows you to do almost all operations that are possible with VMWare using API calls. In order to be able to easily create and deploy new virtual machines, it can be a good idea to standardize and create VM’s using a (Python) script that calls the API. In this post, I will give some examples on how to easily create a new VM using Pysphere and the VMWare API.
Clone a z/VM Linux user/guest instance from another Linux guest
Cloning a virtual machine or user as it is called in z/VM terms from within another guest/user running Linux can be used in a lot of different scenario’s. Especially when the people managing the z/VM platform and virtual machines are less experienced with z/VM, it could be good to let them create a clone without the need to access a 3270 terminal. This scenario can also be used to let users do some self service and provide them with new instances without intervention.
Install and use the openlava job scheduler and openlava web GUI on CentOS
Scheduling on Linux mostly happens with cron or any of it’s variations. Although cron is very powerful, it lacks a few features to use it in a flexible way and especially when you want to create dependencies or “communicate” with jobs running on other hosts, it has it’s shortcomings. Cron wasn’t really designed with those features in mind. Fortunately there are a few nice schedulers out there which can be used to overcome those limitations. A few of them are SOS Jobscheduler, GNUBatch and openlava. Openlava is a limited open source fork of LSF which is now owned by IBM. Openlava doesn’t come with a GUI but there is another project, Openlava web which enables control over openlava via a web interface.
Write to and manage syslog messages with logger and rsyslog
Syslog is the target where you want all log message to go on all systems that you manage. Almost all Linux distributions use a syslog implementation to gather messages. Recently, rsyslog became the most used syslog-implementation for Linux. Messages can be saved locally or sent to a remote syslog server. When creating your own applications or tools or when you want to log messages coming from processes that don’t support writing to syslog directly, you can use Logger.
Use DRBD in a cluster with Corosync and Pacemaker on CentOS 7
When configuring a cluster, you want tot keep managing the server as simple as possible. Theoretically, the results given by any node in the cluster should be equal as you want the cluster to be transparent to the end-user. Part of doing this, is having the same data available on every node of the cluster when it’s active. One way to do this, is using a central file-share, for example over NFS but this also has disadvantages. Another way is to have a distributed file system that stays on the nodes itself. DRBD is one of them. This post explains how to integrate DRBD in a cluster with Corosync and Pacemaker.
Forward a TCP port to another IP or port using NAT with Iptables
Besides using NAT for accessing the internet with multiple machines using a single IP address, there are many other uses of NAT. One of them is to forward all traffic that is sent to a certain TCP port to another host. In practice, this technique can be used to test a service on a new host without adjusting anything on the client. The users or the clients do not need to be pointed to a new machine in order to test it. When the test would be unsuccessful, removing the NAT-rule is all it takes to switch back.
Find and recover space used by deleted files, shown by df but not by du
When you’re running low on space on a file system, that can cause various unexpected behavior of the system, depending on which file system is filling up. For me, when that happens, I usually first issue a disk free (df) to see which is the file system that is almost full. Once I know which file system, I go and search which files take up the most space in that file system and take action. Sometimes, df show that a file system is almost full while, when summing up all the space by all files doesn’t even come near that value.
Install and use Debian Wheezy as Xen virtualization host
Earlier, I explained how to setup CentOS or RHEL as a KVM virtualization host. You can find that explanation here. It also contains some basic terminology about virtualization which is also applicable for Xen. When talking about KVM, somehow, I immediately associate it with the Red Hat family just as when you talk about Xen, I associate it to Debian derivatives. So for this post, I’ll use Debian to install a host that will run Xen-VM’s by using paravirtualization.
Start with a simple 2-node OpenStack setup with KVM
OpenStack is something that gets more and more in the picture and even if you’re only a little interested in the latest technologies, you must have heard from OpenStack here or there. But what is it exactly and more important, how does it work practically. The best way to figure that out is just to get going with it and try to install it and play around. Here you can find a brief explanation and a tutorial or waltrough to deploy a small OpenStack environment on top of CentOS 7 or RHEL 7.
Read and write a CMS minidisk on a z/VM system from a user/guest running Linux
When you’re managing a z/VM host running multiple Linux images, it can be interesting to have access to the data which resides on a CMS owned minidisk. Such type of minidisk is probably used to control the z/VM related configuration of the guest itself in some way. Also, when scripting, the explained technique can be used to read or write data from and to one of the minidisks. One example could be to change the PROFILE.EXEC from a user from within the user itself.
Terse, unterse and transfer datasets between z/OS and other platforms via FTP
Tersing files can be compared to tarring, zipping, rarring files on the x86 platform. It allows you to store a file or multiple files (members) in an archive. The archive is easily transferable and when the data is unpacked, it is guaranteed to be the same as the original source. Especially when transferring PDS’es and variable blocked (VB, VBS) datasets, this is important. Transferring normal datasets over FTP through other platforms from z/OS can be challenging an this article should provide some help.
Execute queries on a Microsoft SQL server from the Linux CLI with ODBC and Kerberos authentication
A while ago, the Windows-world and the Linux-world were not the best friends in communicating with each other. Especially not when it came to proprietary stuff like Microsoft SQL server. in contrast to everybody’s expectations, somewhere end 2011, beginning 2012, Microsoft released an ODBC driver for SQL server for Linux. This driver allows executing queries from a Linux machine to a Microsoft SQL Server database. The driver can be used in combination with Kerberos tickets and AD authentication to execute queries.
Use inotify-tools on CentOS 7 or RHEL 7 to watch files and directories for events
Today, I discovered that the package inotify-tools is nowhere to be found in standard CentOS7 or RHEL7 repositories. Alternatives like incron seem to be absent as well. The inotify-tools can be used to watch a directory or file for activity and take an action when a file is changed, added, edited or simply read. You can find a workaround (or call it solution) for the absence of the inotify-tools in the repositories here.
Mount Windows (CIFS) shares on Linux with credentials in a secure way
In almost all cases, when mounting a CIFS-share on a Linux host, you will need to supply some credentials. Either you could enter the credentials by hand every time you need the share or add the credentials to /etc/fstab to automatically mount the share. Entering the password manually is secure but not comfortable, leaving the password in /etc/fstab is comfortable but not secure since the file /etc/fstab is world readable.
Spacewalk 2.2 installation on CentOS 7
Spacewalk is the upstream-project for Redhat Satellite. It’s a tool which is mainly used to list, deploy and manage packages and updates to Fedora, CentOS, SLE and even Debian installations. Currently, there is no official documentation on how to run Spacewalk on a el7 based installation like CentOS 7. The installation is not a straightforward as one might think but it is possible.
Install and use CentOS 7 or RHEL 7 as KVM virtualization host
When thinking about virtualization, everybody immediately thinks about VMWare. And it must be said, the product they offer is very decent but also comes with a “decent” price. As an alternative, it’s worth looking into KVM for your virtualization. As with the VMWare product range, KVM offers full virtualization and it can compete with VMWare regarding stability and performance.