Clone a z/VM Linux user/guest instance from another Linux guest

Cloning a virtual machine or user as it is called in z/VM terms from within another guest/user running Linux can be used in a lot of different scenario’s. Especially when the people managing the z/VM platform and virtual machines are less experienced with z/VM, it could be good to let them create a clone without the need to access a 3270 terminal. This scenario can also be used to let users do some self service and provide them with new instances without intervention.

This post will describe how to create a new z/VM user and provide it with the data from another instance, customize it and prepare it to be bootable. All of this should be done from a Linux guest running on the sam z/VM environment without the need to open a terminal or have any deep knowledge of z/VM.

For this explanation, I was using a guest (or user as it should be called in z/VM terms) is running Linux for zSeries (zLinux as it once was) on a z/VM host. I was using SLES11 running on z/VM 6.3. The system from where I will execute all steps is called the cloner. As a source for the new VM, I will use the volume that was the result of a minimal installation of SLES11 (Suse Linux Enterprise System). This source needs to be created once and can be used as much as you want to generate different clones.

Connect the cloner to the FCP-device of the z/VM machine

This first step is to connect the cloner to the FCP-devices on the z/VM host. In my example, I’m using different devices to connect to the LUN devices. Device 6022 will be used to connect to the source LUN (the result of the minimal install) and FCP device 6023 will be used to connect to the target LUN (the disk that will be used by the new VM).

Attaching devices requires the necessary privileges on the z/VM environment. To be sure that we can communicate from the guest (user) to the host (CMS), we need to make sure that kernel module vmcp is loaded in the cloner:

cloner:/ # modprobe vmcp

After loading vmcp, we can send commands to the z/VM host and attach two FCP-devices to our cloner. Cloner is the hostname of the Linux VM and also the z/VM user’s name.

cloner :/ # vmcp att 6022 cloner
cloner :/ # vmcp att 6023 cloner

After attaching the FCP devices, we need to put them online and we can check if the device correspondends with the correct WWPN’s of the SAN:

cloner:/ # echo "1">/sys/bus/ccw/drivers/zfcp/0.0.6022/online
cloner:/ # echo "1">/sys/bus/ccw/drivers/zfcp/0.0.6023/online
cloner:/ # lszfcp -H -P
0.0.6022 host3
0.0.6023 host2
0.0.800f host0
0.0.800f/0x50050768023595b2 rport-0:0-0
0.0.800f/0x50050768023595b1 rport-0:0-1
0.0.6023/0x50050768021595b1 rport-2:0-0
0.0.6023/0x50050768021595b2 rport-2:0-1
0.0.6022/0x50050768021595b2 rport-3:0-0
0.0.6022/0x50050768021595b1 rport-3:0-1

As you can see, the cloner was already connected to FCP device 8000F and now we can also use devices 6022 and 6023 to connect the cloner to the source and target LUN’s.

Connect the cloner to the source and target LUN

Now that we have access to a few FCP-devices, we can connect our cloner to the source and target LUN. The source LUN contains a minimal SLES install (installed with the GUI and bootable from it’s own device, configured with a fixed IP). The target LUN is a a new LUN of the same size.

  • LUN 0x003c000000000000 is the source for our clone
  • LUN 0x003d000000000000 is the target for our clone

Connect to the LUN using the previously connected FCP adapters:

cloner:/ # echo "0x003c000000000000">/sys/bus/ccw/drivers/zfcp/0.0.6022/0x50050768021595b1/unit_add
cloner:/ # echo "0x003d000000000000">/sys/bus/ccw/drivers/zfcp/0.0.6023/0x50050768021595b1/unit_add
cloner:/ # lszfcp -D
0.0.6022/0x50050768021595b1/0x003c000000000000 3:0:1:60
0.0.6023/0x50050768021595b1/0x003d000000000000 2:0:0:61
0.0.800f/0x50050768023595b2/0x003b000000000000 0:0:0:59

In the above output we can see that the LUN’s are connected to the system including the disk of the cloner itself.

To use the LUN’s on Linux, we need to know the device name in /dev. We can find these with lsscsi:

cloner:/ # lsscsi
[0:0:0:59] disk IBM 2145 0000 /dev/sda
[2:0:0:61] disk IBM 2145 0000 /dev/sdd
[3:0:1:60] disk IBM 2145 0000 /dev/sdc

Clone the data from source to target

In the previous step, we connected the source and target LUN to the cloner and got to know the device names that represent those LUN’s. The next step is to clone the data from our source LUN to the target LUN:

cloner:/ # dd if=/dev/sdc of=/dev/sdd bs=4096 conv=notrunc,noerror
2621440+0 records in
2621440+0 records out
10737418240 bytes (11 GB) copied, 55.4229 s, 194 MB/s

After cloning the data, we need to inform the cloner that we have a new partition configuration since we cloned the partitions from the source LUN to the target LUN

cloner:~ # partprobe

Customize the data of the clone

In the previous step we cloned the filesystem of another Linux instance so we need to customize the data on the cloned filesystem in order to make it bootable from another FCP device and LUN and to not interfere with the original system.

To customize the cloned filesystem, we’ll mount the cloned filesystem on /mnt/clone and adjust some files:

cloner:/ # mkdir /mnt/clone
cloner:/ # mount /dev/sdd2 /mnt/clone

Change the network configuration and hostname:

cloner:/ # vi /mnt/clone/etc/sysconfig/network/ifcfg-eth0
cloner:/ # vi /mnt/clone/etc/HOSTNAME

Change the IPL configuration to include the correct FCP-device and new LUN. The /dev/disk/by-path/ccw- corresponds to the portname and LUN which can be found with lszfcp -D for FCP-device 6023.

cloner:~ # vi /mnt/clone/etc/zipl.conf
[defaultboot]
default=SLES11_SP3

[SLES11_SP3]
image = /boot/image-3.0.76-0.11-default
target = /boot/zipl
ramdisk = /boot/initrd-3.0.76-0.11-default,0x2000000
parameters = "root=/dev/disk/by-path/ccw-0.0.6023-zfcp-0x50050768021595b1:0x003d000000000000-part2 hvc_iucv=8 TERM=dumb resume=/dev/disk/by-path/ccw-0.0.6023-zfcp-0x50050768021595b1:0x003d000000000000-part2"

Now all file adjustments have been done and we need to generate a new initrd and write a new bootsector that includes the correct FCP-device and LUN:

Let’s chroot into the cloned filesystem:

cloner:/ # mount --bind /proc /mnt/clone/proc
cloner:/ # mount --bin /dev /mnt/clone/dev
cloner:/ # chroot /mnt/clone

Re-create the initrd:

cloner:/ # mkinitrd
Kernel image: /boot/image-3.0.76-0.11-default
Initrd image: /boot/initrd-3.0.76-0.11-default
Root device: /dev/disk/by-id/scsi-360050768028087ffd800000000000041-part2 (/dev/sdd2) (mounted on / as ext3)
Resume device: /dev/system/swap
Kernel Modules: dm-mod dm-log dm-region-hash dm-mirror dm-snapshot scsi_mod scsi_dh scsi_dh_hp_sw scsi_dh_alua scsi_dh_emc scsi_dh_rdac mbcache jbd ext3 scsi_tgt scsi_transport_fc qdio zfcp linear crc-t10dif sd_mod
Features: dm block zfcp lvm2 resume.userspace resume.kernel
41247 blocks

Re-create the bootsector using zipl:

cloner:/ # zipl -V
Using config file '/etc/zipl.conf'
Target device information
Device..........................: 08:30
Partition.......................: 08:32
Device name.....................: sdd
Device driver name..............: sd
Type............................: disk partition
Disk layout.....................: SCSI disk layout
Geometry - start................: 1525760
File system block size..........: 4096
Physical block size.............: 512
Device size in physical blocks..: 19445760
Building bootmap in '/boot/zipl'
Adding IPL section 'SLES11_SP3' (default)
initial ramdisk...: /boot/initrd-3.0.76-0.11-default
kernel image......: /boot/image-3.0.76-0.11-default
kernel parmline...: 'root=/dev/disk/by-path/ccw-0.0.6023-zfcp-0x50050768021595b1:0x003d000000000000-part2 hvc_iucv=8 TERM=dumb resume=/dev/disk/by-path/ccw-0.0.6023-zfcp-0x50050768021595b1:0x003d000000000000-part2'
component address:
kernel image....: 0x00010000-0x008a6dff
parmline........: 0x008a7000-0x008a71ff
initial ramdisk.: 0x02000000-0x028237ff
internal loader.: 0x0000a000-0x0000a3ff
Preparing boot device: sdd.
Detected SCSI PCBIOS disk layout.
Writing SCSI master boot record.
Syncing disks...
Done.

At this stage, the filesystem is prepared and should be bootable from the changed LUN and FCP-device.

Let’s leave the chroot and cleanup mounts and devices:

cloner:/ # exit
cloner:~ # umount /mnt/clone/dev
cloner:~ # umount /mnt/clone/proc
cloner:~ # umount /mnt/clone
cloner:/ # echo "0">/sys/bus/ccw/drivers/zfcp/0.0.6023/online
cloner:/ # vmcp det 6023 cloner

Create a new z/VM user

The next thing to do is to create a new user (or VM) on the z/VM environment. In a previous post, I described how you can access CMS-files from a Linux guest running on z/VM. This technique will be used to create the user and a PROFILE.EXEC for that new user. More information here: Read and write a CMS minidisk on a z/VM system from a user/guest running Linux.

First, we need to load the kernel module fuse to be able to access the CMS-files:

cloner:~ # modprobe fuse

The next steps are to connect to minidisk 191 owned by user cloner, which is running our current Linux system. On the minidisk, we need to write a file that contains the configuration of the new z/VM user. Then, we’ll use dirmaint to create that new user on the z/VM machine. The new user will be called newvm.

Put minidisk 191 online and mount it:

cloner:~ # chccwdev -e 191
Setting device 0.0.0191 online
Done
cloner:~ # lsdasd 191
Bus-ID Status Name Device Type BlkSz Size Blocks
==============================================================================
0.0.0191 active dasda 94:0 FBA 512 0MB 1000
cloner:~ # mkdir /mnt/mdisk
cloner:~ # cmsfs-fuse --ascii /dev/dasda /mnt/mdisk

Create a new, empty file and set it’s attributes to FB 80:

cloner:~ # touch /mnt/mdisk/NEWVM.DIRECT
cloner:~ # setfattr -n user.record_format -v F /mnt/mdisk/NEWVM.DIRECT
cloner:~ # setfattr -n user.record_lrecl -v 80 /mnt/mdisk/NEWVM.DIRECT

Create a temporary file and add the configuration for the new user to the NEVWM.DIRECT file. Make sure that the configuration contains the correct device numbers for your configuration. Device 6023 corresponds to the FCP-adapter which I put in the zipl configuration and which will be used from the newly generated bootloader:

USER NEWVM P4SSWD 512M 1G G
   COMMAND define storage 512M standby 512M
   COMMAND SET RUN ON  
   CPU 00 BASE
   CPU 01
   CPU 02
   CPU 03
   IPL CMS PARM AUTOCR
   CONSOLE 0009 3215
   NICDEF 7000 TYPE QDIO LAN SYSTEM VSW1
   SPOOL 000C 2540 READER *
   SPOOL 000D 2540 PUNCH A
   SPOOL 000E 1403 A
   LINK MAINT 0190 0190 RR
   LINK MAINT 019D 019D RR
   LINK MAINT 019E 019E RR
  AMDISK 191 FB-512 AUTOG 1000 USER M BLKSIZE 4096
  DEDICATE 6023 6023

I created the file in /tmp/newvm.direct so it needs to be copied to the cloner’s 191 minidisk. It’s important to not directly edit the file on the minidisk but copy it after changing. See the post about writing to minidisks from Linux to find out more about this.

cloner:~ # cp /tmp/newvm.direct /mnt/mdisk/NEWVM.DIRECT

The next step is to use the newly created file to actually create the user using dirmaint. The rest of the user creation get’s handled by dirmaint (that’s what it’s made for):

cloner :~ # vmcp smsg vmutil dirm add NEWVM

At this time, the new z/VM user gets created. This is an asynchronous proces so it’s hard to tell when the creation is completed.

Create a PROFILE.EXEC for the new z/VM user

Once the user has been created, we need to create a 191-minidisk and a PROFILE.EXEC for the new user containing the correct boot-device in order to easily IPL that new user.

First link the 191-minidisk of the new user NEWVM to 291 of the cloner:

cloner:~ # vmcp link NEWVM 191 291 m

Put minidisk 291 online in the cloner and mount it:

cloner:~ # chccwdev -e 291
Setting device 0.0.0291 online
Done
cloner:~ # lsdasd 291
Bus-ID Status Name Device Type BlkSz Size Blocks
==============================================================================
0.0.0291 active dasdb 97:0 FBA 512 0MB 1000
cloner:~ # mkdir /mnt/mdisk2
cloner:~ # cmsfs-fuse --ascii /dev/dasdb /mnt/mdisk2

Create a new PROFILE.EXEC on the 291 minidisk and set the correct attributes for the file:

cloner:~ # touch /mnt/mdisk2/PROFILE.EXEC
cloner:~ # setfattr -n user.record_format -v F /mnt/mdisk2/PROFILE.EXEC
cloner:~ # setfattr -n user.record_lrecl -v 80 /mnt/mdisk2/PROFILE.EXEC

Create a temporary file and write the following to the file. The LOADDEV PORTNAME and LUN corresponds to the portname which can be found with lszfcp -D for FCP-device 6023.

/***************************/
/*  LINUX PROFILE EXEC     */
/***************************/
ADDRESS COMMAND
'CP TERMINAL MODE VM'
'CP SPOOL CONSOLE * START'
'CP SET PF11 RETRIEVE FORWARD'
'CP SET PF12 RETRIEVE BACKWARD'
'CP SET PF23 RETRIEVE FORWARD'
'CP SET PF24 RETRIEVE BACKWARD'
'CP LINK TCPMAINT 592 592 RR'
'ACCESS 592 E'
'CP SET LOADDEV PORTNAME 50050768 021595b1 LUN 003d0000 00000000'
'============================'
'ENTER IPL 6023 TO BOOT LINUX'
'============================'

Now copy the temporary file containing the user’s PROFILE.EXEC, which we just created, to the mounted 291 minidisk:

cloner:~ # cp /tmp/newvm.profexec /mnt/mdisk2/PROFILE.EXEC

After these steps, the system is ready to boot but first we should cleanup our devices and mounts:

cloner:~ # fusermount -u /mnt/mdisk
cloner:~ # fusermount -u /mnt/mdisk2
cloner:~ # chccwdev -d 191
cloner:~ # chccwdev -d 291
cloner:~ # vmcp det 291

This should be all it takes to clone a system. You could repeat these steps multiple times, using a different FPC device, different LUN and different IP address. After the clone, logon to the newly created user with the password written in the NEWVM.DIRECT file and enter IPL 6023 to start the clone. With a little bit of trying, this should be possible from the cloner as well.

Leave a Reply

Your email address will not be published. Required fields are marked *