Gathering Machine Information (Virtualized with Xen or not)

In this post, I’ll teach you how to gather all the available Virtual Machine information from Dom0, and on later posts, I’ll teach you how to do something useful with it. In this case the available commands are used to get information from the Xen hypervisor, without the need to install specific daemons inside each virtual machine. Some of the commands might be used without Xen. All commands were tested under Debian Testing x64.

Content 1 Black box approach 1.1 Network Information 1.2 System Information (General) 1.2.1 Xm dmesg 1.2.2 XenTop 1.2.3 VMstat 1.3 System Information (Processor) 1.4 System Information (Disk) 1.4.1 System Information (Disk I/O) 2 Some References

//

In this approach, we’ll try to gather as much information as needed from the DomU Virtual Machines without the need to install a daemon inside them. In this case, all we have to do is to obtain external information by examining Dom0′s events. There are some existing applications to help us in this approach, as we will see through this chapter.

Network Information

To gather the available network information, we must know that all the virtual machine’s network interconnections are based on the creation of “virtual bridges” inside Dom0. In this way, we can just monitor some stats inside domain 0 and monitor the output of /proc/net/dev in order to know what’s happening inside each virtual connection.

Example of information obtained (cat /proc/net/dev):

   Inter-|   Receive                                                                            |  Transmit
    face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
       lo:11057561497 29704758    0    0    0     0          0         0 11057561497 29704758    0    0    0     0       0          0
    peth0:11571633325 32617843   39   76    0    20          0   1371334 188033813708 141820549    0    0    0     0       0          0
     eth0:8135468250 22099821    0    0    0     0          0   3878810 50425653969 49284734    0    0    0     0       0          0
   xenbr1:69518661167 38933555    0    0    0     0          0        16 946684071 9312595    0    0    0     0       0          0
   xenbr2:1451764730 3177266    0    0    0     0          0     53007 25266007984 16786057    0    0    0     0       0          0
   xenbr3:131143810 1161664    0    0    0     0          0     36999 17574585288 11619502    0    0    0     0       0          0
   vif57.0:   19980      50    0    0    0     0          0         0 25691555   17010    0    7    0     0       0          0
   vif63.0:156382479  133845    0    0    0     0          0         0 122291227  527508    0   17    0     0       0          0
   vif63.1:1114455320 1222978    0    0    0     0          0         0 4858388654 3253890    0    2    0     0       0          0

Receive:

• bytes

Bytes received

• packets

Number of received packets

• errs

• drop

Number of dropped packets

• fifo

• frame

• compressed

• multicast

Transmit:

• bytes

• packets

• errs

• drop

• fifo

• colls

• carrier

• compressed

System Information (General)

Xm dmesg

Equivalent to Linux dmesg.

Usage:

xm dmesg

XenTop

The xentop command gives an general view of what’s happening on Dom0, and on each other virtual machines.

Example (xentop -i 1 -b)

         NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k) MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR SSID
     Domain-0 -----r      75742    0.0    1370112   66.2   no limit       n/a     2    4        0        0    0        0        0        0 2149627072
   xenvm.pendotiba.gta.ufrj.br --b---         54    0.0     131072    6.3     131072       6.3     2    1       19    25089    2        0     2782     3426 2149627072
   xenvm.varzeadasmocas.gta.ufrj.br --b---       1000    0.0     131072    6.3     131072       6.3     2    2  1241054  4863993    2        0    15680    60743 2149627072

• NAME

Name of the virtual machine

• STATE

Current state of the virtual machine ( r- running; b- blocked; p- paused; s- shutdown; c- crashed; d- dying)

• CPU(sec)

• CPU(%)

Average CPU use.

• MEM(k)

Used memory in kb

• MEM(%)

Percentage of used memory

• MAXMEM(k)

Max memory avaiable

• MAXMEM(%)

Percentage of used memory

• VCPUS

Number of virtual CPUs allocated to the machine

• NETS

Number of network interfaces

• NETTX(k)

• NETRX(k)

• VBDS

• VBD_OO

• VBD_RD

• VBD_WR

• SSID

VMstat

VMstat is a program that allows gathering of some system information.

command:

   jacarepagua:/etc/xen-tools/role.d# xm info

returns:

   host                   : jacarepagua
   release                : 2.6.26-2-xen-amd64
   version                : #1 SMP Fri Mar 27 07:12:15 UTC 2009
   machine                : x86_64
   nr_cpus                : 2
   nr_nodes               : 1
   cores_per_socket       : 2
   threads_per_core       : 1
   cpu_mhz                : 2397
   hw_caps                : bfebfbff:20100800:00000000:00000140:0000e3bd:00000000:00000001
   total_memory           : 2021
   free_memory            : 384
   node_to_cpu            : node0:0-1
   xen_major              : 3
   xen_minor              : 2
   xen_extra              : -1
   xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
   xen_scheduler          : credit
   xen_pagesize           : 4096
   platform_params        : virt_start=0xffff800000000000
   xen_changeset          : unavailable
   cc_compiler            : gcc version 4.3.1 (Debian 4.3.1-2)
   cc_compile_by          : waldi
   cc_compile_domain      : debian.org
   cc_compile_date        : Sat Jun 28 09:32:18 UTC 2008
   xend_config_format     : 4

Some explanations:

• hw_caps

A vector showing what hardware capabilities are supported by your processor. This is equivalent to, though more cryptic, the flags field in /proc/cpuinfo on a normal Linux machine.

• free_memory

Available memory (in MB ) not allocated to Xen, or any other Domains.

• xen_caps

The xen version, architecture. Architecture values can be one of: x86_32, x86_32p (i.e. PAE enabled), x86_64, ia64. xen_changeset The xen mercurial changeset id. Very useful for determining exactly what version of code your Xen system was built from.

Example ( vmstat -S M #used in megabytes… might be K or k):

Command:

   jacarepagua:/etc/xen-tools/role.d# vmstat -S M #megabytes

returns:

   procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
    r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
    0  0     51     20    105    119    0    0     6    13    7    7  0  0 99  0

• Procs

r: The number of processes waiting for run time.
b: The number of processes in uninterpretable sleep.

• Memory

swpd: the amount of virtual memory used.

free: the amount of idle memory.

buff: the amount of memory used as buffers.

cache: the amount of memory used as cache.

inact: the amount of inactive memory. (-a option)

active: the amount of active memory. (-a option)

• Swap

si: Amount of memory swapped in from disk (/s).

so: Amount of memory swapped to disk (/s).

• IO

bi: Blocks received from a block device (blocks/s).

bo: Blocks sent to a block device (blocks/s).

• System

in: The number of interrupts per second, including the clock.

cs: The number of context switches per second.

• CPU

These are percentages of total CPU time.

us: Time spent running non-kernel code. (user time, including nice time)

sy: Time spent running kernel code. (system time)

id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.

wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.

st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown.

System Information (Processor)

Usage:

   jacarepagua:/etc/xen-tools/role.d# iostat -c

returns:

   Linux 2.6.26-2-xen-amd64 (jacarepagua)     07/10/2009     _x86_64_    (2 CPU)

   avg-cpu:  %user   %nice %system %iowait  %steal   %idle
              0.43    0.00    0.22    0.09    0.04   99.22

ps. CPU Utilization Report The first report generated by the iostat command is the CPU Utilization Report. For multiprocessor systems, the CPU values are global averages among all processors. The report has the following format:

• %user

Show the percentage of CPU utilization that occurred while executing at the user level (application).

• %nice

Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.

• %system

Show the percentage of CPU utilization that occurred while executing at the system level (kernel).

• %iowait

Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.

• %steal

Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.

• %idle

Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.

System Information (Disk)

Example (vmstat -d):

   disk- ------------reads------------ ------------writes----------- -----IO------
          total merged sectors      ms  total merged sectors      ms    cur    sec
   sda   948812 257177 38732699 5055948 2231588 7198104 75491800 101776092      0   5059
   sdb      356   2589    8330    2296    119     42    1264     124      0      1

Reads:

• total

Total reads completed successfully

• merged

grouped reads (resulting in one I/O)

• sectors

Sectors read successfully

• ms

milliseconds spent reading

Writes:

• total

Total writes completed successfully

• merged

grouped writes (resulting in one I/O)

• sectors

Sectors written successfully

• ms

milliseconds spent writing
IO:

• cur

I/O in progress

• s

seconds spent for I/O

System Information (Disk I/O)

Only Total per disk:

   jacarepagua:/etc/xen-tools/role.d# iostat -d

   Linux 2.6.26-2-xen-amd64 (jacarepagua)     07/10/2009     _x86_64_    (2 CPU)

   Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
   sda               1.53        18.57        36.21   38801171   75636160
   sdb               0.00         0.00         0.00       8330       1264

Per partition:

   jacarepagua:/etc/xen-tools/role.d# iostat -p -d

   Linux 2.6.26-2-xen-amd64 (jacarepagua)     07/10/2009     _x86_64_    (2 CPU)

   Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
   sda               1.53        18.57        36.21   38801171   75638944
   sda1              1.18        10.69        19.80   22328339   41363880
   sda2              0.02         0.53         0.53    1106774    1105528
   sda3              0.33         7.35        15.88   15365434   33169536
   sdb               0.00         0.00         0.00       8330       1264
   sdb1              0.00         0.00         0.00       7474       1264

More detailed info:

   jacarepagua:/etc/xen-tools/role.d# iostat -p -x

   Linux 2.6.26-2-xen-amd64 (jacarepagua)     07/10/2009     _x86_64_    (2 CPU)

   avg-cpu:  %user   %nice %system %iowait  %steal   %idle
              0.43    0.00    0.22    0.09    0.04   99.22

   Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
   sda               0.12     3.45    0.46    1.07    18.57    36.20    35.88     0.05   33.57   1.59   0.24
   sda1              0.06     1.63    0.33    0.85    10.69    19.80    25.94     0.02   21.02   1.64   0.19
   sda2              0.05     0.06    0.02    0.00     0.53     0.53    50.09     0.00   36.04   3.97   0.01
   sda3              0.02     1.76    0.11    0.22     7.35    15.88    70.38     0.03   78.10   2.04   0.07
   sdb               0.00     0.00    0.00    0.00     0.00     0.00    20.20     0.00    5.09   3.70   0.00
   sdb1              0.00     0.00    0.00    0.00     0.00     0.00    21.00     0.00    4.27   3.02   0.00

• tps

Indicate the number of transfers per second that were issued to the device. A transfer is an I/O request to the device. Multiple logical requests can be combined into a single I/O request to the device. A transfer is of indeterminate size.

• Blk_read/s

Indicate the amount of data read from the device expressed in a number of blocks per second. Blocks are equivalent to sectors with 2.4 kernels and newer and therefore have a size of 512 bytes. With older kernels, a block is of indeterminate size.

• Blk_wrtn/s

Indicate the amount of data written to the device expressed in a number of blocks per second.

• Blk_read

The total number of blocks read.

• Blk_wrtn

The total number of blocks written.

• kB_read/s

Indicate the amount of data read from the device expressed in kilobytes per second.

• kB_wrtn/s

Indicate the amount of data written to the device expressed in kilobytes per second.

• kB_read

The total number of kilobytes read.

• kB_wrtn

The total number of kilobytes written.

• MB_read/s

Indicate the amount of data read from the device expressed in megabytes per second.

• MB_wrtn/s

Indicate the amount of data written to the device expressed in megabytes per second.

• MB_read

The total number of megabytes read.

• MB_wrtn

The total number of megabytes written.

• rrqm/s

The number of read requests merged per second that were queued to the device.

• wrqm/s

The number of write requests merged per second that were queued to the device.

• r/s

The number of read requests that were issued to the device per second.

• w/s

The number of write requests that were issued to the device per second.

• rsec/s

The number of sectors read from the device per second.

• wsec/s

The number of sectors written to the device per second.

• rkB/s

The number of kilobytes read from the device per second.

• wkB/s

The number of kilobytes written to the device per second.

• rMB/s

The number of megabytes read from the device per second.

• wMB/s

The number of megabytes written to the device per second.

• avgrq-sz

The average size (in sectors) of the requests that were issued to the device.

• avgqu-sz

The average queue length of the requests that were issued to the device.

• await

The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.

• svctm

The average service time (in milliseconds) for I/O requests that were issued to the device.

• %util

Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.

Some References

http://linux.die.net

http://www.debianhelp.co.uk/resources.htm

//

Introducing and applying the Virtualization Concept

virtualization

In this post, I’ll talk about the virtualization technique. I’m working deeply with it for some time, and I wish I could bring some of its usefulness to end-users. During the post, I’ll try to define some basic concepts of virtualization, and show how can you make benefit of it. On the next posts, I’ll explain deeply the virtualization concepts, and talk about the most important virtualization solutions available in the market. Hope you enjoy the post!

For those who don’t know, virtualization consists in a series of techniques that allow you to run instances of operating systems inside other operating systems. For instance, if you are using Windows, and don’t want to reboot the machine in order to use Linux or other system prior installed (like in a dual-boot case), you can start a new instance of Linux (virtual machine running Linux) inside windows, and use it side-by-side with windows and other systems at the same time. Using formal words, the operating system that runs underneath this so called virtual machines is called host. And the virtual machines running over it are called “Guests”

To provide this, the virtual machine system implements a layer, called hypervisor,  between hardware and the operating system’s software. This layer provides a hardware abstraction to the virtual machines, that is responsible for intercepting the virtual machine’s calls, send it to the hardware, get the results and then return it to the virtual machine. There’s an high overhead in this procedure, but as we will see in the next posts, there are many technologies, like Intel VT and AMD-V that can significantly reduce this implicit overhead.

The main virtualization products are: VMware (closed-source, paid), Xen (open-source), and Sun’s Virtual Box (open-source). In my toughs, Xen and VMware are developed for datacenters’ needs, while Virtual Box are ideal for “home use”. There’s also a “home” version of VMware but, to be honest, Virtual box is much better in almost all aspects. To tell the true, if you don’t want to deal with kernel and develop software to work specifically with virtualization, and just enjoy virtualization benefits, you can safely use Virtual Box. My experience says that Virtual Box has some of the best features, and it’s the only one that brings “real” graphic adapters (opengl and directx support) to virtual machines. The speed is fantastic, like you’re not using virtualization. I’ll talk about the new Virtual Box’s capabilities in the next post.

server consolidation

The virtualization technique brings a brand new horizon to computer services, mainly on data centers, where you can do the known “server consolidation”, that consists in changing the paradigm:

“one service per physical machine”

to:

“many isolated services running over the same physical machine”

The great advance is that the prerogative of virtualization is to ensure that all this guest machines must be completely isolated and virtualized-agnostic from each other. That means that if a single virtual machine is compromised, the other virtual machines and the host will keep running without any major problems.

Now you can ask: How virtualization may improve my daily computer use? And I bet that after reading this post, this question will be undoubtedly answered. I’ll cite some of the possibilities brought by virtualization to end users:

• Run multiple operating systems in the same machine, at the same time, sharing the hardware resources like disk, memory, USB devices and network interfaces.
• Have a multi-system testbed for software developing and system compatibilities issues. For example, develop a multi-platform system and test it on all supported systems in the same machine.
• Virtual machines’ disks are stored in files. That means that you can have a backup of your virtual machines, copy or clone them, and migrate them to other computers if desired.
• Save the current execution state of the running machines and then, recover this state if something goes wrong. This is specially good if you want to try a new software or driver configuration, and are worried if something will went wrong. To mitigate that, just take a screen shot of the actual state of the machine, try the modification, and if there’s a problem, just load the screen shot and reload the old machine.
• Have a safe “sand box” for critical operations. For instance I bet you like to do some Internet Banking and some e-commerce transactions. Its a great idea to install a rock-solid virtualized Linux distribution, install the needed browsers and safe configurations, and use this virtual machine to do this critical operations. As told before, you can prepare this virtual machine, take a screen shot of it, use it for the critical operations, and after, just restore the original screen shot state. That means that all possible vulnerabilities possibly installed during the Internet session(eg. keyloggers, spyware, trojans) that might capture you sensible information will be completely removed upon machine’s old state recovery.

As you can see, there are many uses for virtualization (I think there’s a lot more than was cited), and many of them are highly applicable on our lifes. In the next posts, I’ll talk about the existing virtualization solutions, and how to configure them to satisfy your needs.

Hugs, Hugo Carvalho.