Gathering Machine Information (Virtualized with Xen or not)

14 07 2009

In this post, I’ll teach you how to gather all the available Virtual Machine information from Dom0, and on later posts, I’ll teach you how to do something useful with it. In this case the available commands are used to get information from the Xen hypervisor, without the need to install specific daemons inside each virtual machine. Some of the commands might be used without Xen. All commands were tested under Debian Testing x64.

Content

//

In this approach, we’ll try to gather as much information as needed from the DomU Virtual Machines without the need to install a daemon inside them. In this case, all we have to do is to obtain external information by examining Dom0’s events. There are some existing applications to help us in this approach, as we will see through this chapter.

Network Information

To gather the available network information, we must know that all the virtual machine’s network interconnections are based on the creation of “virtual bridges” inside Dom0. In this way, we can just monitor some stats inside domain 0 and monitor the output of /proc/net/dev in order to know what’s happening inside each virtual connection.

Example of information obtained (cat /proc/net/dev):

   Inter-|   Receive                                                                            |  Transmit
    face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
       lo:11057561497 29704758    0    0    0     0          0         0 11057561497 29704758    0    0    0     0       0          0
    peth0:11571633325 32617843   39   76    0    20          0   1371334 188033813708 141820549    0    0    0     0       0          0
     eth0:8135468250 22099821    0    0    0     0          0   3878810 50425653969 49284734    0    0    0     0       0          0
   xenbr1:69518661167 38933555    0    0    0     0          0        16 946684071 9312595    0    0    0     0       0          0
   xenbr2:1451764730 3177266    0    0    0     0          0     53007 25266007984 16786057    0    0    0     0       0          0
   xenbr3:131143810 1161664    0    0    0     0          0     36999 17574585288 11619502    0    0    0     0       0          0
   vif57.0:   19980      50    0    0    0     0          0         0 25691555   17010    0    7    0     0       0          0
   vif63.0:156382479  133845    0    0    0     0          0         0 122291227  527508    0   17    0     0       0          0
   vif63.1:1114455320 1222978    0    0    0     0          0         0 4858388654 3253890    0    2    0     0       0          0

Receive:

  • bytes

Bytes received

  • packets

Number of received packets

  • errs
  • drop

Number of dropped packets

  • fifo
  • frame
  • compressed
  • multicast

Transmit:

  • bytes
  • packets
  • errs
  • drop
  • fifo
  • colls
  • carrier
  • compressed

System Information (General)

Xm dmesg

Equivalent to Linux dmesg.

Usage:

xm dmesg

XenTop

The xentop command gives an general view of what’s happening on Dom0, and on each other virtual machines.

Example (xentop -i 1 -b)

         NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k) MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR SSID
     Domain-0 -----r      75742    0.0    1370112   66.2   no limit       n/a     2    4        0        0    0        0        0        0 2149627072
   xenvm.pendotiba.gta.ufrj.br --b---         54    0.0     131072    6.3     131072       6.3     2    1       19    25089    2        0     2782     3426 2149627072
   xenvm.varzeadasmocas.gta.ufrj.br --b---       1000    0.0     131072    6.3     131072       6.3     2    2  1241054  4863993    2        0    15680    60743 2149627072
  • NAME

Name of the virtual machine

  • STATE

Current state of the virtual machine ( r- running; b- blocked; p- paused; s- shutdown; c- crashed; d- dying)

  • CPU(sec)
  • CPU(%)

Average CPU use.

  • MEM(k)

Used memory in kb

  • MEM(%)

Percentage of used memory

  • MAXMEM(k)

Max memory avaiable

  • MAXMEM(%)

Percentage of used memory

  • VCPUS

Number of virtual CPUs allocated to the machine

  • NETS

Number of network interfaces

  • NETTX(k)
  • NETRX(k)
  • VBDS
  • VBD_OO
  • VBD_RD
  • VBD_WR
  • SSID

VMstat

VMstat is a program that allows gathering of some system information.

command:

   jacarepagua:/etc/xen-tools/role.d# xm info

returns:

   host                   : jacarepagua
   release                : 2.6.26-2-xen-amd64
   version                : #1 SMP Fri Mar 27 07:12:15 UTC 2009
   machine                : x86_64
   nr_cpus                : 2
   nr_nodes               : 1
   cores_per_socket       : 2
   threads_per_core       : 1
   cpu_mhz                : 2397
   hw_caps                : bfebfbff:20100800:00000000:00000140:0000e3bd:00000000:00000001
   total_memory           : 2021
   free_memory            : 384
   node_to_cpu            : node0:0-1
   xen_major              : 3
   xen_minor              : 2
   xen_extra              : -1
   xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
   xen_scheduler          : credit
   xen_pagesize           : 4096
   platform_params        : virt_start=0xffff800000000000
   xen_changeset          : unavailable
   cc_compiler            : gcc version 4.3.1 (Debian 4.3.1-2)
   cc_compile_by          : waldi
   cc_compile_domain      : debian.org
   cc_compile_date        : Sat Jun 28 09:32:18 UTC 2008
   xend_config_format     : 4

Some explanations:

  • hw_caps

A vector showing what hardware capabilities are supported by your processor. This is equivalent to, though more cryptic, the flags field in /proc/cpuinfo on a normal Linux machine.

  • free_memory

Available memory (in MB ) not allocated to Xen, or any other Domains.

  • xen_caps

The xen version, architecture. Architecture values can be one of: x86_32, x86_32p (i.e. PAE enabled), x86_64, ia64. xen_changeset The xen mercurial changeset id. Very useful for determining exactly what version of code your Xen system was built from.

Example ( vmstat -S M #used in megabytes… might be K or k):

Command:

   jacarepagua:/etc/xen-tools/role.d# vmstat -S M #megabytes

returns:

   procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
    r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
    0  0     51     20    105    119    0    0     6    13    7    7  0  0 99  0
  • Procs

r: The number of processes waiting for run time.
b: The number of processes in uninterpretable sleep.

  • Memory

swpd: the amount of virtual memory used.

free: the amount of idle memory.

buff: the amount of memory used as buffers.

cache: the amount of memory used as cache.

inact: the amount of inactive memory. (-a option)

active: the amount of active memory. (-a option)

  • Swap

si: Amount of memory swapped in from disk (/s).

so: Amount of memory swapped to disk (/s).

  • IO

bi: Blocks received from a block device (blocks/s).

bo: Blocks sent to a block device (blocks/s).

  • System

in: The number of interrupts per second, including the clock.

cs: The number of context switches per second.

  • CPU

These are percentages of total CPU time.

us: Time spent running non-kernel code. (user time, including nice time)

sy: Time spent running kernel code. (system time)

id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.

wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.

st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown.

System Information (Processor)

Usage:

   jacarepagua:/etc/xen-tools/role.d# iostat -c

returns:

   Linux 2.6.26-2-xen-amd64 (jacarepagua)     07/10/2009     _x86_64_    (2 CPU)

   avg-cpu:  %user   %nice %system %iowait  %steal   %idle
              0.43    0.00    0.22    0.09    0.04   99.22

ps. CPU Utilization Report The first report generated by the iostat command is the CPU Utilization Report. For multiprocessor systems, the CPU values are global averages among all processors. The report has the following format:

  • %user

Show the percentage of CPU utilization that occurred while executing at the user level (application).

  • %nice

Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.

  • %system

Show the percentage of CPU utilization that occurred while executing at the system level (kernel).

  • %iowait

Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.

  • %steal

Show the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.

  • %idle

Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.

System Information (Disk)

Example (vmstat -d):

   disk- ------------reads------------ ------------writes----------- -----IO------
          total merged sectors      ms  total merged sectors      ms    cur    sec
   sda   948812 257177 38732699 5055948 2231588 7198104 75491800 101776092      0   5059
   sdb      356   2589    8330    2296    119     42    1264     124      0      1

Reads:

  • total

Total reads completed successfully

  • merged

grouped reads (resulting in one I/O)

  • sectors

Sectors read successfully

  • ms

milliseconds spent reading

Writes:

  • total

Total writes completed successfully

  • merged

grouped writes (resulting in one I/O)

  • sectors

Sectors written successfully

  • ms

milliseconds spent writing
IO:

  • cur

I/O in progress

  • s

seconds spent for I/O

System Information (Disk I/O)

Only Total per disk:

   jacarepagua:/etc/xen-tools/role.d# iostat -d
   Linux 2.6.26-2-xen-amd64 (jacarepagua)     07/10/2009     _x86_64_    (2 CPU)

   Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
   sda               1.53        18.57        36.21   38801171   75636160
   sdb               0.00         0.00         0.00       8330       1264

Per partition:

   jacarepagua:/etc/xen-tools/role.d# iostat -p -d
   Linux 2.6.26-2-xen-amd64 (jacarepagua)     07/10/2009     _x86_64_    (2 CPU)

   Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
   sda               1.53        18.57        36.21   38801171   75638944
   sda1              1.18        10.69        19.80   22328339   41363880
   sda2              0.02         0.53         0.53    1106774    1105528
   sda3              0.33         7.35        15.88   15365434   33169536
   sdb               0.00         0.00         0.00       8330       1264
   sdb1              0.00         0.00         0.00       7474       1264

More detailed info:

   jacarepagua:/etc/xen-tools/role.d# iostat -p -x
   Linux 2.6.26-2-xen-amd64 (jacarepagua)     07/10/2009     _x86_64_    (2 CPU)

   avg-cpu:  %user   %nice %system %iowait  %steal   %idle
              0.43    0.00    0.22    0.09    0.04   99.22

   Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
   sda               0.12     3.45    0.46    1.07    18.57    36.20    35.88     0.05   33.57   1.59   0.24
   sda1              0.06     1.63    0.33    0.85    10.69    19.80    25.94     0.02   21.02   1.64   0.19
   sda2              0.05     0.06    0.02    0.00     0.53     0.53    50.09     0.00   36.04   3.97   0.01
   sda3              0.02     1.76    0.11    0.22     7.35    15.88    70.38     0.03   78.10   2.04   0.07
   sdb               0.00     0.00    0.00    0.00     0.00     0.00    20.20     0.00    5.09   3.70   0.00
   sdb1              0.00     0.00    0.00    0.00     0.00     0.00    21.00     0.00    4.27   3.02   0.00
  • tps

Indicate the number of transfers per second that were issued to the device. A transfer is an I/O request to the device. Multiple logical requests can be combined into a single I/O request to the device. A transfer is of indeterminate size.

  • Blk_read/s

Indicate the amount of data read from the device expressed in a number of blocks per second. Blocks are equivalent to sectors with 2.4 kernels and newer and therefore have a size of 512 bytes. With older kernels, a block is of indeterminate size.

  • Blk_wrtn/s

Indicate the amount of data written to the device expressed in a number of blocks per second.

  • Blk_read

The total number of blocks read.

  • Blk_wrtn

The total number of blocks written.

  • kB_read/s

Indicate the amount of data read from the device expressed in kilobytes per second.

  • kB_wrtn/s

Indicate the amount of data written to the device expressed in kilobytes per second.

  • kB_read

The total number of kilobytes read.

  • kB_wrtn

The total number of kilobytes written.

  • MB_read/s

Indicate the amount of data read from the device expressed in megabytes per second.

  • MB_wrtn/s

Indicate the amount of data written to the device expressed in megabytes per second.

  • MB_read

The total number of megabytes read.

  • MB_wrtn

The total number of megabytes written.

  • rrqm/s

The number of read requests merged per second that were queued to the device.

  • wrqm/s

The number of write requests merged per second that were queued to the device.

  • r/s

The number of read requests that were issued to the device per second.

  • w/s

The number of write requests that were issued to the device per second.

  • rsec/s

The number of sectors read from the device per second.

  • wsec/s

The number of sectors written to the device per second.

  • rkB/s

The number of kilobytes read from the device per second.

  • wkB/s

The number of kilobytes written to the device per second.

  • rMB/s

The number of megabytes read from the device per second.

  • wMB/s

The number of megabytes written to the device per second.

  • avgrq-sz

The average size (in sectors) of the requests that were issued to the device.

  • avgqu-sz

The average queue length of the requests that were issued to the device.

  • await

The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.

  • svctm

The average service time (in milliseconds) for I/O requests that were issued to the device.

  • %util

Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.

Some References

http://linux.die.net

http://www.debianhelp.co.uk/resources.htm

//

Advertisements

Actions

Information

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




%d bloggers like this: