Recently we have been experimenting with the Xen hypervisor. In my testing, I have found that Linux performance is better on Xen than VMWare and we are considering it for Linux rollouts.
Normally when we roll out a new server for a customer, we have a simple PHP script installed as a cron job that runs vmstat and logs the CPU utilization of the server into our database every 5 minutes. It's very useful for benchmarking, monitoring and troubleshooting mysterious performance problems. I needed a similar script for Xen.
A search in Google revealed a Perl script by Tom Brown to record the Xen domain CPU utilisation.
However the following limitations led me to modify it:
- Does not work properly with multi-core CPUs. CPU utilisation can go over 100%.
- Unfortunately sleep() does not sleep for precisely the number of seconds you define causing the CPU utilization to go over 100% again. There is some perturbation, either because Dom-0 is still virtualised or some other reason.
- No easy way of logging to a database.
So i rewrote parts of the script and renamed in xenstat.pl (after vmstat). You can download xenstat.pl here. To use run "perl xenstat.pl" in domain 0. The following output will be generated, with a new statistic generated every 5 seconds:
35_hobbes 0.35% cpu usage 15.29 sec over 0.02 days
54_garuda_0 0.63% cpu usage 52.62 sec over 0.05 days
59_gyrfalcon 0.69% cpu usage 57.39 sec over 0.05 days
Dom-0 1.57% cpu usage 15358.39 sec over 5.68 days
35_hobb 54_garu 59_gyrf Dom-0 Idle
2009-10-01 12:13:58 0.0 0.0 1.0 3.3 95.6
2009-10-01 12:14:04 0.0 0.0 0.2 7.3 92.4
2009-10-01 12:14:09 0.0 0.0 0.2 3.2 96.5
2009-10-01 12:14:14 0.0 0.0 0.2 3.1 96.6
In the above output, the first few lines summarise the CPUs and running domains. Then we have the statistics generated every 5 seconds.
You can also define the interval to poll (in seconds), and the number of samples just like vmstat:
[root@server ~]# perl xenstat.pl 3 2
cpus=2
35_hobbes 0.34% cpu usage 15.32 sec over 0.03 days
54_garuda_0 0.63% cpu usage 53.24 sec over 0.05 days
59_gyrfalcon 0.68% cpu usage 57.74 sec over 0.05 days
Dom-0 1.57% cpu usage 15362.69 sec over 5.68 days
35_hobb 54_garu 59_gyrf Dom-0 Idle
2009-10-01 12:14:59 0.0 0.0 1.7 5.7 92.5
2009-10-01 12:15:02 0.0 0.0 0.3 10.4 89.3
[root@server ~]#
To log the CPU utilisation using the Perl script, I didn't want to install a database client in Dom-0. So I added another parameter, a URL to a web server to call with the CPU info as GET parameters. I assume wget is installed in your Dom-0.
[root@server ~]# perl xenstat.pl 10 1 [192.168.0.1] cpus=2 54_garuda_0 0.49% cpu usage 3065.81 sec over 3.62 days 59_gyrfalcon 3.62% cpu usage 5039.03 sec over 0.80 days Dom-0 1.57% cpu usage 9782.15 sec over 3.62 days --10:46:42-- [192.168.0.1] Connecting to 192.168.0.1:80... connected. HTTP request sent, awaiting response... 200 OK Length: 498 [text/html] Saving to: `STDOUT' 100%[============================================>] 498 --.-K/s in 0s 10:46:42 (67.8 MB/s) - `-' saved [498/498] 2009-09-29 10:46:42 0 2 2 96
This will accumulate statistics for 10 seconds then send it to the above url in this format:
[192.168.0.1]
This allows you to log the data using a REST-ful web service.
I hope this is useful for anyone using Xen. This has been a good experience down memory lane too as I haven't coded in Perl for nearly 10 years!
1 Oct 2009 changes:
# Now works correctly when guest vm is started or shutdown (xenstat.pl detects changes and resets itself). # Fixed reboot handling - cpu usage will go negative # Change cpu formating from %3d to %7.1f. Needed when you have large number of guest vm's. # Added domain names every 20 lines. # Changed name of Domain-0 to Dom-0 to fit better into header line. # Switched to use HiRes sleep timer. Thanks Tim Thomas.
Other tools: see xentop, which can run in batch mode, but cannot post to web server.
The original script written by Tom Brown.



If you’re like me, you have bits and pieces of your online life scattered absolutely everywhere — photos, comments, tweets, blogs, favorite links — stashed away in all kinds of profiles. And, of course, many of those profiles offer RSS feeds, ripe for the mashing. Wouldn’t it be nice to aggregate all those on one page? I think so!
After Atlanta we go onto Miami, which I am only familiar with through 
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_a.png?x-id=eb6b55e2-85da-88bd-a263-dc329004c320)

















"...Vorträge von Entwicklern sind einfach. Die Fachthemen werden priorisiert und hierarchisch geordnet. Die Hierarchie wird "Agenda" genannt, und jedes Thema ergibt eine Folie, die alle Stichworte als Bulletpoints auflistet - welcome to the PowerPoint Hell. Genau 5 Minuten nach Beginn des Vortrags werden die Mails gechecked, nach 7 Facebook, und die Schnarchgeräusche ab Minute 10 lassen sich auch durch lautes Sprechen nicht mehr übertönen. Zwei Wochen kann sich der Zuhörer noch genau an den Titel der Präsentation erinnern, der Inhalt ist vollständig vergessen. 
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_a.png?x-id=20aff681-c4c1-8462-a15b-12d73157b0de)