[Xen-API] [xcp 1.6] rrdd issues

[Sébastien RICCIO <sr@xxxxxxxxxxxxxxx>]

Hi list!

First, I would like to send my congratulations to the team for the final 
release of XCP 1.6!
I've recently (manually) upgraded our production XCP 1.1 pool to XCP 1.6 
and this worked flawlessly.
After running the upgrade I can see that there seems to be a new feature 
for getting metrics informations such as iostat, power management, etc. 
and this is good news :)
The less good news is that it seems it doesn't work (at least on our 
machines, some Dell M610 blades)
Here is what I can see in xensource.log, on the master:

Nov 30 01:16:30 xen-blade11 xcp-rrdd-xenpm: [ 
info|xen-blade11|0||xcp-rrdd-xenpm] Done outputing to 
/dev/shm/metrics/xcp-rrdd-xenpm
Nov 30 01:16:31 xen-blade11 /opt/xensource/libexec/xcp-rrdd: 
[error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: 
xcp-rrdd-xenpm
Nov 30 01:16:31 xen-blade11 /opt/xensource/libexec/xcp-rrdd: 
[error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: 
xcp-rrdd-iostat
Nov 30 01:16:32 xen-blade11 xcp-rrdd-squeezed: [ 
info|xen-blade11|0||xcp-rrdd-squeezed] Done outputing to 
/dev/shm/metrics/xcp-rrdd-squeezed
Nov 30 01:16:36 xen-blade11 xcp-rrdd-xenpm: [ 
info|xen-blade11|0||xcp-rrdd-xenpm] Done outputing to 
/dev/shm/metrics/xcp-rrdd-xenpm
Nov 30 01:16:36 xen-blade11 /opt/xensource/libexec/xcp-rrdd: 
[error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: 
xcp-rrdd-xenpm
Nov 30 01:16:36 xen-blade11 /opt/xensource/libexec/xcp-rrdd: 
[error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: 
xcp-rrdd-iostat
Nov 30 01:16:37 xen-blade11 xcp-rrdd-squeezed: [ 
info|xen-blade11|0||xcp-rrdd-squeezed] Done outputing to 
/dev/shm/metrics/xcp-rrdd-squeezed
Nov 30 01:16:41 xen-blade11 xcp-rrdd-xenpm: [ 
info|xen-blade11|0||xcp-rrdd-xenpm] Done outputing to 
/dev/shm/metrics/xcp-rrdd-xenpm
Nov 30 01:16:41 xen-blade11 /opt/xensource/libexec/xcp-rrdd: 
[error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: 
xcp-rrdd-xenpm
Nov 30 01:16:41 xen-blade11 /opt/xensource/libexec/xcp-rrdd: 
[error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: 
xcp-rrdd-iostat
Nov 30 01:16:42 xen-blade11 xcp-rrdd-squeezed: [ 
info|xen-blade11|0||xcp-rrdd-squeezed] Done outputing to 
/dev/shm/metrics/xcp-rrdd-squeezed
Nov 30 01:16:46 xen-blade11 xcp-rrdd-xenpm: [ 
info|xen-blade11|0||xcp-rrdd-xenpm] Done outputing to 
/dev/shm/metrics/xcp-rrdd-xenpm
Nov 30 01:16:46 xen-blade11 /opt/xensource/libexec/xcp-rrdd: 
[error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: 
xcp-rrdd-xenpm
Nov 30 01:16:46 xen-blade11 /opt/xensource/libexec/xcp-rrdd: 
[error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: 
xcp-rrdd-iostat
Nov 30 01:16:47 xen-blade11 xcp-rrdd-squeezed: [ 
info|xen-blade11|0||xcp-rrdd-squeezed] Done outputing to 
/dev/shm/metrics/xcp-rrdd-squeezed
Nov 30 01:16:51 xen-blade11 xcp-rrdd-xenpm: [ 
info|xen-blade11|0||xcp-rrdd-xenpm] Done outputing to 
/dev/shm/metrics/xcp-rrdd-xenpm
Nov 30 01:16:51 xen-blade11 /opt/xensource/libexec/xcp-rrdd: 
[error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: 
xcp-rrdd-xenpm
Nov 30 01:16:51 xen-blade11 /opt/xensource/libexec/xcp-rrdd: 
[error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: 
xcp-rrdd-iostat
Nov 30 01:16:52 xen-blade11 xcp-rrdd-squeezed: [ 
info|xen-blade11|0||xcp-rrdd-squeezed] Done outputing to 
/dev/shm/metrics/xcp-rrdd-squeezed
Nov 30 01:16:56 xen-blade11 xcp-rrdd-xenpm: [ 
info|xen-blade11|0||xcp-rrdd-xenpm] Done outputing to 
/dev/shm/metrics/xcp-rrdd-xenpm
Nov 30 01:16:56 xen-blade11 /opt/xensource/libexec/xcp-rrdd: 
[error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: 
xcp-rrdd-xenpm
Nov 30 01:16:56 xen-blade11 /opt/xensource/libexec/xcp-rrdd: 
[error|xen-blade11|0 monitor|main|rrdd_server] Failed to process plugin: 
xcp-rrdd-iostat
Nov 30 01:16:57 xen-blade11 xcp-rrdd-squeezed: [ 
info|xen-blade11|0||xcp-rrdd-squeezed] Done outputing to 
/dev/shm/metrics/xcp-rrdd-squeezed
Nov 30 01:17:01 xen-blade11 xcp-rrdd-xenpm: [ 
info|xen-blade11|0||xcp-rrdd-xenpm] Done outputing to 
/dev/shm/metrics/xcp-rrdd-xenpm
xenstore.log on the slaves (all):
Nov 30 05:49:03 xen-blade15 /opt/xensource/libexec/xcp-rrdd: 
[error|xen-blade15|0 monitor|main|rrdd_server] Failed to process plugin: 
xcp-rrdd-xenpm
Nov 30 05:49:38 xen-blade15 last message repeated 7 times
Nov 30 05:50:43 xen-blade15 last message repeated 13 times
Nov 30 05:51:48 xen-blade15 last message repeated 13 times
Nov 30 05:52:53 xen-blade15 last message repeated 13 times
Nov 30 05:53:58 xen-blade15 last message repeated 13 times
Nov 30 05:54:59 xen-blade15 last message repeated 12 times
Nov 30 05:56:04 xen-blade15 last message repeated 13 times
Nov 30 05:57:09 xen-blade15 last message repeated 13 times

Seems there is a problem with xcp-rrdd-xenpm, but if i run it manually:

[root@xen-blade15 ~]# /opt/xensource/libexec/xcp-rrdd-plugins/xcp-rrdd-xenpm
[20121130T05:02:26.139Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Entering 
main loop ..
[20121130T05:02:26.139Z|debug|xen-blade15|0||mscgen] unset=>unset 
[label="Plugin.get_header"];
[20121130T05:02:26.139Z|debug|xen-blade15|0||mscgen] unset=>unset 
[label="Plugin.get_path"];
[20121130T05:02:26.140Z| info|xen-blade15|0||xcp-rrdd-xenpm] Obtained 
hdr=DATASOURCES
, path=/dev/shm/metrics/xcp-rrdd-xenpm

[20121130T05:02:26.140Z|debug|xen-blade15|0||mscgen] unset=>unset 
[label="Plugin.register"];
[20121130T05:02:26.141Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Sleeping 
for 3.8 seconds...
[20121130T05:02:29.972Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Forking 
command /usr/sbin/xenpm get-cpufreq-states
[20121130T05:02:29.995Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Forking 
command /usr/sbin/xenpm get-cpuidle-states
[20121130T05:02:30.002Z| info|xen-blade15|0||xcp-rrdd-xenpm] Done 
outputing to /dev/shm/metrics/xcp-rrdd-xenpm
[20121130T05:02:30.011Z|debug|xen-blade15|0||mscgen] unset=>unset 
[label="Plugin.register"];
[20121130T05:02:30.012Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Sleeping 
for 4.9 seconds...
[20121130T05:02:34.971Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Forking 
command /usr/sbin/xenpm get-cpufreq-states
[20121130T05:02:34.993Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Forking 
command /usr/sbin/xenpm get-cpuidle-states
[20121130T05:02:34.999Z| info|xen-blade15|0||xcp-rrdd-xenpm] Done 
outputing to /dev/shm/metrics/xcp-rrdd-xenpm
[20121130T05:02:35.010Z|debug|xen-blade15|0||mscgen] unset=>unset 
[label="Plugin.register"];
[20121130T05:02:35.011Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Sleeping 
for 5.0 seconds...
[20121130T05:02:39.981Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Forking 
command /usr/sbin/xenpm get-cpufreq-states
[20121130T05:02:40.004Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Forking 
command /usr/sbin/xenpm get-cpuidle-states
[20121130T05:02:40.015Z| info|xen-blade15|0||xcp-rrdd-xenpm] Done 
outputing to /dev/shm/metrics/xcp-rrdd-xenpm
[20121130T05:02:40.020Z|debug|xen-blade15|0||mscgen] unset=>unset 
[label="Plugin.register"];
[20121130T05:02:40.021Z|debug|xen-blade15|0||xcp-rrdd-xenpm] Sleeping 
for 5.0 seconds...
(seems to work)


Also iostat plugin dies. I tryed to run it from command line:

[root@xen-blade11 plugins]# 
/opt/xensource/libexec/xcp-rrdd-plugins/xcp-rrdd-iostat
[20121130T00:13:15.359Z|debug|xen-blade11|0||xcp-rrdd-iostat] Entering 
main loop ..
[20121130T00:13:15.359Z|debug|xen-blade11|0||mscgen] unset=>unset 
[label="Plugin.get_header"];
[20121130T00:13:15.360Z|debug|xen-blade11|0||mscgen] unset=>unset 
[label="Plugin.get_path"];
[20121130T00:13:15.361Z| info|xen-blade11|0||xcp-rrdd-iostat] Obtained 
hdr=DATASOURCES
, path=/dev/shm/metrics/xcp-rrdd-iostat
[20121130T00:13:15.361Z|debug|xen-blade11|0||mscgen] unset=>unset 
[label="Plugin.register"];
[20121130T00:13:15.362Z|debug|xen-blade11|0||xcp-rrdd-iostat] Sleeping 
for 4.6 seconds...
[20121130T00:13:20.026Z|debug|xen-blade11|0||xcp-rrdd-iostat] Forking 
command /usr/sbin/tap-ctl list
"/usr/bin/tap-ctl list" returns an unexpected output, which makes this 
plugin obsolete!!
Offending line: pid=12339 minor=5 state=0 
args=vhd:/var/run/sr-mount/5056918c-aee6-dd11-53b4-43f9e4c59e0d/867e2f8d-9438-4101-bbd0-c0a349ab672b.vhd
(note there is also a typo in the error message /usr/bin/tap-ctl should 
be /usr/sbin/tap/ctl
If i run the tap-list command manually:

[root@xen-blade11 plugins]# /usr/sbin/tap-ctl list
   12339    5    0        vhd 
/var/run/sr-mount/5056918c-aee6-dd11-53b4-43f9e4c59e0d/867e2f8d-9438-4101-bbd0-c0a349ab672b.vhd
   15130    3    0        vhd 
/var/run/sr-mount/5056918c-aee6-dd11-53b4-43f9e4c59e0d/a42f99a5-848f-4cef-af43-3dc493ed2a57.vhd
   29700    1    0        vhd 
/var/run/sr-mount/5056918c-aee6-dd11-53b4-43f9e4c59e0d/915ae08f-9921-4fc7-bb12-5a2406990638.vhd
   30679    6    0        vhd 
/var/run/sr-mount/97805239-b538-f13a-fe79-68f7f5d1ab22/23848d6b-3d4a-4b24-a6d2-782d12061458.vhd
    3723    0    0        vhd 
/var/run/sr-mount/97805239-b538-f13a-fe79-68f7f5d1ab22/f619f6fc-09a0-4bc2-a20a-16bc2feeb0da.vhd
    5289    4    0        vhd 
/var/run/sr-mount/5056918c-aee6-dd11-53b4-43f9e4c59e0d/2e0b1484-4c73-46e9-944e-01716c64a620.vhd
    8603    7    0        vhd 
/var/run/sr-mount/97805239-b538-f13a-fe79-68f7f5d1ab22/9bbef03c-5369-42fb-a5ab-1b65edb96202.vhd
    9696    2    0        vhd 
/var/run/sr-mount/5056918c-aee6-dd11-53b4-43f9e4c59e0d/c7fd2830-0e41-4bcf-adda-acc21674bf77.vhd
Seems some things are broken at the moment. I don't know if it's 
specific to our hardware or if it's a general problem.
Please note that I've also installed a clean xcp 1.6 on a stand-alone 
server (no upgrade, no pool, standard setup) and the same errors occurs.
Any idea what could be wrong?

Do you need more informations to sort this out ?

Thanks a lot for your help.

Cheers,
Sébastien

_______________________________________________
Xen-api mailing list
Xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api