So there are all these scripts that you run on your remote hosts for monitoring. Since Zenoss has a built-in nagios parser, you basically run all of these scripts via SSH. What sucks is that establishing X number of SSH sessions for Y number of devices in Zenoss builds up a significant number of TCP connections and other wonderfully painful bottlenecks on your Zenoss system. Now, because all of these scripts already exist on the remote host, you can just as easily run it as root with a simple SNMP Extend line in your snmpd.conf. The line you would need to add would look something like this:
extend remote_command /customdir/customscript custom args -s 123
Do not forget to restart snmpd for the changes to take place. The next step would be to create a Zenoss DataSource for this --- bearing in mind that Zenoss works better with OID numbers than MIB names --- the straightforward approach would simply be to walk the SNMP tree and convert to OID:
snmpwalk -v2c -cpublic hostname 'NET-SNMP-EXTEND-MIB::nsExtendResult."remote_command"' -On
Two things to note in that command:
- If you want to see the MIB name, just simply remove the -On
- If you noticed that I used snmpwalk instead of snmpget, then you will understand that I do so because sometimes I like to walk the entire Extension tree using nsExtendObjects and snmpget would just barf on that
Rather than painstakingly reproduce all that using screenshots, I will illustrate the template creation steps using
zenossYAMLTool syntax, which is how I normally make changes to my Zenoss system (to avoid tons of images here as well as tediously clicking through the GUI). Assuming that I will not be graphing in this template, the YAML needed to create the template looks like this:
- action: add_template
description: Result Threshold retrieving Output Summary
targetPythonClass: Products.ZenModel.Device
templateName: Result2Output
templatePath: /Server/Linux/TestCase
GraphDefs: []
Now that we have the OID, adding a DataSource is fairly easy:
DataSources:
- dsName: remote_command
cycletime: 300
enabled: true
eventClass: /Cmd/Fail
oid: 1.3.6.1.4.1.8072.1.3.2.3.1.4.x.114.101.109.111.116.101.95.99.111.109.109.97.110.100
parser: Auto
severity: 3
sourcetype: BasicDataSource.SNMP
DataPoints:
- dpName: remote_command
isrow: true
rrdtype: GAUGE
You will find the red-highlighted number will increment as you add additional Extensions into the OID table. I had originally had a script that would generate the OID value for a specified extend_command, but since I cannot predict which position it would appear in the OID table, we will have to rely on snmpget/snmpwalk -On.
In order for it to trigger an alert, create a MinMax Threshold that will trigger an event (to keep it simply, we will be assuming 0=success and 1=failure here):
Thresholds:
- thresholdName: remote_failure_output
enabled: true
escalateCount: 0
eventClass: /Perf/Snmp
maxval: '0'
minval: ''
severity: 3
dsnames:
- remote_command_remote_command
For the complete YAML file, please read my next blog on how to use transforms to extract the nsExtendOutputFull after the nsExtendResult triggers and event.
And that is it for now. Of course, for the die-hard nagios plugin fanatic, you may be apt to point out that using SSH enables you to pass DataPoints via the stdout as such:
STATUS: Some useful output message here|data1=100;;; data2=10;20;30
This would be added to Zenoss with a single DataSource that contains multiple DataPoints. For my
zenossYAMLTool syntax, the YAML would look something like this:
DataSources:
- dsName: remote_command
cycletime: 300
enabled: true
eventClass: /Cmd/Fail
parser: Auto
severity: 3
sourcetype: BasicDataSource.COMMAND
usessh: true
DataPoints:
- {dpName: data1, isrow: true, rrdtype: GAUGE}
- {dpName: data2, isrow: true, rrdtype: GAUGE}
To recreate the same effect using SNMP Extend, you would need to add multiple DataSources, each with a single DataPoint, as the OIDs are mapped directly to each DataSource. The script should output the message and the data values on separate lines:
STATUS: Some useful output message here
100
10
And you would make use of the multiple nsExtendOutLine."remote_command".# OIDs to gather your DataSources:
DataSources:
- dsName: remote_command
cycletime: 300
enabled: true
eventClass: /Cmd/Fail
oid: 1.3.6.1.4.1.8072.1.3.2.4.1.2.12.114.101.109.111.116.101.95.99.111.109.109.97.110.100.1
parser: Auto
severity: 3
sourcetype: BasicDataSource.SNMP
DataPoints:
- dpName: remote_command
isrow: true
rrdtype: GAUGE
- dsName: remote_data1
cycletime: 300
enabled: true
eventClass: /Cmd/Fail
oid: 1.3.6.1.4.1.8072.1.3.2.4.1.2.12.114.101.109.111.116.101.95.99.111.109.109.97.110.100.2
parser: Auto
severity: 3
sourcetype: BasicDataSource.SNMP
DataPoints:
- dpName: remote_data1
isrow: true
rrdtype: GAUGE
- dsName: remote_data2
cycletime: 300
enabled: true
eventClass: /Cmd/Fail
oid: 1.3.6.1.4.1.8072.1.3.2.4.1.2.12.114.101.109.111.116.101.95.99.111.109.109.97.110.100.3
parser: Auto
severity: 3
sourcetype: BasicDataSource.SNMP
DataPoints:
- dpName: remote_data2
isrow: true
rrdtype: GAUGE
Yes, the YAML appears longer but it is not more complex, merely an exercise in copy-and-paste repetition. The multiple DataSources can easily be written as a loop in a simply generator script. So you have to decide which way you want to handle it --- though, I suppose, these examples make it seem like just a choice between configuration vs performance.
