Zenoss: The SNMP Chinese Wall, Separating Integers from Strings

In Zenoss, one of the issues that one finds when one devotes oneself to a pure SNMP implementation, avoiding SSH altogether, is that the Zenoss SNMP DataSource only really handles numerical outcomes. That means that it can handle the INTEGER value from nsExtendResult as well as convert the purely numerical STRING value from nsExtendOutLine, but cannot deal with the stdout from an entire nsExtendOutputFull OID.

To remedy this, just simply create a DataSource that uses one of the INTEGER OIDs, then a Threshold that triggers a specific Event Class, and finally a Transform within that Event Class that will replace the INTEGER OID with a STRING OID and snmpget the entire stdout from the SNMP Extend script. Assuming that you have read my SNMP Extend post, let us continue using the same OID for remote_command as well as the same exact template as before. All we need now is a transform for the /Perf/Snmp Event Class that will look something like this:

alert = re.search('threshold of (\w+)_failure_output', evt.message)
if alert:
    ds = evt.eventKey.split('|')[0]
    for t in device.getRRDTemplates():
        for s in t.getRRDDataSources():
            if ds == '%s_%s' % (s.id, s.id):
                 # evt.summary = snmpget of (nsExtendResult OID
                 # rewritten as nsExtendOutputFull OID)

As I was having trouble figuring out how to utilize the existing Zenoss python libraries (and not have to install one of the various proprietary pysnmp/snmppy modules) to create a pure python snmpget, I originally created a python-based subprocess call to the system's netsnmp tools:

from subprocess import *
outoid = s.oid.replace('8072.1.3.2.3.1.4','8072.1.3.2.3.1.2')
cmd = 'snmpget -Ov -%s -c%s %s %s' % (
    device.zSnmpVer, device.zSnmpCommunity,
    device.manageIp, outoid)
proc= Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE)
evt.summary = proc.stdout.readlines()[0].replace('STRING: ','')
break

Eventually, after some research and speaking with Zenoss support, this was the shortest pure python implementation I was able to figure out:

from twisted.internet import reactor
from pynetsnmp import twistedsnmp

def snmpget(proxy, oids):
    data = proxy.get(oids)
    data.addCallback(snmpvalue)

def snmpvalue(result):
    global snmpdata
    snmpdata = result
    reactor.stop()

where the evt.summary would be parsed like this:

proxy = twistedsnmp.AgentProxy(
    community=device.zSnmpCommunity,
    snmpVersion=device.zSnmpVer,
    ip=device.manageIp)
proxy.open()
oids = [s.oid.replace('8072.1.3.2.3.1.4','8072.1.3.2.3.1.2')]
reactor.callWhenRunning(snmpget, proxy, oids)
reactor.run()
proxy.close()
evt.summary = snmpdata[snmpdata.keys()[0]]
break

However, this produces the following zenperfsnmp output:

yyyy-mm-dd HH:MM:SS,123 ERROR zen.zenperfsnmp: [Failure instance: Traceback (failure with no frames): : Connection was closed cleanly.
]
Traceback (most recent call last):
  File "/opt/zenoss/Products/ZenHub/PBDaemon.py", line 382, in pushEvents
    driver.next()
  File "/opt/zenoss/Products/ZenUtils/Driver.py", line 64, in result
    raise ex
PBConnectionLost: [Failure instance: Traceback (failure with no frames): : Connection was closed cleanly.
]

which results in killing zenhub. When posted to the Zenoss Support portal, the Zenoss "engineers agree that the deferred is a bad thing to use inside the event transform." So, even though I find the subprocess method distasteful, it is the optimal (or only?) transform that can be performed. If something knows otherwise, please feel free to comment.

So, either dump the transform into the GUI panel or inject it via the add_transforms feature in my zenossYAMLTool and you should be set. For the complete YAML, download my Result2Output.yaml and modify it accordingly --- it contains additional python code for the transform to handle SNMPv3 using zConfigurationProperties as well as some simple escalation parsing logic.