The problem

SNMPD is correctly delegating SNMP polling requests to another program but the response from that program is not valid. A manual run of the program with the same arguments is responding correctly.

The detail

I've installed the correct LSI raid drivers on a server and want to configure SNMP. As per the instructions, I've added the following to /etc/snmp/snmpd.conf to redirect SNMP polling requests with a given OID prefix to a program:

pass .1.3.6.1.4.1.3582 /usr/sbin/lsi_mrdsnmpmain

It doesn't work correctly for SNMP polling requests:

snmpget -v1 -c public localhost .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.1

I get the following response:

Error in packet
Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: SNMPv2-SMI::enterprises.3582.5.1.4.2.1.2.1.32.1

What I've tried

SNMPD passes two arguments, -g and <oid> and expects a three line response <oid>, <data-type> and <data-value>.

If I manually run the following:

/usr/sbin/lsi_mrdsnmpmain -g .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.0

I correctly get a correct three line response:

.1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.0
integer
30

This means that the pass command is working correctly and the /usr/sbin/lsi_mrdsnmpmain program is working correctly in this example

I tried replacing /usr/sbin/lsi_mrdsnmpmain with a bash script. The bash script delegates the call and logs the supplied arguments and output from the delegated call:

#!/bin/bash
echo "In: '$@" > /var/log/snmp-pass-test
RETURN=$(/usr/sbin/lsi_mrdsnmpmain $@)
echo "$RETURN"
echo "Out: '$RETURN'" >> /var/log/snmp-pass-test

And modified the pass command to redirect to the bash script. If I run the bash script manually /usr/sbin/snmp-pass-test -g .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.0 I get the correct three line response as I did when I ran /usr/sbin/lsi_mrdsnmpmain manually and I get the following logged:

In: '-g .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.0
Out: '.1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.0
integer
30'

When I rerun the snmpget test, I get the same Error in packet... error and the bash script's logging shows that the captured delegated call output is empty:

In: '-g .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.0
Out: ''

If I modify the bash script to only echo an empty line I also get the same Error in packet... message.

I've also tried ensuring that the environment variables that are present when I manually call /usr/sbin/lsi_mrdsnmpmain are the same for the bash script but I get the same empty output.

Finally, my questions

Why would the bash script behave differently in these two scenarios?
Is it likely that the problem that exists with the bash scripts is the same as originally noticed (manually running program has different output to SNMPD run program)?

Updates

eewanco's suggestions

What user is running the program in each scenario?

I added echo "$(whoami)" > /var/log/snmp-pass-test to the bash script and root was added to the logs

Maybe try executing it in cron

Adding the following to root's crontab and the correct three line response was logged:

* * * * * /usr/sbin/lsi_mrdsnmpmain -g .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.1 >> /var/log/snmp-test-cron 2>&1

Grisha Levit's suggestion

Try logging the stderr

There aren't any errors logged

Checking /var/log/messages

When I run it via SNMPD, I get MegaRAID SNMP AGENT: Error in getting Shared Memory(lsi_mrdsnmpmain) logged. When I run it directly, I don't. I've done a bit of googling and I may need lm_sensors installed; I'll try this.

I installed lm_sensors & compat-libstdc++-33.i686 (the latter because it said it was a pre-requisite from the instructions and I was missing it), uninstalled and reinstalled the LSI drivers and am experiencing the same issue.

SELinux

I accidently stumbled upon a page about extending snmpd with scripts and it says to check the script has the right SELinux context. I ran grep AVC /var/log/audit/audit.log | grep snmp before and after running a snmpget and the following entry is added as a direct result from running snmpget:

type=AVC msg=audit(1485967641.075:271): avc:  denied  { unix_read unix_write } for  pid=5552 comm="lsi_mrdsnmpmain" key=558265  scontext=system_u:system_r:snmpd_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=shm

I'm now assuming that SELinux is causing the call to fail; I'll dig further...see answer for solution.

strace (eewanco's suggestion)

Try using strace with and without snmp and see if you can catch a system call failure or some additional hints

For completeness, I wanted to see if strace would have hinted that SELinux was denying. I had to remove the policy packages using semodule -r <policy-package-name> to reintroduce the problem then ran the following:

strace snmpget -v1 -c public localhost .1.3.6.1.4.1.3582.5.1.4.2.1.2.1.32.1 >> strace.log 2>&1

The end of strace.log is as follows and unless I'm missing something, it doesn't seem to provide any hints:

...
sendmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(161),     sin_addr=inet_addr("127.0.0.1")}, msg_iov(1)=    [{"0;\2\1\0\4\20public\240$\2\4I\264-m\2"..., 61}], msg_controllen=32,     {cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=, ...}, msg_flags=0},     MSG_DONTWAIT|MSG_NOSIGNAL) = 61
select(4, [3], NULL, NULL, {0, 999997}) = 1 (in [3], left {0, 998475})
brk(0xab9000)                           = 0xab9000
recvmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(161),     sin_addr=inet_addr("127.0.0.1")}, msg_iov(1)=    [{"0;\2\1\0\4\20public\242$\2\4I\264-m\2"..., 65536}],     msg_controllen=0, msg_flags=0}, MSG_DONTWAIT) = 61
write(2, "Error in packet\nReason: (noSuchN"..., 81Error in packet
Reason: (noSuchName) There is no such variable name in this MIB.
) = 81
write(2, "Failed object: ", 15Failed object: )         = 15
write(2, "SNMPv2-SMI::enterprises.3582.5.1"..., 48SNMPv2-        SMI::enterprises.3582.5.1.4.2.1.2.1.32.1
) = 48
write(2, "\n", 1
)                       = 1
brk(0xaa9000)                           = 0xaa9000
close(3)                                = 0
exit_group(2)                           = ?
+++ exited with 2 +++

It was SELinux that was denying snmpd a delegated call to /usr/sbin/lsi_mrdsnmpmain (and probably beyond).

To identify it, I ran grep AVC /var/log/audit/audit.log and for each entry, I ran the following:

echo "<grepped-output>" | audit2allow -a -M <filename>

This creates a SELinux policy package that should allow the delegated call through. The package is then loaded using the following:

semodule -i <filename>.pp

I had to do this 5 times as there were different causes of denial (unix_read unix_write, associate, read write). I'll look to combine the modules into one.

Now when I run snmpget I get the correct delegated output:

SNMPv2-SMI::enterprises.3582.5.1.4.2.1.2.1.32.1 = INTEGER: 34