How to run command on Zabbix agents?

Question

I want to run a command on Zabbix agents:

• Some simple unix commands, to obtain our reporting data.
• When there is some processing required on the agent side.

There seem to be a variety approaches being talked about. So how to execute such commands on a Zabbix agent?

Endways

Answer 1

Run commands from the server directly from a new item.

First, set: EnableRemoteCommands=1 in the agent conf file (for all of your agents). To enable this feature.

Create a new item. A field on the "new item" page says 'key'. Enter:

system.run[command]

As the 'key' string. Where command is the command you want to be downloaded and run on the agent. Here is an example:

system.run[sysctl dev.cpu.0.temperature | cut -d ' ' -f 2 | tr -d C]

Perhaps you need to run something substantially more complex that is too long to fit in there? Then you will need to make a custom script. Put your custom scripts on a local webserver, or somewhere on the web.

Then you might set the item's key to:

system.run[ command -v script && script || wget script_url -O /path/to/script && script]

To fetch and download the missing script to the agent the first time it's executed. However that is a rather crude hack. Not very elegant.

A better way is to go to "Administration" --> "Scripts" in the menu. From there, you can create a new script to use in an item which may be configured to run on any of your agents.

Make a special custom item to re-run your script periodically (like a cron job). The job of the special script item is to update the agent with a collection of your other needed custom scripts.

Of course you could just write all of your custom scripts directly into zabbix's MYSQL database. And it is very tempting to do that. But be aware that then they'd be lost and vulnerable if your zabbix database ever gets fried or corrupted / lost. Zabbix databases always have a habit of growing large, unwieldy and out-of-control. So don't do that. Storing them separately somewhere else and under version control (git or subversion).

Once that's all sorted, we can finally go ahead and create further custom items to run your custom scripts. Again using:

system.run[script]

as the item's key just as before. Where 'script' is the command (plus any arguments), to execute your custom script locally on the agent.

Endways

Ah. I forgot to mention: EnableRemoteCommands=1 needs to be set in the agent conf file. Otherwise system.run[command] won't work! – Endways

I strongly vote against EnableRemoteCommand. This is for me personally a no-go from security perspective. What we do is that we use external checks, as described here. – Township

2 points: 1) With the possible exception of Windows, EnableRemoteCommand will only allow commands to be run as the 'zabbix' user. The user which zabbix_agentd should be run as. And not root. Witch correct configuration in 'sudoers' file, the 'zabbix' user may be permitted to run only chosen commands with any kind of higher permission. 2) The question is "how to run commands on agent". The zabbix documentation page about external checks feature says they can only be run on the server (and not the agent). – Endways

1) Being able to gain access to other servers is already have the way to gain root access on other servers. 2) Indeed, sorry. I mixed up external checks with UserParameters. That's what we use and what reduces the number of possible commands that can be executed. – Township

Ah indeed UserParameters is generally more popular method than system.run[]. But 1) sounds a little concerning however. Can you please clarify about that, or give link specifically in regards to zabbix_agent? Many thanks. – Endways

If a hacker is able to break into a single system, this is bad. If this system has permission to log into all other systems, it is even worse. It's nothing zabbix-specific, but I have an example for you: CVE-2013-5743 was a vulnerability affecting the zabbix server. If arbitrary remote commands are enabled, the whole infrastructure is affected through the server. – Township

Having a login for all systems, is already very bad and can cause serious damage. Other than that it's only a matter of time until a hacker gains root privileges (maybe through other vulnerabilities). You want to minimize the number of systems in your infrastructure that can control all others. – Township

Answer 2

1. Define the user parameter at the client (where zabbix agent is located) at /etc/zabbix/zabbix_agentd.conf

   The key should be unique. I am using lsof as an example: UserParameter=open_file,lsof | wc -l

2. Restart the agent: service zabbix-agent restart

3. Test if the key is working using zabbix_get utility. To do that from the zabbix server, invoke the following: /usr/local/bin/zabbix_get -s <HOST/IP of the zabbix agent> -k open_file (It should return a number in this case)

4. Create an item with the key at the zabbix server at the template level (the return type should be correctly defined, otherwise zabbix will not accept it):

   Type: Zabbix Agent (Active)

   Key: open_file

   Type of Information: Numeric (unsigned)

   Data Type: decimal

5. You may create a graph using the item to monitor the value at regular interval.

Here is the official documentation.

Isabelleisac