Check Mk Create Own Special Agent

Quite often I am using Check-MK system monitoring (in place of Nagios), mostly because it has quite nice defined custom plugin interface, but not quite good documented in more advanced aspects. In this blog post I will focus on how to wrote your own Special Agent plugin type, so let’s dig around of it!

1 Why Special Agent ?

If we want to write a custom check, first we have to provide to check-mk server monitored data, there is a few option to do that:

• Agent Based Checks (https://checkmk.com/cms_legacy_devel_agentbased.html) - simply, create script in /usr/lib/check_mk_agent/plugins/ on the agent side, which outputs data in section <<<yourCustomCheck>>>. When check-mk server asks agent for data by connecting to client on 6556 port, the script will be executed and output delivered to check-mk server for processing by your custom check, on the server side.
• Custom datasource - there is an option to create custom datasource program/script in any language, which will be invoked instead Agent. It can be done by rule in Host & Service Parameters -> Individual program call instead of agent access. This approach has some disadvantages, when we use it, the Agent on the monitored host would not be invoked, so Agent Based Checks will not be working (the build in Agent Based checks as well would not work), what is more, in this case we can call only one datasource program for monitored host.
• Piggy back (https://checkmk.com/cms_piggyback.html) - very nice mechanism, which is used for example in vSphere plugin. In short, one of monitored host, can deliver data for checks for another monitored hosts. This approach is very well documented, so i recommend to read the documentation. The main disadvantage is that we need separate host which doing checks an return data for another hosts - in some cases it works well e.g. mentioned earlier vSphere plugin but in some we need something more.
• Special Agent (https://checkmk.com/cms_datasource_programs.html#specialagents) - most powerful mechanism, but also most not documented one :). If I would have to explain what is Special Agent, I would say that this is Agent Based Checks with non-limited custom datasources approach. In this blog post I will show how to write and register that type of agent.

2 Datasource for Special Agent

To create custom Special Agent, there is a some steps to create one. First we need create a script which will be our datasource, it provides data for our custom checks. Datasource script of our agent have to be placed in folder: local/share/check_mk/agents/special. Filename for that agent also must be special, I mean it has to follow schema: agent_agentName in my case: agent_myspecial, below is simple code for my Special Agent:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

#!/usr/bin/python3

import argparse
import requests


if __name__ == '__main__':
    arg_parser = argparse.ArgumentParser()
    required_param_group = arg_parser.add_argument_group('required arguments')
    required_param_group.add_argument('-a', '--adress_ip', required=True, help="Host adress")
    required_param_group.add_argument('-p', '--port', required=True, help="Port number")
    args = arg_parser.parse_args()

    r = requests.get('http://{}:{}'.format(args.adress_ip, args.port))

    print('<<<myspecial_response_code>>>')
    print(r.status_code)

The code is straightforward, my script expects to get IP address and port via params, then it try to perform GET request on that IP and port. At the end, it print to stdout response code for that request.

3 Register Special Agent

Next step which has to be taken is registering agent_myspecial script as a datasource for our Special Agent. It has be done by creating in local/share/check_mk/web/plugins/wato a file for example myspecial_register.py with piece of code as below – filename in this step does not matter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

#!/usr/bin/python

#import required to register agent
from cmk.gui.plugins.wato import (
    IndividualOrStoredPassword,
    RulespecGroup,
    monitoring_macro_help,
    rulespec_group_registry,
    rulespec_registry,
    HostRulespec,
)
#import structure where special agent will be registered
from cmk.gui.plugins.wato.datasource_programs import RulespecGroupDatasourcePrograms

#Some WATO form definition, to ask user for port number
def _valuespec_special_agent_myspecial():
    return Dictionary(
        title=_("Digaround mySpecial Agent"),
        help=_("This agent is responsible for check if something response on given http port"),
        optional_keys=[],
        elements=[
            ("port", TextAscii(title=_("HTTP port"), allow_empty=False)),
        ],
    )


#In that piece of code we registering Special Agent
rulespec_registry.register(
    HostRulespec(
        group=RulespecGroupDatasourcePrograms,
        #IMPORTANT, name must follow special_agents:<name>, 
        #where filename of our special agent located in path local/share/check_mk/agents/special/ is  agent_<name>
        name="special_agents:myspecial",
        valuespec=_valuespec_special_agent_myspecial,
    ))

There is a some structures which has to be imported to register custom datasource, I commented most important parts of the code. After save this file, check_mk server has to be restarted to read our custom files, if everything went well, when we go to Host & Service Parameters -> Datasource Programs we can see that our Digaround mySpecial Agent is registered and visible on the list:

Now we can click, and create rule for agent to activate it on the test host:

4 Invoke Special Agent

Created rule does not cause that mySpecialAgent will be automatically invoked, first we have to register “invoker” for agent. It can be done, by create file agent_myspecial in local/share/check_mk/checks.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

#!/usr/bin/env python3

#Function get params (in this case is port, passed via WATO agent rule cunfiguration, hostname and ip addres of host, 
#for which agent will be invoked 
def agent_myspecal_arguments(params, hostname, ipaddress):
    args = []

    args += ["-a", ipaddress]
    args += ["-p", params['port']]


    return args

#register invoke function for our agent
#key value for this dictionary is name part from register datasource of our agent (name="special_agents:myspecial" remember?)
special_agent_info['myspecial'] = agent_myspecal_arguments

In this case code is quite simple, we define function which gets parameters form check_mk environment and prepare arguments which has to be passed to our agent while executed. Next, defined function is registered for our agent – myspecial in dictionary special_agent_info.

5 Create custom check:

Last step in our journey is create some custom check, to consume data from our custom special agent and alert if condition are meet. If someone creates custom check for check_mk daily I think the code is self-explanatory, in short check takes return code passed by our special agent, checks if it is 200, if not alerts.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

#!/usr/bin/python

def inventory_myspecial_response_code(info):
    inventory = []

    for line in info:
        inventory.append(('Myspecial Response', {}))

    return inventory



def check_myspecial_response_code(item, params, info):
    exit_code = 0
    http_code = int(info[0][0])
    msg = 'Returned code: {}'.format(http_code)

    if http_code != 200:
       exit_code = 1


    return exit_code, msg


check_info["myspecial_response_code"] = {
    'check_function': check_myspecial_response_code,
    'inventory_function': inventory_myspecial_response_code,
    'service_description': '%s',
    'has_perfdata': False,
    'group': 'myspecial_group',
}

After save this file in local/share/check_mk/checks/myspecial_response_code, reload check_mk server. Check_Mk Discovery check should now report our custom Myspecial Response check, if we add it we should see it works!

6 Conclusion

Special Agent are quite powerful feature in Check_MK, it can be used to write really complicated checks which has to gather data from strange sources. I hope explained well how to create one if some non-standard requirements shows. One thing worth mention, above tutorial works in Check_MK line 1.6, keep in mind also that 1.7 release is coming, where authors announced some big changes in custom check interfaces - so probably code above will require some tuning then. Anyway if you want simple bootstrap to write you own Special Agent take a look my github , where you can browse code above.