SCOM alerts in Zabbix console

In previous post I was describing possible simple approach how to get single point of view of two independent monitoring solutions - Zabbix and SCOM - in one SCOM console. But what if you need opposite direction of integration?

Of course you can accomplish it and what is beautiful you can do it also only with SC Orchestrator in the middle. Let's see how.

In Orchestrator I have two Runbooks:

1) Send Alert to Zabbix

This Runbook monitors new alerts in SCOM. Then for each new alert I'm writing event message to the custom event log which is located on Orchestrator server. You can create it easily with:

$LogName = “Zabbix SCOM Sync”

$Source = “ZabbixSCOMSync”

New-EventLog -LogName $LogName -Source $Source

I'm using for new event the same error levels as they have in SCOM (as severity). The rest of magic of creating new Zabbix alerts is configured directly in Zabbix. The reason is that I didn't find any way in JSON-RPC how to do that via Zabbix's API. So for creation of alerts I'm utilizing Zabbix agent on Orchestrator server.

At first Install Zabbix agent on your Orchestrator server. Then go to Zabbix console and add it as host. After that I created new Item with following configuration:

This item will be responsible for catching all event written to the event log.

As final step you need to create three triggers in Zabbix. Each one for particular SCOM's severity level:

Now you are ready to run you new Runbook. Wen new alert will be created in SCOM, it will be captured with Runbook and written to the synchronization Event log from which it is immediately picked up by Zabbix agent and written to the Zabbix console as new Alert. 

Note: I'm adding to the message also ID of SCOM alert.

2) Resolve Zabbix's alert when it close in SCOM

Second Runbook is for acknowledging and closing Zabbix's alerts when the original SCOM's alert is closed.

First activity is triggered only when SCOM alert is closed. Then second activity is doing all the magic:

• Looking in the event log to find exact time it was written to it. It is the time when Zabbix created this alert in it's own console.
• Searching (with JSON-RPC) all open trigger's events in Zabbix and get the one with the same time as event in event log was created
• Acknowledge and close (with JSON-RPC) identified Zabbix event

And that's it. Try to close your SCOM alert and check in Zabbix console. It should be something like this:

Issues

The biggest issue was limiting on Zabbix's API. It is not possible to directly create new alert so I have to use Zabbix's agent for that.

The more serious issue is that it is not possible to identify events for acknowledging by searching in their messages. I spend a few hours with finding way but without luck. Finally I hade to use time comparison, but fortunately it seems to be working very well.

Another problem was that Zabbix is cutting text in messages to only 20 characters if they are from log monitoring. Fortunately you can change this behavior directly in Zabbix backend:

In my environment I changed it to 200 characters.

And that's it. I hope it was useful also for you and maybe it can open new integration scenarios for you as well.