MQ Statistics with triggers and MQWeb

25 January 2019

What?

IBM MQ can generate statistics for queues, channels, … To be able to do some capacity management, these statistics will be transferred to ELK (Elasticsearch, Logstash & Kibana). This solution will use two Python scripts to get statistics in JSON format from MQWeb and saves the statistics to a file. With filebeat the generated files are collected and transferred to ELK.

How?

Enable IBM MQ Statistics

IBM MQ puts statistic messages on the SYSTEM.ADMIN.STATISTICS.QUEUE. This is done when the queuemanager property STATQ has value ON or when the queue property STATQ has value ON. By default the statistics are generated every 1800 seconds. The interval can be changed by setting the STATINT property of the queuemanager.

It’s also possible to force the generation of the statistics by issuing the following command on a queuemanager:

RESET QMGR TYPE(STATISTICS)

Gather statistics from a queue

Statistic messages are messages with format MQADMIN. The message payload is a collection of PCF structures. MQWeb has an API to get messages from a queue and translate MQADMIN messages into a readable JSON structure. The mqstats.py script will use MQWeb to get these messages and output them to stdout or to a file. The script ends when there are no more messages to process.

Triggering

The mqstats.py script must run when messages are available on the queue. To avoid the creation of a new daemon process, the IBM MQ triggering system is used. IBM MQ can trigger the script whenever the queue depth of the queue changes from 0 to 1. Because IBM MQ passes an MQTMC2 structure to the triggered process, another script mqstatstrigger.py is written. This script will translate the MQTMC2 structure into commandline arguments for mqstats.py and execute it.

The mqstats.py script can be used directly from the commandline. Use the –help argument to show help information. When the MQ trigger doesn’t fire (because there are still messages on the queue), run the mqstats.py from the commandline to process the remaining messages.

Follow these steps to setup IBM MQ triggering:

Initiation queue

When a trigger event occurs, the queuemanager puts a trigger message on a initiation queue. This trigger message will be retrieved by the trigger monitor.

DEFINE QL(MQ.STATISTICS.TRIGGER.Q1) +
DESCR('Initiation Queue For Collecting Statistics')

Define a process

The process defines what the trigger monitor must execute when the trigger is fired.

DEFINE PROCESS(MQ.STATS) +
APPLICID('python mqstatstrigger.py') +
USERDATA('output=/var/elk') +
DESCR('ELK statistics script')

The USERDATA value is used by the mqstatstrigger.py script to pass arguments to mqstats.py

Alter the local queue

The queue that is responsible for the trigger must be associated with the initiation queue and the process:

ALTER QL(SYSTEM.ADMIN.STATISTICS.QUEUE) +
INITQ(MQ.STATISTICS.TRIGGER.Q1)
PROCESS(MQ.STATS) +
TRIGTYPE(FIRST)

Define a service

The runmqtrm program is the IBM MQ trigger monitor. It must run in background. When a service is used, it can be controlled by the queuemanager by setting the CONTROL property to QMGR. This way the service will automatically start and stops when a queuemanager is started or stopped.

DEFINE SERVICE(MQ.STATS.TRIGMON) +
CONTROL(QMGR) +
STARTCMD('+MQ_INSTALL_PATH+bin/runmqtrm') +
STARTARG('-m +QMNAME+ -q MQ.STATISTICS.TRIGGER.Q1') +
STOPCMD('+MQ_INSTALL_PATH+bin/amqsstop') +
STOPARG('-m +QMNAME+ -p +MQ_SERVER_PID+') +
STDOUT('/var/elk/mqstats.stdout') +
STDERR('/var/elk/mqstats.stderr') +
DESCR('Trigger Monitor for ELK Statistics')

MQWebApp

13 January 2016

The web application that was part of MQWeb is moved to a separate repository MQWebApp and removed from the current development branch of MQWeb. The web application will be a totally rewrite as a Single Page Application for WebSphere MQ using Vue.js and UIKit.

This MQWeb App will only be supported from MQWeb 0.1.0. and above.

Screenshot

Poco Redis

21 November 2015

Today the development of the Poco Redis module is merged into the Poco development branch.

Some examples:

###Setting a key/value with add

Array command;
command.add("SET").add("mykey").add("Hello");

// A set responds with a simple OK string
try
{
  std::string result = redis.execute<std::string>(command);
}
catch(RedisException &e)
{
  ...
}

###Setting a key/value with «

Array command;
command << "SET" << "mykey" << "Hello";

// A set responds with a simple OK string
try
{
  std::string result = redis.execute<std::string>(command);
}
catch(RedisException &e)
{
  ...
}

###Setting a key/value with Command class

Command set = Command::set("mykey", "Hello");

// A set responds with a simple OK string
try
{
  std::string result = redis.execute<std::string>(set);
}
catch(RedisException &e)
{
  ...
}

Poco Redis

22 October 2015

For those interested in a Poco library for Redis … I’ve created a branch on my fork of the Poco repository. I know there is another attempt here, but this project doesn’t really follow the Poco coding standard and I can’t read Chinese comments. Besides that, it gives me also the opportunity to learn another Pub/Sub solution. Currently only send/reply is supported. Pub/Sub will be added soon …

MQWeb 0.0.11 released

4 November 2014

Today MQWeb 0.0.11 is released. You can download this release from Github.

This is a list of the major changes:

  • Add inquire connection (conn)
  • Add inquire namelist (nl)
  • Add inquire process (process)
  • Add inquire service (service)
  • Add inquire authentication information (authinfo)
  • Add inquire authority record (authrec)
  • Support for JSONP
  • Allow to run without mq.web.templates / mq.web.static configuration
  • Add first steps for connection pooling