Towards Event and Log Management Coherence

Blogger: Dan Blum

As I wrote in an earlier post about our Catalyst SIG, Burton Group is working with vendors and end user organizations in the industry to promote common event and log standards. Last week, progress continued as a Common Event Expression (CEE) conference call convened by Mitre brought what may be important clarification on the scope of the effort. Also, participants from Open Group’s XDAS group (including our own Bob Blakley) were added to the CEE editorial board, and an important IETF effort to enhance Syslog came to light.

After the conference call, Mitre summarized the scope discussion as follows:

“CEE will be the most valuable to the community if we take a top-down approach. This means that we start with a couple high-level use case drivers, such as regulatory compliance requirements as well as other log guidance, and determine what log types and data are necessary to meet those needs.

CEE should aim to be a lightweight standard. However, it needs to be flexible/extensible enough to support larger, more complex uses.

At minimum, CEE should require a timestamp and some sort of event classification.

The standard log data should be self describing, possibly in the form of name-value pairs. The next version of Syslog (currently in draft version in IETF) can probably support this within structured data blocks.”

It’s important to caution that the migration towards log and event standards will be a gradual, evolutionary process that won’t replace log and event chaos with a wholly uniform approach. One vendor noted on the call that there are many different eventing/logging use cases and sources, therefore “It would be disastrous to decree a single representational format like XML” and “I don’t see [my company] dropping all investment on rich logging infrastructure and adopting Syslog.”

Even if we don’t get to uniformity, Burton Group strongly agrees that something along the lines of the proposed standard would bring great value to the industry, and we are encouraged by the progress so far. As the ideas become more fully baked, we plan to encourage customers and clients not to wait, but to begin mandating these same log/event coherence concepts (such as event classification, timestamps, and data self-description mechanisms) for the systems and software they buy and develop.

CEE members are also working on defining use cases for event management, and a work-in-progress data dictionary. Now is a good time for security information event management vendors, log management vendors, and enterprise security practitioners to get involved; to participate in or monitor the CEE effort directly, see http://cee.mitre.org/.