[ InAction Note ] Ch5. Message Storage - The KahaDB message store

Preface: 
The JMS specification supports two types of message delivery: persistent and non-persistent. A message delivered with the persistent delivery property must be logged to stable storage. For nonpersistent messages, a JMS provider must make best efforts to deliver the message, but it won’t be logged to stable storage. 

ActiveMQ supports both of these types of message delivery and can also be configured to support message recovery, an in-between state where messages are cached in memory. ActiveMQ supports a pluggable strategy for message storage and provides storage options for in-memory, file-based, and relational databases. 

Persistent messages are used if you want messages to always be available to a message consumer after they’ve been delivered to the broker, even if that consumer isn’t running when the message was sent. Once a message has been consumed and acknowledged by a message consumer, it’s typically deleted from the broker’s message store. 

Nonpersistent messages are typically used for sending notifications or real-time data. You should use nonpersistent messages when performance is critical and guaranteed delivery of the message isn’t required. 

How are messages stored by ActiveMQ? 
It’s important to gain some basic knowledge of the storage mechanisms for messages in an ActiveMQ message store. This will aid in configuration and provide an awareness of what takes place in the ActiveMQ broker during the delivery of persistent messages. Messages sent to queues and topics are stored differently, because there are some storage optimizations that can be made with topics that don’t make sense with queues, as we’ll explain. 

Storage for queues is straightforward—messages are basically stored in first in, first out order (FIFO). See figure 5.1 for a depiction of this. One message is dispatched to a single consumer at a time. Only when that message has been consumed and acknowledged can it be deleted from the broker’s message store. 
 

For durable subscribers to a topic, each consumer gets a copy of the message. In order to save storage space, only one copy of a message is stored by the broker. A durable subscriber object in the store maintains a pointer to its next stored message and dispatches a copy of it to its consumer as shown in figure 5.2. The message store is implemented in this manner because each durable subscriber could be consuming messages at different rates or they may not all be running at the same time. Also, because every message can potentially have many consumers, a message can’t be deleted from the store until it’s been successfully delivered to every interested durable subscriber. 
 

The KahaDB message store: 
The recommended message store for general-purpose messages since ActiveMQ version 5.3 is KahaDB. This is a file-based message store that combines a transactional journal, for reliable message storage and recovery, with good performance and scalability. 

The KahaDB store is a file-based, transactional store that’s been tuned and designed for the fast storage of messages. The aim of the KahaDB store is to be easy to use and as fast as possible. Its use of a file-based message database means there’s no prerequisite for a third-party database. This message store enables ActiveMQ to be downloaded and running in literally minutes. In addition, the structure of the KahaDB store has been streamlined especially for the requirements of a message broker. 

The KahaDB message store uses a transactional log for its indexes and only uses one index file for all its destinations. It’s been used in production environments with 10,000 active connections, each connection having a separate queue. The configurability of the KahaDB store means that it can be tuned for most usage scenarios, from high throughput applications (for example, trading platforms), to storing large amounts of messages (for example, GPS tracking). 

To enable the KahaDB store for ActiveMQ, you need to configure the  element in the activemq.xml configuration file. Here’s a minimal configuration for the KahaDB message store: 

If you want to embed an ActiveMQ broker inside an application, the message store can also be configured programmatically. Here’s an example of a programmatic configuration for KahaDB: 

 

Although the example seems small, it’s enough to create an ActiveMQ broker using the KahaDB message store and listen for ActiveMQ clients connecting over TCP. For more information about embedding ActiveMQ, see chapter 8. 

The KahaDB message store internals 
The KahaDB message store is the fastest of all the provided message store implementations. Its speed is the result of the combination of a fast transactional journal comprised of data log files, the highly optimized indexing of message IDs, and in-memory message caching. Figure 5.3 provides a high-level diagram of the KahaDB message store. 
 

The diagram provides a view of the three distinct parts of the KahaDB message store including the following: 
- The data logs 

The data logs act as a message journal, which consists of a rolling log of messages and commands (such as transactional boundaries and message deletions) stored in data files of a certain length. When the maximum length of the currently used data file has been reached, a new data file is created. All the messages in a data file are reference counted, so that once every message in that data file is no longer required, the data file can be removed or archived. In the data logs, messages are only appended to the end of the current data file, so storage is fast.

- The cache 

The cache holds messages temporarily if there are active consumer(s) for the messages. If there are active consumers, messages are dispatched at the same time they’re scheduled to be stored. If messages are acknowledged in time, they don’t need to be written to disk.

- The BTree indexes 

The BTree indexes hold references to the messages in the data logs that are indexed by their message ID. The indexes maintain the FIFO data structure for queues and the durable subscriber pointers to their topic messages. The redo log is used only if the ActiveMQ broker hasn’t shut down cleanly, and are used to insure the integrity of the BTree index is maintained.

The KahaDB message store directory structure: 
When you start an ActiveMQ broker configured to use a KahaDB store, a directory will automatically be created in which the persistent messages are stored. This directory structure is shown in figure 5.4. 
 

Inside of the KahaDB directory, the following directory and file structures can be found: 
- db log files 

KahaDB stores messages into data log files named db-.log of a predefined size. When a data log is full, a new one will be created, and the log number incremented. When there are no more references to any of the messages in the data log file, it’ll be deleted or archived.

- archive directory 

This exists only if archiving is enabled. The archive is used to store data logs that are no longer needed by KahaDB, making it possible to replay messages from the archived data logs at a later point. If archiving isn’t enabled (the default), data logs that are no longer in use are deleted from the file system.

- db.data 

This file contains the persistent BTree indexes to the messages held in the message data logs.

- db.redo 

This is the redo file, used for recovering the BTree indexes if the KahaDB message store starts after a hard stop.

Configuring the KahaDB message store: 
The KahaDB message store can be configured in the activemq.xml file. Its configuration options control the different tuning parameters (table 5.1): 
 
Table 5.1 Configuration options available for the KahaDB message store 

ActiveMQ provides a pluggable API for message stores, and there are three additional implementations to KahaDB that are shipped with ActiveMQ: 

* The AMQ message store—A file-based message store designed for performance
* The JDBC message store—A message store based on JDBC
* The Memory message store—A memory-based message store

We’ll look at the use cases and configuration for these additional message stores in the next three sections.