fast-persistent-queue

Latest Maven Site Build

0.0.14

What is It?

fast-persistent-queue (FPQ) is an attempt to make a mostly FIFO, durable, fault-tolerant, deliver-at-least-once queue. If your needs are for a strictly FIFO queue, then this isn't the one for you (at least not at the moment). If all is going well, all events are written to an append only log file for durability and only pushed/popped in-memory.

The memory management is setup in segments and are paged in/out from disk - totally separate from the log file. When the threshold for memory usage is exceeded, the newest *complete* segment will be paged to disk, leaving the newest *incomplete* segment for pushing, and some of the oldest segments for "popping".

When popping from the queue, the next event from the oldest segment is returned, which gives us FIFO. However, if the needed segment is paged to disk the pop will use the next segment that *is* in memory - skipping some events, which breaks FIFO. As soon as the segment is in memory and ready, the next pop will return from the "just loaded" segment. This is why the queue is *mostly* FIFO.

The downside is that disk space is not considered important and will basically be doubled if the queue starts backing up. One copy of the data in the log files and the other in the segment page files. A premium is put on speed and therefore even when the queue is backing up, it is operating very fast since pops always come from memory, never (or very seldom) waiting on disk I/O.