A long while back, we published a post highlighting our Reliable Queue implementation on Redis. A lot has happened since then, so let me take this opportunity to let you know where we are.

Updated Redis Client

We made the decision to move away from using Jedis as our Redis client. Jedis had served us well but was starting to show limitations at very high volumes. At Bronto, we regularly push tens of thousands of messages through our queues every second. Sustaining this throughput with Jedis's synchronous API forced us to open a *lot* of connections to our Redis servers, with very busy Redis servers sometimes managing thousands of connections. That's too many connections, and it was starting to have a measurable, negative impact on performance, with a particularly negative effect on replication. We replaced Jedis with our own, Bronto-built Redis client. Our client was built around an entirely asynchronous API, and the resulting command pipelining allowed us to dramatically increase the throughput we can get out of a single connection. Our API also assumes that binary payloads are the core of your interaction with Redis. We give you plenty of sugar to help you translate to and from common types (e.g. String), but we do not force you to consume yet another object marshaling framework.

Reliable Queue Improvements

We were happy with our previous Reliable Queue implementation, but there is always room for improvement. With our new Redis client in hand, we were able to to simplify and optimize existing behaviors, as well as add some new features we find very handy. The new feature list looks like this:
  • At least once delivery, as described in the previous post.
  • Per-Item deferment, allowing individual items to be 'parked' in the queue for some pre-determined time.
  • Opportunistic batching of many queue operations (e.g. enqueue, requeue, release).
  • Optional pre-fetching on the consumption side to decrease effective latency.
  • Performance instrumentation using the excellent Dropwizard (ex-Codahale) Metrics library.
With all of the goodies enabled, we can squeeze a lot of throughput out of a single connection. Synthetic benchmarks (shown below) were excellent, and our experience with real world loads (i.e. realistic queue names, payload sizes) show we can easily handle throughputs in the high tens of thousands. Some Benchmarks These are synthetic benchmarks, so they should be taken with a grain of salt. Some notes on real-world performance follow.
  • Bronto's Redis client implementation
  • Bronto's Reliable Queue implementation
  • Redis running on Intel(R) Xeon(R) CPU E5-2430 @ 2.20Ghz
  • All tests are single threaded, with one connection.
  • All tests use single byte queue name and item payload.
Screen Shot 2015-06-26 at 11.14.51 AMNot bad, and certainly enough to serve as a solid foundation for our distributed processing needs. When we move these synthetic benchmarks into the real world, we find that they hold up very well. Even with more threads producing and consuming, more connections to the Redis server, and larger key names and payloads, we have no trouble pushing high tens of thousands of events through a single Redis server. Map this onto even a modest-sized Redis cluster, and we have more than enough capacity for our needs. Disque A final side note on our Reliable Queue library: We created our initial implementation several years ago and have been refining it over time. Earlier this year, Antirez, the author of Redis, released an alpha version of Disque, a queue-centric variant of the Redis code base that addresses many of the same concerns that inspired us to create our library. We'll be keeping an eye on Disque as it matures, but for now, we are getting plenty of utility out of our implementation.