Building a scalable and highly available messaging system! Lessons learned from collaboration with The Apache Qpid community

How to build a scalable and highly available messaging system and open source community

Abstract

In the performance demanding world of capital markets systems, features like scalability, high availability and high-throughput are second nature. The messaging system we have built at Murex is the foundation of our back-end architecture and will be used heavily for the efficient exchange of financial market events between Murex’s services.

After a year of research, implementation and testing, we have reached what we consider a magic formulae which incorporates all of the above features after testing several technologies. Having worked hard with the highly dedicated and helpful - Red Hat supported - community of Apache QPID, we feel SPA is the best place to share our knowledge and help others who have the same problems. Although the system we have built is unique, we think the lessons we have learned are universal and cover generic aspects including:

- Contributing to support new operating system platforms.
- Hosting a CI to continusouly check the compatibility of newly delivered community code with our use cases.
- Providing feedback to the community to improve NFRs (latency, throughput, etc.)
- Testing, reporting bugs and contributing new functionalities to the QPID communities
- Using the messaging system through the standard AMQP protocol
- Providing 'free' continuous integration for the QPID community's master branch
- Ability of a small team to leverage an open source, community backed projects to provide a high-standard solution for a tough problem

We look forward to sharing our experience with the participants at SPA conference and to collect their feedback too on our experience.

Audience background

Developers, Product Owners, Some messaging knowledge would be useful but not essential

Benefits of participating

By the end of the session, participants should have a pretty good idea how to deploy a scalable messaging service.
They will also learn how an effective open source community collaborates.

Materials provided

Slides

Process

Presentation by presenters from Red Hat and Murex
Q/A and discussion at the end

Detailed timetable

00:00 - 01:00: Presentation
01:00 - 01:15: Q/A and Discussion

Outputs

Slides

History

An adapted version of this presentatuon was given at Riviera Dev 2017

Presenters

1. Adel Boutros
   Murex SAS
2. Andrew Stitcher
   Red Hat