MIAX launched their first fully electronic options trading exchange in 2012. With the goal of offering technologically-advanced marketplaces catering to the needs of the trading community, they brought a second exchange online in 2017 and a third in 2019, now listing and trading options on over 2,600 multi-listed classes. MIAX launched their first cash equity exchange on September 25, 2020.
Under the hood, these exchanges consist of two major divisions: the front office and trading systems (including the core matching function, which matches buyers with sellers), and the back office systems which develops applications for operations, audit, reporting, settlement, listing, regulatory and compliance needs. The front office application stack is custom-built to prioritize speed and low latency, with thousands of processes for a single exchange. The back office stack streams in large quantities of contextual market data which are further used by the various application stacks and produces logs in a wide array of log source types.
How it Started
The MIAX Service Reliability Engineering (SRE) team originally used a prominent SIEM tool as an APM to process events related to the performance and behavior of the application stack. What they built wasn’t the optimal solution, but it achieved what the team needed at the time.
But with all the expansion that came with the new exchanges coming online, they wanted to stay ahead of the curve to ensure they could maintain the reliability, flexibility and scalability they had established for their marketplace while keeping costs reasonable.
Leadership tasked Gopal’s SRE team with finding a solution. Expanding their use of Splunk was deemed the obvious choice – offering many features, flexibility and scope for supporting their wide range of data sources natively. However, challenges remained.
“Sometimes application events generate large spikes of event activity you don’t necessarily need into the stream feeding Splunk. We need to be able to shape the event flow, control it – if there is really no logical business need for the data, we just want to drop it.”
Govardhanen “Gov” Gopal
Principal Technologist/SRE, MIAX
Gopal’s team wanted to find a way to pre-filter the data. He’d heard Cribl Stream offered this functionality, and decided to look into it.
“I immediately tried out the ‘drop’ function and thought, ‘This could definitely do what we need it to!’ ”
Govardhanen “Gov” Gopal
Principal Technologist/SRE, MIAX
As Gopal’s team rolled out Stream as part of their new equities exchange and as they dug in further, they continued to find more to be impressed with.
“I was able to edit and change things on the fly, then see in real time how all the configs were affecting my data. I can change a regex pattern or a lookup, and see events in, events out, how the same event is changed – without restarting anything!”
Govardhanen “Gov” Gopal
Principal Technologist/SRE, MIAX
Ultimately, the SRE team also deployed Stream for their options exchanges.
“It was much easier than expected! All the configuration files are stored in YAML, so we were able to literally copy/paste the routes/pipelines and other configs to another instance with no complications.”
Govardhanen “Gov” Gopal
Principal Technologist/SRE, MIAX
How It’s Going
With Stream in place and powering the data flow for all MIAX exchanges, Gopal and his team are benefiting from the Cribl user community; finding new ways to improve the quality of both the data they deliver and the speed at which they deliver it. The team is leveraging Stream to ingest events that are not so neatly delineated, and are using it to enrich existing events with more metadata which is indexed from the outset in Splunk and enhances the experience for every user of the system.
“The Cribl Community is incredibly responsive and helpful!”
Govardhanen “Gov” Gopal
Principal Technologist/SRE, MIAX
Gov’s team are able to enrich what they are sending to provide vital, time-saving context to the associates serving their customers.
“When our associates need market data from external sources, they don’t need us to go and look it up via external scripts to get the context – it’s already there for them. This makes the client support team faster and much more self-reliant.”
Govardhanen “Gov” Gopal
Principal Technologist/SRE, MIAX
A high value solution
The combination of low cost and strong functionality allows Cribl Stream to deliver a powerful solution in a very cost effective manner.
“Cribl is now integral to our Splunk APM solution. Every new Stream release offers many minor enhancements, but also some major functionality as well which really enhances this stack. At the end of the day, with all the growth we have had, the ability to alert, triage, diagnose and act on a production issue much more effectively is of great value.”
Govardhanen “Gov” Gopal
Principal Technologist/SRE, MIAX
And knowing when to react to a potential issue is a huge efficiency improvement too:
“The value-add is significant; this is a much more effective way of working. Because we’re able to process all these different log types, the universe of potential alerts has become much larger – but we can better identify the most critical issues.”
Govardhanen “Gov” Gopal
Principal Technologist/SRE, MIAX
Find out how your business can implement an observability pipeline to parse, restructure, and enrich data in flight, while cutting costs and simplifying operations.
Get Cribl, and take control of your data.