August-PR-Images_PR Image - 1920x1005

Going native

July 8, 2019
Written by
Ledion Bitincka's Image

Ledion Bitincka is Cribl's CTO and co-founder. Ledion has over a decade of engineering ex... Read Moreperience developing next-generation technologies and leading the launch of enterprise products. He was the Advanced Development Architect at Splunk where he introduced Search-Time Schema and led the design of Hunk and SmartStore. Read Less

Categories: Engineering

Have your cake and eat it too … a post about how a fast feedback loop enables startups (or agile teams) to deliver the best value to customers.

Some background

We started working on Cribl LogStream almost exactly 1 year ago (early July 2018). In this time frame, we’ve managed to GA (Oct 2018) and release 6 more feature updates. Speed and quality are key to the survival of any startup. We’ll focus primarily on speed in this post, which is quite different than performance.

Back in 2018 when we started to design Cribl LogStream we had to choose a technology to build on. However, that decision had to fit in the larger business strategy, which put a high premium on time-to-market (speed) as well as customer adoption of any technology we’d depend on. While we had experience on a broad range of languages (C/C++, Java, GO, Python,…) what seemed to be an overall best fit, both business and problem space was JavaScript. Yes, you read that right – JavaScript seemed to fit very nicely for the problem space with the ability to easily do async IO, good package management, highly active development of both V8 and NodeJS, ability to drop to C/C++ iff needed, lots of hiring candidates etc. So far, that decision has paid off immensely.

Going native

Why change? There are two key disadvantages when taking a runtime as a dependency:

  1. your customers need to adopt that runtime and
  2. you need to support multiple versions.

Contrary to engineers’ belief, not everyone updates to the latest version 🙂 and enterprises have more speed bumps to upgrades.

We’re seeing early signs of both of those disadvantages on a few of our customers and any product area that slows down adoption becomes a prime candidate for being rethought and redone. This time, it is our packaging. What can we do to remove the runtime dependency? The options fall into the rewrite or repackage category – with rewrite failing the speed test. In repackage, the key concept is simply: instead of relying on the customer to provide and maintain the runtime our application depends on, we ship the runtime as part of the application.

It turns out that we could do better than just shipping node (our runtime) as part of Cribl LogStream. NodeJS and V8 are extremely friendly to “embedders” – applications that want to use them as just a component – tons of popular desktop applications do just that, Slack being a very popular one (using electron). NodeJS has the ability to bundle an application into the node binary, at compile time, effectively overriding the default behavior when ran. Thus, we can bundle our application into NodeJS, compile it for all the platforms we need to support and voila, now we have a native binary with no other dependencies! In the C/C++ world this is known as statically linked applications.

With this solution we get both benefits of building using a language that allows us to move fast and minimize customer adoption friction.

As part of this effort we are open sourcing a tool js2bin, that we’ve built to help us with the packaging of Cribl LogStream on multiple platforms.

Alternatives considered

Aren’t there any other tools out there that could do this (why build)? Before embarking on building js2bin we considered 2 other utilities for packing NodeJS applications into a single binary: pkg and nexe, but decided against using either for the following reasons:

  1. They both embed users application and other content by appending to the executable – while this method works, it can also trip malware scanners as the executable contains extraneous content.
  2. They non-trivially patch (see here or here) NodeJS source or it’s build tool chain which we believe leads to a brittle solution
  3. They both try to do a lot more than just embed your application into a native executable, they can pack other data files thus need to provide a virtual filesystem and modify your code while embedding. This goes contrary to our belief of doing something well defined very well.

Parting thoughts

When creating something new, almost all decisions are made using incomplete information. As such, it is your responsibility to actively observe, analyze and adapt as new information becomes available.

.
Blog
Feature Image

Cribl: Empowering Data Freedom with Open Standards and Unmatched Flexibility

Read More
.
Blog
Feature Image

Hello Vegas! Cribl @ AWS re:Invent 2024

Read More
.
Blog
Feature Image

Simplify OpenTelemetry Metrics with Cribl Edge OTLP Conversion

Read More
pattern

Try Your Own Cribl Sandbox

Experience a full version of Cribl Stream and Cribl Edge in the cloud with pre-made sources and destinations.

box

So you're rockin' Internet Explorer!

Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari

Got one of those handy?