One of the most important aspects of security in Cribl Stream deployments is protecting the communication channel between the workers and the leader instance. Cribl has made deploying Stream a very easy process, but ease of deployment does not negate the complexity of TLS.

With my background in Information Security, I’ve worked with public key infrastructure for a long time, and want to demystify how to use Mutual TLS authentication with Cribl Stream.

A Stream Leader has two ports to secure: the user interface where you, as a human, interact with the configuration; and the distributed management port, which interacts with the workers. By default, the ports are 9000 and 4200, respectively. This post focuses on the latter.

There are two main steps to completing a TLS setup with mutual authentication. The first: enabling TLS on the Leader for the distributed management port. The second: enabling certificate authentication on the workers.

Secure Stream Worker-to-Leader: Setting up the Encrypted Channel

First, let’s tackle enabling TLS for the worker/leader channel. This is done on the Leader instance under Distributed Settings. Upload your private key and certificate information to global System Settings > Security > Certificates. Then, at System Settings > Distributed Settings > TLS Settings, select Enabled by sliding the toggle switch to Yes. From there, use the resulting controls to select the certificate you uploaded earlier.

On the workers, you will need to configure $CRIBL_HOME/local/_system/instance.yml to ensure TLS is enabled. Here’s a snippet from the complete file (see cribl.yml reference file on the Docs site for more information):

distributed:
    tls:
      disabled: false

Note: if you are setting up a worker for the first time, and use the bootstrapping command, this setting in the configuration will automatically be changed to the appropriate value.

If you deploy your workers in a container, the CRIBL_DIST_MASTER_URL can be configured with the TLS protocol in the URL to enable encrypted TLS communications with the Leader. This will override the default setting in instance.yml. Here’s the format, from our Distributed Deployment documentation:

CRIBL_DIST_LEADER_URL=tls://<authToken>@leader:4200

If you just want to enable TLS encryption with one-way authentication across this channel, you can stop here.

Setting Up Workers to Present Certificates to the Leader

Now that the certificate is configured on the leader for encrypted TLS communications, we can implement the client certificate exchange to enable mutual authentication. This will allow Stream to permit only explicitly authorized clients holding valid certificates to connect to the Leader.

When a client certificate is presented to a Leader, two things happen:

First, the leader validates the client certificate presented to Stream. This setting is optional (enabling with the Validate Client Certs toggle), but highly recommended. The certificate is checked against the trust store to see if it has been signed by a valid certificate authority (CA). The list of certificates in the “CA Certificates Path” box is checked first (if populated), then against the list of built-in system certificates.

Second, the Leader checks if the common name (CN) matches the Regular Expression in the configuration. The Stream default is to accept any value in the common name field. You can customize this as needed. Validation is performed against the value after the CN= string in the common name, so if your common name is CN=logstream.worker, you would enter logstream\.worker in the common name text box (including the backslash because the value entered is a regular expression).

On the leader, your config should now look something like this:

You’ll notice that the minimum and maximum TLS versions are also configured in the screenshot above. Additionally, the Common Name regex contains an additional check for \d+ which allows for checking for the format: logstream1.worker, logstream2.worker, logstream3.worker, etc.

So Why Can’t I Just Bootstrap a Worker with the Mutual Certificates Already Populated?

To bootstrap a worker, only the shared authentication token is required. Certificates should be viewed as two-factor authentication. They’re meant to be out-of-band. Placing the certificates in the config bundle defeats the purpose of two-factor authentication.

Another item to consider when configuring TLS mutual authentication on workers: Place your certificates into a separate directory outside of $CRIBL_HOME. If you place the certificates inside $CRIBL_HOME, they’ll be removed when the next config bundle is deployed from the Leader.

To set up mutual authentication, add a few extra items need to the worker’s instance.yml, as shown here:

distributed:
    tls:
      disabled: false
      privKeyPath: /path/to/certs/worker.key
      certPath: /path/to/certs/worker.pem
      caPath: /path/to/certs/root.pem
      requestCert: true
      rejectUnauthorized: true # false if ignoring untrusted certs

Or if using environment variables:

With the changes made, restart the worker. You should see the worker reconnect to the Leader successfully.

If the worker doesn’t connect, check cribl.log on both the worker and leader for more context about the problem. You should see errors related to dist leader communications.

If you’re using a self-signed certificate, or a certificate signed by an internal certificate authority, on your Leader, see the section below on considerations for self-signed certificate authorities. You may need to add an extra setting to adjust your environment settings.

Building your own Certificate Authority

This section provides some basic steps you can follow to build your own self-signed certificate authority. We don’t recommend running this in high security environments and should be used for testing purposes only.

Create root certificate authority (CA)

openssl genrsa -out root.key 2048

openssl req -x509 -new -nodes -key root.key -sha256 -days 3165 -out root.pem -subj '/CN=LogStream CA'

Create Leader certificate signing request (CSR)

openssl genrsa -out leader.key 2048

openssl req -new -key leader.key -out leader.csr -subj '/CN=leader.cribl'

Create Worker certificate signing request (CSR)

openssl genrsa -out worker.key 2048

openssl req -new -key worker.key -out worker.csr -subj '/CN=worker.cribl'


Sign Requests with Root Certificate Authority

openssl x509 -req -in leader.csr -CA root.pem -CAkey root.key -CAcreateserial -out leader.pem -days 365 -sha256

openssl x509 -req -in worker.csr -CA root.pem -CAkey root.key -CAcreateserial -out worker.pem -days 365 -sha256

You should now have nine files: leader.csr, leader.key, leader.pem, worker.csr, worker.key, worker.pem, root.key, root.pem, root.srl.

The leader and worker key and pem files should be used on the respective systems. The root.pem file is the root CA and should be used in the CA config and also for the NODE_EXTRA_CA_CERTS environment variable on the workers.

Self-Signed or Internal Certificate Authority Considerations

If the Leader uses a certificate that’s not trusted by a trusted system root certificate (especially if you used the section above on building your own self-signed certificate authority), you may need to add a PEM certificate to a file and add that path to the NODE_EXTRA_CA_CERTS environment variable. Consult our docs’ Securing topic for more information.

Conclusion

In closing, we’ve walked through setting up Stream with Workers using certificate-based authentication to connect to the Leader node. Now all communications are encrypted and require a valid certificate to be presented. I hope this helps you secure your Cribl Stream environment! Join us on the Cribl Slack Community to discuss or for any assistance.

The fastest way to get started with Cribl Stream is to sign-up at Cribl.Cloud. You can process up to 1 TB of throughput per day at no cost. Sign-up and start using Stream within a few minutes.