September 2, 2021
One of the most important aspects of security in LogStream deployments is protecting the communication channel between the workers and the leader instance. Cribl has made deploying LogStream a very easy process, but ease of deployment does not negate the complexity of TLS.
With my background in Information Security, I’ve worked with public key infrastructure for a long time, and want to demystify how to use Mutual TLS authentication with LogStream.
A LogStream Leader has two ports to secure: the user interface where you, as a human, interact with the configuration; and the distributed management port, which interacts with the workers. By default, the ports are 9000 and 4200, respectively. This post focuses on the latter.
There are two main steps to completing a TLS setup with mutual authentication. The first: enabling TLS on the Leader for the distributed management port. The second: enabling certificate authentication on the workers.
First, let’s tackle enabling TLS for the worker/leader channel. This is done on the Leader instance under Distributed Settings. Upload your private key and certificate information to global System Settings > Security > Certificates. Then, at System Settings > Distributed Settings > TLS Settings, select Enabled by sliding the toggle switch to Yes. From there, use the resulting controls to select the certificate you uploaded earlier.
On the workers, you will need to configure
$CRIBL_HOME/local/_system/instance.yml to ensure TLS is enabled. Here’s a snippet from the complete file (see cribl.yml reference file on the Docs site for more information):
distributed: tls: disabled: false
Note: if you are setting up a worker for the first time, and use the bootstrapping command, this setting in the configuration will automatically be changed to the appropriate value.
If you deploy your workers in a container, the
CRIBL_DIST_MASTER_URL can be configured with the TLS protocol in the URL to enable encrypted TLS communications with the Leader. This will override the default setting in
instance.yml. Here’s the format, from our Distributed Deployment documentation:
If you just want to enable TLS encryption with one-way authentication across this channel, you can stop here.
Now that the certificate is configured on the leader for encrypted TLS communications, we can implement the client certificate exchange to enable mutual authentication. This will allow LogStream to permit only explicitly authorized clients holding valid certificates to connect to the Leader.
When a client certificate is presented to a Leader, two things happen:
First, the leader validates the client certificate presented to LogStream. This setting is optional (enabling with the Validate Client Certs toggle), but highly recommended. The certificate is checked against the trust store to see if it has been signed by a valid certificate authority (CA). The list of certificates in the “CA Certificates Path” box is checked first (if populated), then against the list of built-in system certificates.
Second, the Leader checks if the common name (CN) matches the Regular Expression in the configuration. The LogStream default is to accept any value in the common name field. You can customize this as needed. Validation is performed against the value after the CN= string in the common name, so if your common name is
CN=logstream.worker, you would enter
logstream\.worker in the common name text box (including the backslash because the value entered is a regular expression).
On the leader, your config should now look something like this:
You’ll notice that the minimum and maximum TLS versions are also configured in the screenshot above. Additionally, the Common Name regex contains an additional check for
\d+ which allows for checking for the format:
To bootstrap a worker, only the shared authentication token is required. Certificates should be viewed as two-factor authentication. They’re meant to be out-of-band. Placing the certificates in the config bundle defeats the purpose of two-factor authentication.
Another item to consider when configuring TLS mutual authentication on workers: Place your certificates into a separate directory outside of
$CRIBL_HOME. If you place the certificates inside
$CRIBL_HOME, they’ll be removed when the next config bundle is deployed from the Leader.
To set up mutual authentication, add a few extra items need to the worker’s
instance.yml, as shown here:
distributed: tls: disabled: false privKeyPath: /path/to/certs/worker.key certPath: /path/to/certs/worker.pem caPath: /path/to/certs/root.pem requestCert: true rejectUnauthorized: true # false if ignoring untrusted certs
If the worker doesn’t connect, check
cribl.log on both the worker and leader for more context about the problem. You should see errors related to
dist leader communications.
If you’re using a self-signed certificate, or a certificate signed by an internal certificate authority, on your Leader, see the section below on considerations for self-signed certificate authorities. You may need to add an extra setting to adjust your environment settings.
This section provides some basic steps you can follow to build your own self-signed certificate authority. We don’t recommend running this in high security environments and should be used for testing purposes only.
openssl genrsa -out root.key 2048
openssl req -x509 -new -nodes -key root.key -sha256 -days 3165 -out root.pem -subj '/CN=LogStream CA'
openssl genrsa -out leader.key 2048
openssl req -new -key leader.key -out leader.csr -subj '/CN=leader.cribl'
openssl genrsa -out worker.key 2048
openssl req -new -key worker.key -out worker.csr -subj '/CN=worker.cribl'
openssl x509 -req -in leader.csr -CA root.pem -CAkey root.key -CAcreateserial -out leader.pem -days 365 -sha256
openssl x509 -req -in worker.csr -CA root.pem -CAkey root.key -CAcreateserial -out worker.pem -days 365 -sha256
You should now have nine files:
The leader and worker key and pem files should be used on the respective systems. The
root.pem file is the root CA and should be used in the CA config and also for the
NODE_EXTRA_CA_CERTS environment variable on the workers.
If the Leader uses a certificate that’s not trusted by a trusted system root certificate (especially if you used the section above on building your own self-signed certificate authority), you may need to add a PEM certificate to a file and add that path to the
NODE_EXTRA_CA_CERTS environment variable. Consult our docs’ Securing topic for more information.
In closing, we’ve walked through setting up LogStream with Workers using certificate-based authentication to connect to the Leader node. Now all communications are encrypted and require a valid certificate to be presented. I hope this helps you secure your LogStream environment! Join us on the Cribl Slack Community to discuss or for any assistance.
The fastest way to get started with Cribl LogStream is to sign-up at Cribl.Cloud. You can process up to 1 TB of throughput per day at no cost. Sign-up and start using LogStream within a few minutes.