Cribl puts your IT and Security data at the center of your data management strategy and provides a one-stop shop for analyzing, collecting, processing, and routing it all at any scale. Try the Cribl suite of products and start building your data engine today!
Learn more ›Evolving demands placed on IT and Security teams are driving a new architecture for how observability data is captured, curated, and queried. This new architecture provides flexibility and control while managing the costs of increasing data volumes.
Read white paper ›Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure.
Learn more ›Cribl Edge provides an intelligent, highly scalable edge-based data collection system for logs, metrics, and application data.
Learn more ›Cribl Search turns the traditional search process on its head, allowing users to search data in place without having to collect/store first.
Learn more ›Cribl Lake is a turnkey data lake solution that takes just minutes to get up and running — no data expertise needed. Leverage open formats, unified security with rich access controls, and centralize access to all IT and security data.
Learn more ›The Cribl.Cloud platform gets you up and running fast without the hassle of running infrastructure.
Learn more ›Cribl.Cloud Solution Brief
The fastest and easiest way to realize the value of an observability ecosystem.
Read Solution Brief ›AppScope gives operators the visibility they need into application behavior, metrics and events with no configuration and no agent required.
Learn more ›Explore Cribl’s Solutions by Use Cases:
Explore Cribl’s Solutions by Integrations:
Explore Cribl’s Solutions by Industry:
Watch On-Demand
3 ways to fast-track your data lake strategy without being a data expert
Watch On-Demand ›Try Your Own Cribl Sandbox
Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Get inspired by how our customers are innovating IT, security and observability. They inspire us daily!
Read Customer Stories ›Sally Beauty Holdings
Sally Beauty Swaps LogStash and Syslog-ng with Cribl.Cloud for a Resilient Security and Observability Pipeline
Read Case Study ›Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Transform data management with Cribl, the Data Engine for IT and Security
Learn More ›Cribl Corporate Overview
Cribl makes open observability a reality, giving you the freedom and flexibility to make choices instead of compromises.
Get the Guide ›Stay up to date on all things Cribl and observability.
Visit the Newsroom ›Cribl’s leadership team has built and launched category-defining products for some of the most innovative companies in the technology sector, and is supported by the world’s most elite investors.
Meet our Leaders ›Join the Cribl herd! The smartest, funniest, most passionate goats you’ll ever meet.
Learn More ›Whether you’re just getting started or scaling up, the Cribl for Startups program gives you the tools and resources your company needs to be successful at every stage.
Learn More ›Want to learn more about Cribl from our sales experts? Send us your contact information and we’ll be in touch.
Talk to an Expert ›June 23, 2021
Global Keyword Search (aka CLUI) was introduced in LogStream 2.4.0. This feature enables the user to press Ctrl+K (all platforms) or Cmd+K (MacOS) and search across LogStream objects by keyword. Before CLUI was introduced our users had no way to retrieve search results from multiple resources/endpoints at the same time. This meant search results coming from multiple single-endpoints, such as, Sources, Destinations, Collectors, Routes, Pipelines, Functions, Knowledge Libraries, etc. would would force them to go into individual pages and search them one by one, creating unnecessary overhead and non-value-add processing steps just to collect all that required data. At Cribl, we value Customers first, always, hence we came up with a better and more streamlined solution featured in CLUI.
The figure below demonstrates the application of CLUI via a single query where the search results from different endpoints are returned into a single screen.
To power this search engine and provide customers with the best possible experience, we needed to come up with a way to get those results while meeting a lot of expectations:
The first thing that comes to mind is to directly access the data layer. It would work, but we also had to keep in mind security, visibility and access managed by RBAC Roles.
Choosing to send requests to multiple existing endpoints, allows us to reuse existing functionalities, validators and middlewares without implementing any additional changes to them (aka DRY). Criblianians, however, are not dry or lazy; we goats have flavor and real swag! We work hard to earn our goats (who we absolutely love by the way! Have you heard about our goat mascot Ian? Read more about Ian and our origin story here.)
The simplest, quickest and most obvious way to do it is to send standard HTTP requests to endpoints providing us with data. It matches our DRY philosophy and reuses all existing endpoints, middlewares and validators.
The diagram below explains the data flow of a single query request from a user and how it triggers multiple single-point API requests before returning search results.
The code below shows a simplified example of how we can trigger standard HTTP requests from route handler to another endpoint on the same server.
const httpRequest = (url) => {
return new Promise(async (resolve, reject) => {
const req = http.request(
url,
{
headers: {
'Content-Type': 'application/json; charset=utf-8'
},
},
res => {
let resData = '';
res.on('data', function (chunk) {
resData += chunk;
});
res.on('end', async function () {
if (res.statusCode === 200) {
resolve(resData);
} else {
reject(res.statusMessage);
}
});
res.on('error', reject);
},
);
req.on('error', reject);
req.end();
});
};
app.get('/internal-simple', async (req, res) => {
const internalResponse = await httpRequest(`${apiUrl}/foo`);
res.json(JSON.parse(internalResponse));
});
The code above allowed us to match all the requirements, but did not account for performance. When implemented, each request opened up a new connection to the server. This was manageable when we submitted queries with 1 or 2 requests. However, when tested with 20 or more requests for each search query (which is a typical load for our customers) it led us to performance issues, drastically increasing search response time and unnecessarily overburdening the entire system.
The GOAT solution was to send internal express requests, by making requests directly to the router without creating any new connections to the server.
The diagram below explains the data flow of a single query request from a user, and how it directly communicates with the router multiple multiple times without making connections to Express server, before returning search results.
Code below shows a simplified example of how we can directly send requests to the router while not having to create a new connection to the Express server.
Crucial part of this approach is replacing standard server response with an instance of our customized response class. It’s important to catch and handle all write and end executions to follow our custom pattern instead of default behaviour. Such approach also allows us to do some additional optimizations, e.g. prevent unneeded serialization which will be done anyway when returning CLUI response.
class InternalRestResponse extends http.ServerResponse {
promise;
#resolve;
#reject;
#content;
constructor(req) {
super(req);
this.promise = new Promise((resolve, reject) => {
this.resolve = resolve;
this.reject = reject;
})
}
// override json to prevent unneeded serialization in non proxy responses
json = (result) => {
this.resolve(result);
}
// override end which will be executed in case of an error or successful RESTProxy request
end = () => {
if (this.statusCode === 200) {
try {
this.resolve(JSON.parse(this.content));
} catch (e) {
this.statusCode = 500;
this.statusMessage = e.message;
this.handleError();
}
} else {
this.handleError();
}
}
// handle chunks from RESTProxy request
write = (...attrs) => {
const cb = typeof attrs[1] === 'function' ? attrs[1] : attrs[2];
const encoding = typeof attrs[1] === 'function' ? undefined : attrs[1];
this.content += attrs[0].toString(encoding);
cb?.();
return true;
}
writeHead = (statusCode) => {
if (statusCode !== 200) {
this.handleError();
}
return this;
}
handleError = () => {
this.reject({
code: this.statusCode,
message: this.statusMessage,
});
};
}
const httpInternalRequest = async (originalReq, url, opts = {}) => {
const req = new http.IncomingMessage({});
Object.assign(req, {
url,
method: 'GET',
headers: originalReq.headers,
httpVersion: originalReq.httpVersion,
httpVersionMajor: originalReq.httpVersionMajor,
httpVersionMinor: originalReq.httpVersionMinor,
}, opts);
const res = new InternalRestResponse(req);
app.handle(req, res);
return res.promise;
};
app.get('/internal-final', async (req, res) => {
const internalResponse = await httpInternalRequest(req, `${apiUrl}/foo`);
res.json(internalResponse);
});
A fully working sample app is available under https://github.com/lukaszwilk/internal-express-requests.
The fastest way to get started with Cribl LogStream is to sign-up at Cribl.Cloud. You can process up to 1 TB of throughput per day at no cost. Sign-up and start using LogStream within a few minutes..
Rick Salsa Apr 17, 2024
Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari
Got one of those handy?