August 30, 2022
Industry insights

Stream processing at the edge makes “real-time” even faster

It turns out that “real time” is a bit of a misnomer — there’s always some lag between an event and its processing — even if it’s only a few milliseconds. But we can help decrease even that item.

Black background with turquoise lines.
Quix brings DataFrames and the Python ecosystem to stream processing. Stateful, scalable and fault tolerant. No wrappers. No JVM. No cross-language debugging.

Our library is open source—support the project by starring the repo.

If you’re seeking the fastest possible processing speed, “real time” might sound like the finish line. How can a system work quicker than that? It turns out that “real time” is a bit of a misnomer. There’s always some lag between an event and its processing — even if it’s only a few milliseconds.

Why work toward faster processing speeds?

If we can already process data in milliseconds, why do we need to go faster? A gamer might not perceive a difference of a dozen milliseconds. Still, higher latency in their gaming system can mean the difference between dodging an opponent’s death blow and being hit. Game over. It’s not just gaming systems that need to be fast. Self-driving cars, gunshot detection systems, medical alerts, and accident prevention work better when the data is processed faster. Just as gamers sit closer to their routers to pick up a few milliseconds, companies can use edge technology to reduce the distance data travels and increase processing speed.

Check out this simple game made with Quix’s stream processing solution for more on latency in gaming.

Processing data closer to the point of action reduces latency

Reducing the distance between processing and the event taking place reduces latency and increases speed, which has led to new technologies that operate between the cloud and the end user or event. These technologies fall into one of two groups: fog and edge.

Cloud fog and edge graphic.

How much faster is processing at the edge?

A group of researchers measured latency from 8,456 users to 6,341 Akamai edge servers and 69 cloud locations. The researchers found that edge servers can provide lower latency than cloud servers, as shown in the table below.

Edge vs cloud latency graphic.

What are some examples of edge processing?

Whether you use cloud, fog or edge processing will depend on your use case and tolerance for latency. Edge processing is a good choice for applications that rely on instant responses to data. Let’s look at some examples.

Covid detection and prevention in the workplace

Timing plays a significant role in Covid detection and prevention. The sooner you learn that you had contact with someone contagious, the more quickly you can act. Some companies use edge processing to decrease the risk of catching covid in the office. Bob Violino, a contributing writer at Network World, provides several examples in IoT helps make return-to-work safer.

Innovatus Capital Partners displays real-time, air-quality scores on screens throughout the facility to reassure employees. The system combines technologies from Veea edge-computing and Wynd Technologies, a leader in portable air purifiers. By placing small processing devices called Smart Edge Nodes throughout the building, Innovatus is able to collect, process data, and display data from Wynd IoT sensors in real time. Data is also sent to Wynd Technologies for further analysis.

Nutrien Ltd., a provider of farming products, used proximity-monitoring and contact-tracing technology from Triax Technologies to help enforce social distancing. Wearable trace tags were attached to employees’ hardhats or clothing. The tags flash if employees come within 6 feet of each other, providing an instant warning. Gateways strategically placed around the site collect data from the tags for use in contact tracing, reducing the manual work and helping the company stop potential outbreaks faster.

In both of these examples, edge processing reduces latency and traffic to the cloud. The closed system helps meet data security requirements for sensitive data, like employee locations.

Time-sensitive manufacturing processes

Some processes are more finicky than others, requiring constant monitoring and adjustments to produce the desired results. Pentair, a food and beverage industry filtration expert, already offers solutions that consolidate complex data streams and enable teams to optimize filtration processes. The company’s vision for the future is a self-adapting beer membrane filtration system.

The system, called BrewAssist, will use advanced technologies to autonomously find and implement the best filtration settings throughout the brewing process. This self-adapting system will reduce time spent on data extraction, modeling, and manual responses to optimize beer filtration in real time.

Preventative maintenance

A broken machine can mean workplace accidents, downtime and costly repairs. Preventative maintenance programs predict when parts should be replaced or serviced to keep equipment running optimally. Deploying IIoT sensors and onsite processing takes this one step further to enable companies to detect an imminent failure and shut machines down before an accident occurs. These systems can yield significant cost savings. Texmark Chemicals expects its IIoT-enable predictive maintenance program to reduce maintenance costs by 50 percent.

The hardware and software of edge processing

Tapping into the benefits of edge processing requires both the hardware — IIoT sensors, processing nodes, small data centers, networking infrastructure — and software that can collect, process and transform data streams in real time. The stream processing component is where many companies get stuck. Stream processing isn’t one thing. It’s a sophisticated combination of technologies, including data pipelines and message brokers. Figuring out the components and getting them to work together reliably is hard. Think years, not months, of work. This is where Quix fits in. We’ve built a reliable stream processing infrastructure and made it easy for developers to build their backend on it. With Quix, you can implement the sensors and processors you need and be sure that every bit of data is processed with as few milliseconds of lag as possible.

What’s a Rich Text element?

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

Static and dynamic content editing

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to customize formatting for each rich text

Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.

Related content

Featured image for the "Navigating stateful stream processing" post published on the Quix blog
Industry insights

Navigating stateful stream processing

Discover what sets stateful stream processing apart from stateless processing and read about its related concepts, challenges and use cases.
Tim Sawicki
Words by
windowing in stream processing
Industry insights

A guide to windowing in stream processing

Explore streaming windows (including tumbling, sliding and hopping windows) and learn about windowing benefits, use cases and technologies.
Daniil Gusev
Words by
real time feature engineering architecture diagram
Industry insights

What is real-time feature engineering?

Pre-computing features for real-time machine learning reduces the precision of the insights you can draw from data streams. In this guide, we'll look at what real-time feature engineering is and show you a simple example of how you can do it yourself.
Tun Shwe
Words by