back
February 22, 2022
|
Industry insights

The evolution of data processing: why time to value is paramount

Last decade, companies wrestled with big data. The new challenge is how to handle data fast. Here’s how market leaders generate more value by processing and acting on data immediately.

App icons on black and colorful background.
Quix offers a pure Python framework for building real-time data pipelines. It's a Kafka client with a stream processing library rolled into one. No JVM, no cross-language debugging—just a simple Pandas-like API for handling streaming data. Deploy in your stack or on Quix Cloud for scalable, stateful, and fault tolerant stream processing.

Last decade, companies wrestled with big data. The new challenge is how to handle data fast.

Data empowers business leaders to make decisions that increase revenue, enables people to automate machines and power products that can revolutionize every industry, from gaming to finance, mobility, telco and ecommerce.

“Data infrastructure is vital to the delivery of critical services and is required for the functioning of essential sectors of the economy, including financial systems, public utilities, industrial supply chains, media channels and telecommunications.”
“Digital revolution expanding infrastructure investment universe,” in WTW (formerly Willis Towers Watson)

The collection and use of data have expanded and evolved. Now there’s room to get even more value from your data, no matter where your company is in its data journey. In this post, I’ll talk a little about the evolution of data and then focus on how companies can get more from their data by reducing the time between data creation and its use.

The evolution of data from historical to real time

We’ve seen data trends evolve from the relational databases of the 90s through the explosion of solutions to manage big data over the past decade. The 2020s are about speed — enabling people to transform massive volumes of data into action faster.

Data evolution timeline.

Most data professionals see the value in fast or real-time data. Ninety percent of UK and US data professionals and IT decision-makers said they need to increase investment in real-time data analytics solutions in the near term, according to a survey by KX. But analytics is just the tip of the iceberg when it comes to processing data in real time.

Identifying use cases where faster data yields more value

Stream processing means more than simply building real-time dashboards. In most cases, overnight batch processing is sufficient for a dashboard that gets looked at a few times a week.

The real value comes from operationalizing data without going through the whole ETL (extract, transform and load) or ELT process. What’s more, adding context to data as it streams enables people across your organization to better access and use it.

Data loses value the longer it sits

The more time that passes between an event taking place and data about that event resurfacing in a usable form, the less valuable that data will be. Data provides the greatest value when it’s put to work immediately.

Use cases where latency makes data less valuable or irrelevant

I recently wrote about how reducing latency delivers a better customer experience, but speed is even more valuable in cases where a timely response is critical. Consider the difference fast data makes in these examples:

  • Fraud detection: Detecting a suspicious spending pattern and freezing a credit card in real time prevents loss far better than spotting the same pattern the next day.
  • User sentiment: Spotting a change in user sentiment in minutes instead of hours can make the difference between being in sync with your customers and sounding out of touch.
  • Predictive or preventive action: Applications that monitor and react to IoT data in real time can prevent problems such as broken machinery in manufacturing and adverse events in the ICU.


“Data is a critical component of success for all fast-growth B2B companies.”
Dawn Capital

When development cycles get slowed down by data demands

With so many potential applications for data, more and more people within an organization want access to data to build workflows, products and better customer experiences. Companies collect data, but their teams often lack the expertise to clean, transform and properly store data for broader use. Even data scientists and engineers can struggle to wade through the massive amount of data that gets dumped into data lakes and warehouses.

Faced with data-related delays, Uber decided to make data more accessible without relying on a handful of data engineers.

Uber’s growing population of data analysts and city operations managers dedicated hours or even days to figure out, test and review code before putting it into production — and then had to wait for the next deployment cycle to release it. An internal study estimated that these delays cost them millions in lost productivity.

“For a fast-moving company like Uber, such delays hurt our business. An internal study estimated millions of dollars of productivity loss due to this.”
“No Code Workflow Orchestrator for Building Batch & Streaming Pipelines at Scale,” in the Uber Engineering Blog

Inspired by low-code/no-code platforms, Uber’s engineers designed a system that put the complexities of using real-time data in the background. By moving the complexity of processing data upstream, companies make data more accessible across the organization.

Like Uber’s engineering team (but without requiring you to hire legions of engineers), Quix manages the complexity of stream processing for organizations. Our vision is to empower companies of any size to benefit from a no-code/low-code interface.

What’s your next step to reduce your data’s time to value?

Companies already invest in data teams and infrastructure. More than 90% of companies participating in NewVantage Partners annual Data and AI Leadership Executive Survey said they are increasing investments in data and AI initiatives. And these companies see results. 92% of participating companies reported measurable business benefits this year, up from just 48% in 2017 and 70% in 2020.

We see a lot of companies moving from a database approach to data in motion. Companies progress from a limited consumption of a single data stream to unlocking the value of streaming data by ingesting more streams across the organization.

Here’s how we’re seeing leading companies progress on their journey from 2010s-era big data storage to 2020s-era stream processing:

  • Streaming infrastructure: Connect siloed systems and teams with streaming infrastructure. At this stage, companies still write data to a database. It becomes someone else’s problem to figure out how data can be helpful down the road.
  • Stream processing: Data teams use stream processing to gain intelligence from data as it moves from source to destination. This can start with small projects and expand throughout an organization.
  • Expanding access to data: Unlock expertise in the organization by giving more people access to data when it’s at its most valuable. This can be done in a development environment that mirrors the company’s actual, real-time data — but the user’s sandbox is isolated so there is no possibility of breaking production environments.

How Quix helps teams transform data into products, faster

Getting data from the edge to a product is where Quix shines. We help small data teams build data pipelines more quickly and we help big teams skip over the complexity of building reliable stream processing architecture.

That means Quix helps your teams get to the fun stuff faster.

If you’d like to learn more about how Quix could help, book a consultation or join our Slack community to chat with us.

What’s a Rich Text element?

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

Static and dynamic content editing

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to customize formatting for each rich text

Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.

Related content

Featured image for the "Navigating stateful stream processing" post published on the Quix blog
Industry insights

Navigating stateful stream processing

Discover what sets stateful stream processing apart from stateless processing and read about its related concepts, challenges and use cases.
Tim Sawicki
Words by
windowing in stream processing
Industry insights

A guide to windowing in stream processing

Explore streaming windows (including tumbling, sliding and hopping windows) and learn about windowing benefits, use cases and technologies.
Daniil Gusev
Words by
real time feature engineering architecture diagram
Industry insights

What is real-time feature engineering?

Pre-computing features for real-time machine learning reduces the precision of the insights you can draw from data streams. In this guide, we'll look at what real-time feature engineering is and show you a simple example of how you can do it yourself.
Tun Shwe
Words by