September 29, 2022
Industry insights

The Stream — September 2022 edition

The September 2022 edition of The Stream: covering this month in stream processing on the internet.

The Stream September 2022 banner.

Stream processing is defining a new world for organizations that want to be able to use and make sense of data in the moment it is created. Unfortunately, this leading-edge technology is not easily and broadly understood, and even less easily wrangled into a cohesive, performant data pipeline.

In the Gartner market survey, “Adopt Stream Data Integration to Meet Your Real-Time Data Integration and Analytics Requirements,” the research company found that “In our annual survey for the data integration tools market, 47% of organizations reported that they need streaming data to build a digital business platform, yet only 12% of those organizations reported that they currently integrate streaming data for their data and analytics requirements.”

That means that just 5.6% of companies are actually harnessing the power of streaming data to generate immediate insights, power ML capabilities or feed fresh data back into their products. Looking at this on the adoption curve, this represents just the beginning: the so-called “innovators.”

The next chunk of companies to use stream processing — the 41.4% of companies who would fit either into an “early adopters” or “early majority” category — are aware they want and need to be able to use this technology, but can’t easily do so.

It’s tricky. Combining Kafka, Kubernetes, and many other newer technologies is not for the faint of heart (or light of wallet). Large-scale companies that have already deeply invested in developer resources here such as John Deere, and Alibaba are leading the charge, but where does that leave midmarket and digital-first companies with fewer resources?

This month, we’re connecting with organizations of all sizes and industries by hosting community meetups in London, Berlin, Munich and Austin, speaking at events like Big Data London, and joining the Kafka user community at Current in Austin in early October. Our goal is to bring the power of stream processing to everyone by building shared knowledge, understanding and great tools to democratize data stream processing. Join us.

An image of a cat along tech company logos.

Streaming-first infrastructure for real-time machine learning

A streaming infrastructure can improve ML prediction latency and continual learning. Chip Huyen and Anthony Alford explain why an event-driven microservices architecture is a better choice for using continual learning than a REST-based architecture.

Learn more

Logos of four open source stream processing technologies.

Explore four popular open-source stream processing technologies

Gary Stafford offers a technical two-part post comparing these stream processing projects: Apache Spark Structured Streaming, Apache Kafka Streams, Apache Flink, and Apache Pinot.

Explore more →

Zhamak Dehghani Big Data conference.

Understanding the Data Mesh with Zhamak Dehghani

Dig into one of the most talked about trends in data, the Data Mesh. Dehghani offers “atomic steps to rewire the sociotechnical backbone of your organization,” and how to apply it to get value from data rapidly, sustainably and at scale.

Read more →

Colorful event banner saying Current.

Destination: Austin, Texas for Current

It’s not just the smell of brisket in the air — it’s the Kafka user community coming together at Current, the conference for data professionals to learn about data streaming, Oct. 4-5.

Register now →

More insights

  • According to Memgraph, the most common use case for stream processing is monitoring sensor or device data.
  • In the latest Redpanda University course, you’ll learn about stateless and stateful stream processing and work through an advanced end-to-end tutorial. Enroll for free.
  • Ververica offers a new video on streaming concepts for Flink: “Exactly Once Fault Tolerance Guarantees.” Watch on YouTube.
  • What’s next for streaming analytics on Spark and Flink? Mike Ridley discusses complex event processing with Macrometa on YouTube.

What’s a Rich Text element?

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

Static and dynamic content editing

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to customize formatting for each rich text

Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.

Related content

Banner image for the article "Streaming ETL 101" published on the Quix blog
Industry insights

Streaming ETL 101

Read about the fundamentals of streaming ETL: what it is, how it works and how it compares to batch ETL. Discover streaming ETL technologies, architectures and use cases.
Tun Shwe
Words by
LLMOps: large language models in production with Quix
Industry insights

LLMOps: running large language models in production

LLMOps is a considered, well structured response to the hurdles that come with building, managing and scaling apps reliant on large language models. From data preparation, through model fine tuning, to finding ways to improve model performance, here is an overview of the LLM lifecycle and LLMOps best practices.
Tun Shwe
Words by
What is stream processing
Industry insights

What is stream processing?

An overview of stream processing: core concepts, use cases enabled, what challenges stream processing presents, and what the future looks like as AI starts playing a bigger role in how we process and analyze streaming data
Tun Shwe
Words by