2023 Sessions On-Demand


All Sessions

Introducing Quix Streams—a Python-Kafka Library for Data-Intensive Workloads

Tomas Neubauer

Session Speaker
CTO, Quix
Tomas Neubauer is co-founder and CTO at Quix, responsible for the technical direction of the company across the full technical stack, and working as a technical authority for the engineering team. He was previously the technical lead at McLaren, where he led architecture uplift for Formula One racing real-time telemetry acquisition. He later led platform development outside motorsport, reusing the know-how he gained from racing.

This talk will introduce Quix Streams, an open-source Python library for data-intensive workloads on Kafka.

We will discuss the unique problems that this library is designed to solve, and how it was shaped by the challenges building a Kafka-based solution for Formula 1 cars at McLaren—a solution that needed to process a colossal firehose of sensor data coming in at thousands of samples per second. We’ll also explain why we decided to combine a Kafka API approach with a stream processing library and provide developers with a familiar Pandas DataFrame-like interface.

You’ll also see the library in action with a sentiment analysis demo. In this demo, we’ll calculate sentiment scores for incoming messages in a demo chat app—all in real time, using the HuggingFace Transformer’s API. At the end, we will connect to Twitter streaming API to send a high volume of data into the pipeline to simulate this use case at scale.

You’ll see how the library can simplify tasks such as:
– Subscribing to topics, deserializing incoming messages into table rows
– Running calculations on a rolling window of messages
– Using memory states to apply different functions such as aggregation or filtering
– Automatically outputting the results of calculations into downstream topics
– Managing state without the hassle of checkpointing and queues

Also, we want to give everyone ‌the opportunity to explore the library themselves. We’ll share the library’s GitHub repo and getting started tutorial and show attendees how they can get the sentiment analysis demo up and running in their own environments.