2023 Sessions On-Demand


All Sessions

Deeply Declarative Data Pipelines

Ryanne Dolan

Session Speaker
Data Pipelines, LinkedIn
Ryanne works on data pipelines and streaming infra at LinkedIn. Primary author of MirrorMaker2. Previously at Twitter, Cloudera, Hortonworks.

With Flink and Kubernetes, it’s possible to deploy stream processing jobs with just SQL and YAML. This low-code approach can certainly save a lot of development time. However, there is more to data pipelines than just streaming SQL. We must wire up many different systems, thread through schemas, and, worst-of-all, write a lot of configuration.

In this talk, we’ll explore just how “declarative” we can make streaming data pipelines on Kubernetes. I’ll show how we can go deeper by adding more and more operators to the stack. How deep can we go?