2023 Sessions On-Demand

#

All Sessions

Leveraging Pinot’s Plugin Architecture to Build a Unified Stream Decoder

Mike Davis

Session Speaker
DoorDash
Software Engineer, Data Platform, DoorDash
Mike has been a Software Engineer within the Data Platform at DoorDash for two years within the Analytics Engine team and has over 10 years experience working in big data and streaming technologies.

Real-time events produced at DoorDash use an in-house framework, Iguazu, that embeds custom serialization metadata which is used for generic and dynamic stream processing. While this makes consuming these events easy for our microservice applications, this provides a challenge when attempting to consume these events more generally.

In this talk we’ll walk through how we’ve taken advantage of the highly extensible framework within Apache Pinot to develop a custom StreamMessageDecoder implementation incorporating the Iguazu consumer library. By leveraging this approach we are able to seamlessly integrate with the existing real-time ecosystem at DoorDash to power real-time analytics for any team or product at DoorDash.