2023 Sessions On-Demand

#

All Sessions

Interpolating Time Series Data On Demand

Lakshmanan Velusamy

Session Speaker
Stealth Startup
Engineering Lead, Stealth Startup
Lakshmanan is a Backend Engineer and Technical Lead with 16+ years of experience in building scalable backend systems and services. He is currently building an in-house, state of the art analytics platform from scratch to crunch data and empower customers with real-time insights.

Many real-world datasets are time series in nature, continuously recording measurements and status of entities over time. In time series datasets, missing data and gaps can occur frequently due to system failures or data processing errors. Time series interpolation (aka Gap filling) is a recently introduced feature in Pinot 0.11 that allows you to fill in missing data in your time series dataset.

When using gap filling in Pinot, you can configure the size of the gap to be filled, the type of interpolation algorithm to use (such as linear or spline), and the maximum number of missing data points that can be filled. Pinot also provides options to customize the behavior of gap filling for different use cases, such as supporting multi-dimensional gap filling and handling noisy data.

Gap filling is a powerful feature in Pinot that helps you to work with incomplete data sets more effectively and make more accurate predictions. It can also help you reduce the polling interval to store fewer data to improve storage cost and performance.