Spotify's RAP Index Shows a New Way to Serve Industrial Data Lake Queries

Data Lakes Were Built for Scans, Not Lookups

If you've spent any time around industrial data pipelines, you know the drill. The historians, the SCADA logs, the MES records—they all pile up in a data lake. Great for batch analytics, terrible for answering a single question like "what was the vibration reading on pump 4 at 14:32:07?" That kind of point query is the bread and butter of operational systems, and it's exactly where traditional lake architectures fall apart.

Spotify recently shared their solution to this problem, and while it's aimed at music streaming, the pattern is directly relevant to industrial networking. They call it Random Access Parquet, or RAP. The idea is simple: keep the data in your lake, but add an external index that lets you jump straight to the exact row you need, without scanning thousands of files.

The Cost of Copying Data for Every Use Case

In industrial settings, we often end up with two copies of everything. One in the data lake for analytics, one in a time-series database or historian for operational queries. That duplication eats storage, creates sync headaches, and introduces latency when you need fresh data in both places.

Spotify faced the same problem at massive scale. They have petabytes in Bigtable for online serving and exabytes in Google Cloud Storage. Copying data between those systems was getting too expensive. Their answer was to stop copying and instead make the lake itself support fast point queries.

How RAP Works: An Index Layer Over Parquet

RAP builds an external index that maps query keys—like a pump ID or a timestamp—directly to the Parquet file and the row offset within that file. When a query comes in, the system looks up the key in the index, then issues a targeted range read from object storage. No full file scans, no listing thousands of objects.

The index is append-only. As new data lands in the Iceberg table, the index builder adds new entries without touching the immutable Parquet files. That's crucial because it preserves compatibility with your existing analytics tools. The same dataset continues to serve batch jobs, machine learning pipelines, and now, low-latency operational queries.

Why This Matters for Industrial Networking

Think about a modern plant floor. Sensors stream data to the edge, then to a central lake. Engineers want to run predictive maintenance models on that data. Operators need to look up a specific sensor reading in real time. Compliance teams want to audit historical values. All of these can now run against the same dataset.

Spotify's approach removes the need for a separate operational database just to serve point queries. That's a big deal for industrial IT teams who are tired of maintaining duplicate stacks.

Storage Layout Tweaks That Cut Latency Further

Spotify didn't stop at the index. They also introduced several layout optimizations to make point queries even faster. Sorting data by the query key reduces the number of files you need to touch. Clustering related records together helps too. And they interleave value columns so that a single read can pull multiple attributes in one go.

Some queries can even be served entirely from the index, without reading the Parquet file at all. That's like having a shortcut that bypasses the warehouse entirely. The trade-off is a modest increase in file or index size, but the reduction in storage operations is worth it.

Secondary Indexes for Multiple Query Dimensions

In industrial networks, you rarely query by just one key. You might want to find all events for a specific asset, or all alarms from a particular vendor, or everything that happened during a shift. Spotify's design supports secondary indexes that work across multiple dimensions—like buyer ID or seller ID in their case, but in our world, asset ID, sensor type, or location.

The clever part is that these secondary indexes are managed at the serving layer, so you can add new access paths without rewriting the Parquet files or changing your data pipeline. That flexibility is valuable when plant requirements evolve faster than your data architecture.

Building on Open Standards

RAP isn't a proprietary format. It builds on Apache Parquet and Apache Iceberg, both open standards. That means you can adopt this pattern without locking into a specific vendor. The index layer is an addition, not a replacement, so your existing tools keep working.

Google Cloud has also been pushing Apache Iceberg for AI and operational workloads, but Spotify's approach is different. Instead of a full lakehouse architecture, they added a dedicated index optimized for point queries. That's a lighter-weight solution that might be easier to integrate into existing industrial systems.

The Community Weighs In

Data engineers have taken notice. Some see RAP as a natural extension of open data formats to support interactive workloads. Others point out that as cloud object storage gets faster, the bottleneck shifts to query planning and metadata operations. RAP tackles that by precomputing indexes, so the heavy lifting is done upfront.

For industrial networking professionals, the takeaway is clear: the gap between analytical and operational data access is closing. You don't have to choose between a data lake for analysis and a database for real-time lookups. With patterns like RAP, you can have both on the same data, with less duplication and lower latency.

It's not a silver bullet, of course. You still need to think about index maintenance, query patterns, and storage costs. But if you're tired of copying data between systems just to answer a simple question, Spotify's approach is worth a look.