Feldera

🔄 Incrementally Incremental Computation Incremental View Maintenance has always felt a bit sci-fi to me: a record changes in a SQL table, and the system instantly ripples that update through hundreds of SQL operators, refreshing every affected view. But what if not just the data could change incrementally… but the queries themselves? Traditionally, modifying queries meant throwing away all prior computation and starting from scratch. Not anymore. Feldera now supports incremental query modifications. This is an incredibly powerful capability. You can make arbitrary changes to your program — add, delete, or modify views — even if other views depend on them. Feldera will: 🔍 Identify all affected views ♻️ Discard any state invalidated by the modification, while preserving all other state 🧮 Use existing state in the system to compute new and modified views. The result is that many pipeline reconfigurations can be performed in a fraction of the time it would take to backfill the pipeline from scratch. One of our customers described this as "incrementally incremental computation." What a nice way to summarize a complex feature in three words! Learn more here: https://lnkd.in/gNSZY2j8

🥉🥈🥇 Toward Real-Time Medallion Architecture One of the biggest advantages of open table formats like Delta Lake and Apache Iceberg is that you can query the same data with multiple engines. No vendor lock-in, just pick the best tool for each job. Case in point: We’re working with a customer who built their Medallion architecture using Apache Spark and Delta Lake. But as their business evolved, so did user expectations—especially around latency. Spark’s batch processing couldn’t keep up with the demand for fresh insights. The solution? We replaced periodic Spark jobs with the Feldera Incremental View Maintenance (IVM) engine. Feldera picks up changes to lower-tier tables and updates higher-tier tables in real-time, reducing the end-to-end latency of the pipeline from hours to minutes. Prior to open table formats, in the world of legacy databases, such a migration would require completely re-architecting the entire data stack in order to use a new query engine. But thanks to the use of Delta Lake, the transition was completely seamless to the rest of the company: ✅ The data hasn't moved and can be read using Spark, Trino, etc., just like before the migration. ✅ Same ingestion pipeline from Kafka to the Bronze layer. ✅ Existing BI dashboards reading the Gold layer stayed untouched—only now they show near-real-time data! Want to see how it works? Join Gerd Zellweger's webinar on replacing batch analytics with Feldera: https://lnkd.in/eFDqXb5Q

🧪 If You Cannot Measure It, It Doesn’t Exist! That’s why we built Feldera’s new latency measurement framework. It gives you precise visibility into: ✅ End-to-end pipeline latency ✅ Internal processing latency of our incremental query engine With this new API, you can see how Feldera reduces your time-to-insight from hours to seconds in your Grafana dashboard. 👉 Dive into the details: https://lnkd.in/gw8HRVYm

🤫 The Dirty Secret of Incremental View Maintenance: You Still Need Batch 🏆 Incremental View Maintenance sounds like the Holy Grail: update your complex analytics instantly as data streams in. 🛑 The problem? It breaks down the moment you leave the demo stage: every real system needs to backfill massive amounts of historical data before handling real-time updates, and most #IVM engines choke when asked to do it. That’s why an IVM engine has to be a batch engine first! 🛠️ Efficient backfill is one of the biggest hurdles in building a production-ready IVM engine. Over the past few months, we’ve been working on a solution, and we’re about to announce it. 💡 In the meantime, I wrote a blog that explains what exactly makes backfill tricky. Using a simple example, I show why even a powerful IVM system like Feldera requires special care to make backfill efficient: https://lnkd.in/gbHNJWT4.

"This is the true power of incremental compute. By only needing compute resources proportional to the size of the change, instead of the size of the whole dataset, businesses can dramatically slash compute spend for their analytics." 👇 https://lnkd.in/e7_ZJSAc

🚨 SQL Compatibility Nightmare? You’re Not Alone. Did you know that after nearly 40 years, every major database still speaks its own SQL dialect? From how NULLs are sorted to even the semantics of basic operations like "+", migrating or federating queries is a minefield. For data teams, this means: ** Painful migrations and vendor lock-in ** Hours wasted rewriting SQL for each system ** Surprises when "compatible" queries break or give the wrong result At Feldera, we faced this head-on. Instead of picking a side, we built on Apache Calcite, a flexible, open-source SQL engine. This way, we can: ** Support arbitrarily complex SQL ** Rapidly add features and extensions customers actually need (think: ASOF joins, Lateral Column Aliases, recursive queries) ** Deliver sub-second, incremental analytics with minimal code changes 💡 We are seeing data engineering teams port hundreds of thousands of lines of Spark SQL to Feldera, cutting down time-to-insight from hours to seconds, even for pipelines with hundreds of joins. What’s the weirdest SQL dialect portability issue you’ve run into? Let me know below. https://lnkd.in/g7pVres2 If you want to see true incremental SQL in action, join us on Slack or book a demo to learn more! #SQL #DataEngineering #RealTimeAnalytics #Feldera