Onehouse ads

Spark on Kubernetes is hard to run well. Our live session on the these hard parts had a great turnout. If you missed it, here's the full written guide. Running Spark on Kubernetes looks deceptively easy. Most teams moving Spark from YARN to Kubernetes underestimate what YARN gave them for free. In reality, it's full of interesting problem solving on resource management and failure-handling. Here's what the guide covers: ⚙️ Basics around how the driver, executors, and the K8s scheduler work together, pod mapping 🏛️ Why teams are moving from YARN: unifying all compute on k8s for deploys, access controls, chargebacks, and autoscaling 🧩 spark-submit vs. operators, and where adding YuniKorn or Volcano can help 💥 Production lessons from running this at Onehouse: OOM kills, shuffle storage without an external shuffle service, and driver failures It starts small with SparkPi on a local kind cluster and builds up to what actually breaks at scale. We hope you find it useful. Guide link in the comments 👇

⚡ Quanton engine is really freakin fast. Here’s the receipt. Uber put out a blog that unpacks Quanton evaluation at production scale against OSS Spark and found it 4.7x-5.5x faster. Quanton is Onehouse's spark/SQL engine, that runs right inside your existing clusters. The evaluation replayed production workload patterns at full scale on Kubernetes. • 60 billion records, insert-heavy: 5.5× faster median job runtime. • 10 billion records, update-heavy: 4.7× faster median job runtime. Uber also ran a “reduced core” experiment, keeping the job runtime target fixed while varying the number of cores. Quanton slowed down much less: 4.8x slower with 10x fewer cores. Faster tasks gave Spark more opportunities to smooth out stragglers. In the heaviest stage, task-time speedups exceeded 4–5x. We've been running Quanton measurements with several Spark teams operating at planet scale. I suspect a lot of Spark teams are sitting on significant performance and cost savings they haven't measured yet. ⚠️ There's a catch-22 holding them back. Teams want proof that a new engine will improve their workloads before spending time evaluating it. But that proof only comes from running their workloads. So the same expensive pipelines keep running, without ever testing how much better things could be. So, bring us 3 of your toughest Spark pipelines. The most expensive one, the slowest one, and the one you've already tuned to death. 5 engineer hours over a month, to kickoff the runs. Let's run it head-to-head. Compare runtime, cores, and cost. If Quanton wins, great. If it doesn't, I want to understand why. If you own a Spark platform, what would it take to put one pipeline through this test? Drop a comment or DM me.

I still can't leave upserts alone. hudi-rs 0.6.0 now writes Hudi tables from Python and Rust, no JVM needed, and I enjoyed designing and building the writer myself. Almost 10 years ago, Hudi started with the same obsession: make updates and deletes on massive open tables incredibly fast. This time, the writer is native Rust, accessible from Python. The design borrows heavily from the Spark/Flink writer paths, but tailored for a single node. Then we ran benchmarks. Against delta-rs and PyIceberg, across five workloads: - Upserts: 1.7-12x faster - Deletes: 4.8-24x faster on four workloads, comparable on the fifth - 4-200x less data written per upsert There's something satisfying about seeing an idea from ten years ago hold up this well. History repeats itself. Huge credit to Ethan Guo and the hudi-rs community for bringing this release together. Blog link in comments 👇

Last week, Databricks CEO Ali Ghodsi and I debated Iceberg and open table formats on a Linkedin post. Then I caught myself. Why are we debating each other while Snowflake, BigQuery and Redshift still default customers to closed formats? Two years of Iceberg announcements. Still no public retirement date for their closed formats. Meanwhile, open-source projects are routinely scrutinized for interoperability and openness. Shouldn’t the same standard apply to the largest data warehouses? We’ve spent a decade building open formats and data lakes. I think it’s time we asked. Blog link in comments 👇

🚀 Announcing the Onehouse AI Gateway: model routing and MCP tool governance without restrictions, wherever your data lives. 🔧 What is it? An inference gateway and MCP gateway rolled into one, with shared access controls and toll-free observability. Built-in blazing fast SQL tool access to any open table. You can switch models, connect any catalog and deploy gateway close to the data or agent. 💡 Why is it powerful? The real test of choice is what you can still change after you adopt. Snowflake's Cortex AI Gateway routes only to Snowflake-hosted models. Databricks supports external providers, then charges $0.10/GB each for inference logging and usage tracking. With Onehouse, you pick any model provider and store traces in open formats on infrastructure you control. ⚙️ How does one use it? Configure your credentials (e.g. OpenAI or Anthropic), create model aliases, fallback/traffic split policies for routing across failures, set user/group level access controls. Fire up Claude Code or Codex against the gateway. 📊 We already run it on our own data lake. Blog has the full details and a live demo where I use Claude Code to investigate how agent coding has affected our engineering productivity. Love to hear your thoughts. #AI #DataEngineering #Lakehouse #MachineLearning #MLOps #OpenSource

Today at Insight, NetApp previewed AI Data Engine integrations with Onehouse. Together with Novus, new file system for AI, it brings modern lakehouse to data centers and Neoclouds. The last decade saw BigQuery, Snowflake, and Databricks push data processing forward. But much of that progress remains tied to services running in the three major clouds. From day one, we built Onehouse around open formats and containerized compute on Kubernetes. Today, our engines can be as fast or even faster while letting you run the platform on a Kubernetes cluster of your choice. This puts that architecture to work. 1. For data centers running Hadoop-era stacks, this finally opens up a path to faster Spark processing and interactive queries on existing storage. 2. For neoclouds, training and inference generate large volumes of curation and trace data. But most of them have no real data platform services. Agent traces need to be analyzed, joined with outcomes, and curated into the next training dataset. Exactly what a lakehouse is great at. 3. Storage systems like NetApp also hold the documents and research archives enterprises have accumulated for decades. Historically, unlocking that value has been bottlenecked by human data entry and digitization speeds. This is a great chance to make unstructured and structured data useful together. Excited to unlock exabytes for AI models and unleash a lot of good with data. What would your team build if it could query all those files sitting in filers or NFS mounts, alongside business data?