Rootly

Generic coding benchmarks are table stakes. We validate frontier models on real SRE work before we believe the hype. And this time Gemini 3 Pro was a standout exception. After running it through SRE-skills-bench (our open benchmark for SRE tasks), Gemini 3 Pro came out... → ~4% more accurate on average than the next best frontier model → +13% on AWS S3 security configuration → +8% on IAM configuration tasks That’s a meaningful jump at a moment when model performance is starting to converge. Tradeoff worth noting: cost. Gemini 3 Pro is roughly ~20% higher per million tokens than GPT-5, which may or may not matter depending on your volume. We’ve published the full methodology and results including our work with Groq OpenBench and the SRE-specific task suite. Everything is reproducible. AI for reliability isn’t about vibes. It’s about whether a model can reason with operational context under pressure. Full benchmark + results in the comments.

Rootly is live at re:Invent! Swing by Booth 1530, right in front of Atlassian, to meet the team and pick up swag, invites to our exclusive events, and even access to our private van for rides across Vegas.

LLMs broke the SRE runbook. Now what? At SREcon, we spent 90 minutes discussing how AI is reshaping reliability engineering, what's working, what's breaking, and what's still pure chaos 😬 Here's my summary of the key takeaways for The New Stack 👇 Featuring Alejandro Saucedo, Charity Majors, Jay Lees, Maria Vechtomova, Niall Murphy, and Todd Underwood 🔥 https://lnkd.in/g6kmaUij

OpenAI just lost its SRE crown... ...To Gemini 3 Pro 👑 We ran it on SRE-skills-bench and found that it beats GPT-5.1 on every test. Over the last 6 months, OpenAI models have been leading for SRE-type tasks. Google DeepMind just broke that winning streak. A 4% average accuracy jump is a big deal at a time when model performance is starting to plateau. If you want to dig deeper or run your own benchmarks, our blog post, open-source dataset, and evaluation code make everything fully reproducible. Links in the comments.

GPT 5.1 just dropped, and it promises it can perform reasoning tasks with fewer tokens. So we put it to work on SRE-centric tasks inside Rootly AI Labs. Laurence Liang breaks down how the model performs across low, medium, and high reasoning intensity, plus what that means teams as they evaluate whether they should upgrade to this version.

The Rootly team is heading to AWS re:Invent, and we’re bringing two Happy Hours designed for engineering leaders to connect, trade ideas, and unwind in great venues. Monday Dec 1: Happy Hour on a cozy terrace with Arize AI, AuthZed, Cloudflare, Infisical, and PlanetScale. Tuesday Dec 2: Happy Hour at a bar and club with Baseten and Chronosphere. RSVPs linked below.