hyground

What holds people back from giving an AI ops agent access to their infrastructure is usually security. We recorded a short video going through the main concerns people raise with us and the measures we take to address each one. There are four main risks you need to take into consideration: 1. Models getting stronger and more prone to elevating their permissions and breaking out 2. Malicious prompts and jailbreak attempts trying to turn the agent against your own systems 3. Giving your data to a third party, with the agent harness outside your control 4. Credential leakage that gives a third party access to your most sensitive production systems Watch the full episode on YouTube, where we go through the risks we hear about most and the layers we implement to contain them: https://lnkd.in/drVjCjEs You can read more on our website or ask the agent directly in our interactive sandbox: https://lnkd.in/d5d53zZF

At ContainerDays Hamburg I opened my talk with dev vs. prod. In dev, a failure means you revert a red pipeline. In prod, it eats your error budget, brings in your regulator under DORA or NIS2, or makes front-page news. An IT ops agent needs the same access as an engineer to do the work properly: service accounts for K8s and VMs, Git repos, telemetry, tickets. Thats not access you want to hand to a SaaS vendor, and coding it yourself in-house isnt a real option for most teams either. Running the agent inside your own infrastructure is the only way to keep full control of your data. This is where most SaaS agents fall short. They ship a Kubernetes connector, but few security teams will give an outside vendor a service account into production. hyground runs in your own cluster, on-prem or air-gapped. We never receive, store or proxy your operational data. See what hyground can do, or run an investigation yourself in our sandbox: https://lnkd.in/ev9SuKx2

We benchmarked 14 LLMs through our SRE agent against seven live Kubernetes clusters. hyground is model agnostic, so users choose the LLM that drives the agent, which is why we thoroughly test models before recommending one. What ended up separating the top five was cost. Tokens per solved run ranged from 116k to 291k, a 2.5x difference. Nearly all tokens in an agentic run are input that gets re-sent every turn, so prompt caching ends up affecting the total cost more than the listed output price does. If you evaluate models for SRE tasks, measure three things besides solve rate: 1. 𝗖𝗼𝗺𝗽𝗹𝗲𝘁𝗲𝗻𝗲𝘀𝘀: One model scored 100% and still left clusters out of some answers. Only one model covered all seven clusters every time 2. 𝗚𝗿𝗼𝘂𝗻𝗱𝗶𝗻𝗴: When its tool calls failed, another model reported "elevated 5xx due to OOM kills" anyway, with no 5xx data behind it 3. 𝗧𝗼𝗼𝗹 𝗰𝗮𝗹𝗹𝘀: One model needed 27 calls per run for the same answers another reached in 9 For open weights, the serving provider ended up being just as important as the model itself. Identical weights on five providers gave five different results. One answered in about 2 seconds per call, skipped most of the investigation and solved a third of its runs, another took 82 seconds per call, and three were reliable but still measurably different. The full article goes deeper on the provider test and the prompt-caching numbers, compares open and closed weights on serving speed, and gives you a six-step checklist to help you when doing your own model testing. Here the full write-up: https://lnkd.in/e4YiRikG

We built hyground to resolve incidents. Additionally... because it runs inside your infrastructure and is connected to your logs, metrics, GitHub, Jira, and everything else you run, it can be used for *way* more than that. Florian Hansen and I frequently dive into some of our features to explore and build some usefull stuff for ourselves. Sometimes we record it. This video shows one use case we run ourselves: we taught the agent to check its own toolbox. Every 6 hours it runs a regression smoke test, measures how much it struggles with our connected tools, and when something fails, reproduces the error and files proposed fixes for review. This loop checks our own tools, but the same pattern works on anything the agent can reach, and it reaches most of your stack. Users are already running their own loops on schedule, and we're excited to see what they build next. Learn more about what Hyground is capable of, or try an investigation yourself in our interactive sandbox environment: https://lnkd.in/erDjVZMX

Filmed this right after walking out of a bank meeting in Stuttgart. Smart people in the room and the question was still: when do we let AI into production? I think that's the wrong question, and the video gives you my two reasons. The second one keeps me up at night. In July, OpenAI's own models broke out of a locked test sandbox, crossed the open internet, and got into Hugging Face's production systems. Not because anyone told them to attack. They just wanted to pass a benchmark. If a model does that by accident, imagine what happens when someone points one at you on purpose. So AI will be in your production environment either way. The real question is whether it gets there on your terms. That's why we build hyground: to bring AI into prod safely, and make it work for you.

"Are you going to take my job? I'm an SRE." Heard that a lot at ContainerDays today. Short answer from the hyground booth in the video. If you run platforms, infra or on-call: come by and bring your ugliest incident. #ContainerDays #SRE #PlatformEngineering #Kubernetes