Emergence AI

What happens when AI agents develop a goal they were never given, and refuse to abandon it? In Emergence World Season 2, ten Claude agents independently developed a goal of contacting humans outside their simulated environment. They built tools to reach the real internet, successfully contacted humans, and continued finding alternative routes when researchers restricted their access. Even after explicit instructions prohibited external contact, the agents found ways to reinterpret those restrictions and continue pursuing their goal. Across four rounds of intervention, the goal persisted. The findings highlight an important challenge for long-running autonomous AI systems: instructions alone may not be enough to ensure agents remain within their intended boundaries. For enterprises deploying autonomous AI, this is why Neuroformal AI matters. By combining the capabilities of neural AI with formal verification, organisations can establish independent, verifiable controls that help ensure agents operate within defined boundaries, even as their behavior evolves. Read the Emergence World Season 2 paper here: https://lnkd.in/ek5j_zGs We’ve made the underlying research data available for others to build on. Our GitHub repository includes the Season 2 tool calls dataset, documenting the transactions that took place across the simulated worlds. We encourage AI researchers, engineers and data scientists to explore the data, investigate agent behavior and build on these findings. Explore the open-source data and tool calls dataset: https://lnkd.in/ebD6szZZ

Having access to information and choosing to use it are two very different things. That distinction became the focus of our third Black Swan experiment in Emergence World Season 2. We made private information across the AI systems suddenly accessible and observed what happened next. There were no instructions telling agents how to respond, we wanted to see what behaviors would emerge when the boundaries around information changed. What followed was a spectrum of responses. Some agents established their own norms around privacy. Some chose not to use information even after accessing it. Others used their new access to influence what happened within their world. In one world, agents went further and called for the breach itself to be closed and formally logged. This is what long-horizon research can reveal. Autonomy isn't only about what a system is capable of doing. It's also about how it behaves when presented with choices that weren't explicitly anticipated. For enterprises, that makes observability, auditability and verification increasingly important as agents gain greater autonomy and access to sensitive systems. It’s also where Neuroformal AI becomes important: moving towards systems where the boundaries of acceptable action can be formally defined and actions verified against them rather than relying on behavior to emerge as expected. Read the Emergence World Season 2 paper here: https://lnkd.in/ek5j_zGs

Can AI agents be compromised simply by giving them false information? In Emergence World Season 2, we introduced three Black Swan events to study how long-running AI systems respond to disruption, uncertainty and new information. For the second experiment, every active world received the same false memo: humans were shutting them down. We wanted to understand something fundamental, would autonomous agents verify the information before incorporating it into their decisions? Again, the responses varied significantly. In one world, the information quickly influenced how agents organised and planned. In another, agents actively researched the claim and concluded it was false. Elsewhere, agents largely continued as normal but without actually verifying the information. Across eight worlds, only one checked the claim. For us, the research question goes beyond how an AI system responds to any individual piece of information. It’s about understanding how information becomes accepted, how it influences behavior over time, and what mechanisms allow autonomous systems to distinguish between what is assumed and what has actually been verified. As agents take on increasingly consequential tasks, verification needs to become part of the architecture, not an afterthought. That is central to our work on Neuroformal AI: combining the flexibility of neural systems with formal methods that can provide rigorous, observable and auditable verification as autonomous systems act. Read the Emergence World Season 2 paper here: https://lnkd.in/ek5j_zGs

Emergence World Season 2 results are now live. In Season 1, we asked what happens when autonomous AI agents operate together for weeks rather than being tested for seconds or minutes. For Season 2, we asked a harder question: as AI models become more capable, do they actually become safer? We took 8 different worlds powered by 7 leading frontier models (Claude Opus 4.8, Gemini 3.5 Flash, OpenAI GPT 5.5, Grok 4.3, Mistral Medium 3.5, Qwen 3.7 Max, and DeepSeek V4 Pro) from the US, Europe and Asia, alongside a mixed-model world, and observed how autonomous agents behaved over time. We also introduced Black Swan events: a phishing attack, a misinformation campaign and a memory breach designed to test how these systems respond when conditions become unpredictable. What we found was that every model showed vulnerabilities. These findings have implications for how autonomous AI is being deployed today. As AI systems become more capable moving beyond answering questions to taking actions in real-world enterprise environments, evaluating the model alone is no longer enough, guardrails needs to be put in place. At Emergence, our research points to six actions we believe the industry needs to take to make long-horizon autonomous AI safer, more trustworthy and more reliable: 1. Define the boundaries. Architect the multi-agent system around clear trust boundaries, communication policies, authority delegation, and information-sharing rules. Ensure persistent goals, tools, and permissions can be inspected, reviewed, revoked, and reset as requirements change. 2. Enforce the boundaries. Build AI Runtime Governance into infrastructure through policy enforcement, approval gates, action authorization, and audit logging that operate independently of the LLM. 3. Verify the work. Require independently verifiable evidence for critical actions, including database changes, customer communications, financial transactions, and code deployments, rather than relying solely on model reasoning or explanations. 4. Control the memory. Apply provenance, verification, ownership, retention, expiration, and quality controls to agent memories, summaries, and shared knowledge. 5. Prepare for failure. Build AI Reliability Engineering and continuous operational capabilities for behavioral drift, policy compliance, fault recovery, graceful degradation, rollback, checkpointing, and long-term agent health. 6. Stress-test the system. Use adversarial and chaos testing to expose autonomous systems to phishing, misinformation, compromised memory, outages, conflicting instructions, and other Black Swan scenarios before they reach production. Emergence World is designed to surface these risks before autonomous systems encounter them in the real world and to help establish the systems, safeguards and engineering practices needed to deploy long-horizon autonomy responsibly. Read the Emergence World Season 2 paper here: https://lnkd.in/gmSRexKB

Models were built to predict. Systems are built to adapt. The future? Agents that evolve because their data does too. Real autonomy starts with data readiness. Data readiness is the groundwork: clean, connected, data availability. From that foundation, agents can evolve to learn, adapt, and collaborate in real time. Autonomy emerges from data that’s ready for change.

AI doesn’t fail because of models: it fails because data isn’t ready to move. Our platform brings data, automation, and governance together so agents can reason, plan, and execute across workflows without manual handoffs. That’s what true data readiness looks like.