DatologyAI

The research teams from Thomson Reuters and DatologyAI got together to walk through how the Thomson-1 model was built. They get into the data, the mid-training, and what it takes to build a frontier model on open weights. If you're building your own model or want to own your intelligence, it's worth the watch. Congrats to Thomson Reuters on the project. Thanks to the presenters: Jonathan Richard Schwarz, Yejin Bang, Shengzhuang Chen, Ari Morcos, Josh Wills, Amro Abbas, and Katherine Mentzer for sharing your insights. Watch the full conversation: https://lnkd.in/gpSVaJpv

Every AI model is only as good as the data it learns from, and the industry is running low on the high-quality text that made today's models great. Researchers call this the data wall. One obvious response is to have models generate more of their own training data. The trouble is that when a model learns from the potentially dubious outputs of other models, over and over, it drifts away from reality and degrades, a problem known as model collapse. The approach that actually works, and that's now used in nearly every serious model, takes a different path. Rather than inventing new data, the team at DatologyAI pioneered a technique at scale to transform real documents into forms a model can learn from more efficiently. Because the knowledge stays grounded in real material, the model keeps getting better rather than collapsing. It's a lot like taking a handful of good ingredients and making several different and amazing dishes from them. In our latest blog, a new type for us, we have created an explainer for anyone who wants to understand synthetic data without reading a research paper. It covers what synthetic data really is, why the common "it's just fake data" worry misses the point, and where it fits across the stages of model training. Link to the blog in the comments below.

𝗛𝗼𝘄 𝘁𝗼 𝗴𝗲𝘁 𝘁𝗵𝗲 𝗰𝗼𝗿𝗿𝗲𝗰𝘁 𝗮𝗻𝘀𝘄𝗲𝗿 𝗳𝗼𝗿 𝟯𝟱𝘅 𝗹𝗲𝘀𝘀 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗰𝗼𝗺𝗽𝘂𝘁𝗲. Most of the work on making models cheaper to run is really about making the model smaller, like distillation, quantization, pruning, and speculative decoding. They all work, and they're all chasing the same number: the cost of a single token. But the token cost is only half the bill. The other half is how many tokens the model burns to get to an answer, and almost nobody optimizes that. We tend to treat answer length as just how a model responds, not as something you can change. It turns out you can change the response, and it makes a huge difference. Our new paper, Brevity is the Soul of Inference Efficiency, shows that length comes from the data a model is trained on, so you can curate for it. Feed it pretraining data curated for brevity and correctness, and your model will learn to answer in far fewer tokens. Here's what that looks like on a 4B vision-language model: The model reaches a correct answer with 35× less compute than a same-size, same-task baseline and stays within ~1 pp of that baseline's accuracy. The whole gap is about length. Our model answers in about 30 tokens; the baseline takes around 1,284. Self-hosted, that's roughly $1.34 versus $47 per million correct answers. And no, shorter doesn't mean worse. When you match the output length, the curated model comes out +17.55pp more accurate. You don't pay with quality by having a shorter answer. So it's not just cost per token we should be looking at. The number you were not paying attention to that is blowing up your bill is the cost per correct answer, and you can do something about it.

Rethinking the Future of AI: It’s All About the Data Hear Ari Morcos, CEO of DatologyAI, on the Latent Space podcast to discuss the importance of data in the current industry. Ari describes how “models are what they eat”—and that careful curation: filtering, rebalancing, sequencing, and generating synthetic data is the key to training smaller models to better performance, all with less compute. His own journey, from focusing on model architectures to realizing the outsized impact of data quality, highlights how curation breaks through the diminishing returns of naive scaling. At DatologyAI, the mission is clear: automate this complex process to make world-class curated data accessible to everyone, ushering in a new era where data efficiency, not brute force scale, drives AI progress. Interested in joining our team and solving the world’s most interesting data problems? Apply here: https://lnkd.in/gz-unQ48 Watch the podcast here: https://lnkd.in/gK6c_dRu

A lot of companies talk about “doing” synthetic data, but what does it actually take to generate high-quality synthetic pretraining data at the trillion-token scale, necessary for training foundation models? The DatologyAI team answers this question with BeyondWeb, a synthetic data generation system that outperforms all publicly available methods for synthetic pretraining data generation. BeyondWeb establishes a new Pareto frontier for the trade-off between model size and quality in language model pretraining. It delivers superior performance at every scale, with our 3B model surpassing the results of most 8B-scale baselines, using the same token budget. This new frontier reshapes the economics and accessibility of high-performance AI development, enabling organizations to achieve state-of-the-art results with computational budgets that were previously insufficient for competitive model development. Read more about the impact of BeyondWeb—and what it means for the future of the AI industry—here: https://lnkd.in/g_6-3QBQ

𝐅𝐫𝐨𝐧𝐭𝐢𝐞𝐫-𝐜𝐥𝐚𝐬𝐬 𝐯𝐢𝐬𝐢𝐨𝐧-𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐦𝐨𝐝𝐞𝐥 𝐚𝐜𝐜𝐮𝐫𝐚𝐜𝐲. 𝐀𝐭 𝐮𝐩 𝐭𝐨 ~150× 𝐥𝐞𝐬𝐬 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐜𝐨𝐦𝐩𝐮𝐭𝐞. 𝐀𝐧𝐝 𝐮𝐩 𝐭𝐨 3.3× 𝐥𝐨𝐰𝐞𝐫 𝐢𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 𝐜𝐨𝐬𝐭. Without bigger clusters. Without elaborate post-training. Just curated, high-quality training data. That's what our latest research, 20/20 Vision-Language Models, delivers. And the implications for any enterprise building multimodal AI are significant. Holding architecture and compute fixed and varying only the training data, our 2B model lands within 1.8 percentage points of Qwen3-VL-2B at ~87× less training compute. Our 4B matches Qwen3.5-2B at 2.4× lower inference cost. For companies building multimodal AI, that translates into: - 𝐁𝐞𝐭𝐭𝐞𝐫 𝐦𝐨𝐝𝐞𝐥𝐬 𝐟𝐫𝐨𝐦 𝐬𝐦𝐚𝐫𝐭𝐞𝐫 𝐝𝐚𝐭𝐚, 𝐧𝐨𝐭 𝐛𝐢𝐠𝐠𝐞𝐫 𝐜𝐥𝐮𝐬𝐭𝐞𝐫𝐬. Frontier-class accuracy without committing to hyperscaler-scale pretraining budgets. - 𝐋𝐨𝐰𝐞𝐫 𝐫𝐢𝐬𝐤 𝐨𝐧 𝐞𝐯𝐞𝐫𝐲 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐫𝐮𝐧. Per-capability seed variance drops by ~67%, making the recipe robust to the seed lottery that plagues large training runs. - 𝐋𝐨𝐰𝐞𝐫 𝐜𝐨𝐬𝐭 𝐭𝐨 𝐬𝐞𝐫𝐯𝐞, 𝐟𝐨𝐫𝐞𝐯𝐞𝐫. Up to 3.3× lower inference FLOPs than comparable models, with Pareto-dominance on response cost at every scale tested. This is capable multimodal AI that can run on edge devices, not just in a data center. 𝐇𝐨𝐰 𝐲𝐨𝐮 𝐞𝐧𝐠𝐢𝐧𝐞𝐞𝐫 𝐲𝐨𝐮𝐫 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐝𝐚𝐭𝐚 𝐢𝐬 𝐭𝐡𝐞 𝐦𝐨𝐚𝐭. Companies with high-quality proprietary data and the discipline to curate and engineer it well don't need to match hyperscaler budgets dollar-for-dollar to achieve amazing performance. The standard approach to competitive multimodal AI has been to throw more compute at it. The new approach is to treat training data as the strategic lever that it is. Read the full write-up: https://lnkd.in/gwn6w-Mp