FriendliAI

When does it actually make sense to move off pay-per-token to dedicated GPUs? 🤔 Pay-per-token is the easiest way to start building with LLMs — no GPUs to provision, and costs scale directly with usage. But as traffic grows, the economics shift. We modeled a representative GLM-5.1 workload (10K input / 500 output tokens per request, 50% cache hit rate) and compared the two pricing models: 🔵 Model APIs (pay-per-token): cost scales linearly with token volume 🔵 Dedicated Endpoints (reserved GPUs): fixed hourly cost, independent of throughput The two curves cross at a point. Below it, pay-per-token is cheaper because you only pay for what you use. Above it, dedicated capacity wins because you're utilizing GPUs more efficiently. In our example, the effective cost on a Dedicated Endpoint came out to ~$0.64 per million tokens — compared to $1.40 for the equivalent GLM-5.1 input price on Model APIs. There's no universal crossover point. It comes down to your tokens per request, concurrency, traffic patterns, and latency targets. The honest answer is to measure your workload, benchmark both, and decide on real numbers. FriendliAI makes migration straightforward. And because both run on an OpenAI-compatible API, switching is just a base URL and model ID change. Full breakdown in the comments below👇

You picked the AI model. But can your GPUs actually run it? Start with the minimum VRAM required to load the model: W = P × B Where: P = parameter count B = bytes per parameter at your target precision Real-world example: → DeepSeek-V4-Flash (284B at 4-bit): 142 GB → GLM-5.1 (754B at FP8): 754 GB → MiniMax-M2.5 (230B at 4-bit): 115 GB Those numbers load the model, but running it efficiently is another story. KV cache grows with concurrency and context length, and you'll also want headroom for system overhead. A common heuristic is reserving an additional 20% of weight memory, though high-concurrency or long-context workloads may require 30–40%+. → DeepSeek-V4-Flash: 142 GB + 28.4 GB = 170.4 GB → GLM-5.1: 754 GB + 150.8 GB = 904.8 GB → MiniMax-M2.5: 115 GB + 23.0 GB = 138.0 GB Benchmarks pick the model, and memory calculations help you choose the hardware. Want a deeper breakdown? We cover the methodology in our free Inference Performance Optimization Guide: https://lnkd.in/g6pYsDkz

One benchmark won't tell you which inference provider to pick. The Pareto front will. A single point—best TTFT, peak throughput—only describes one operating condition. Your workload doesn't run at one point. Vary batch size and concurrency and you get a whole curve: the Pareto front of latency vs. throughput. Each non-dominated point is the best throughput achievable at a given latency. That's the thing to compare. ❓ Why it matters: Providers can swap places along the curve. Provider A wins at the benchmark point; Provider B owns the rest of the frontier your traffic actually hits. 💡 Pick your point by workload: → chat / voice agents → minimize TTFT → batch / summarization → maximize TPS → search / RAG → somewhere in between Real question isn't "who wins the benchmark?" It's "who has the better frontier for my model, my SLA?" That's the question worth asking — and worth measuring. Full methodology in the comments below 👇

NVIDIA Nemotron 3 Ultra — 550B total, 55B active, 1M token context, Hybrid Transformer-Mamba MoE model is available Day 0 on FriendliAI’s frontier inference cloud 🙌 We’re built for the hardest part of agentic AI: orchestration. It reads your full context, breaks down complex goals, delegates across sub-agents, and synthesizes everything back at a speed that makes long-running agents viable in production. What FriendliAI brings to production-ready: ⚡ 2×-5× faster 💰 50–90% cost savings ⛑️ 99.99% uptime 🔒 Deploy on your own infra Deploy Nemotron 3 Ultra → https://lnkd.in/gXt3T_CK Find your Nemotron model → https://lnkd.in/g3Ttd9Di Link to blog in the comments 👇 #NVIDIA #Nemotron #AgenticAI

🪄 The trick to faster LLM inference? Predict multiple tokens ahead — then verify them in parallel. FriendliAI Dedicated Endpoints support draft-model speculative decoding, with draft models trained and automatically paired by FriendliAI. Single toggle at endpoint creation. No application code modifications. Why it matters: ⚡ 𝗙𝗮𝘀𝘁𝗲𝗿 𝗼𝘂𝘁𝗽𝘂𝘁. The target-model verifies multiple proposed tokens in a single parallel forward pass — at roughly the cost of a standard single-token decoding step — letting the same memory traffic produce multiple accepted tokens. 🎯 𝗤𝘂𝗮𝗹𝗶𝘁𝘆 𝗽𝗿𝗲𝘀𝗲𝗿𝘃𝗲𝗱. Verification follows the target model’s own next-token distribution. 🔧 𝗭𝗲𝗿𝗼 𝗼𝘃𝗲𝗿𝗵𝗲𝗮𝗱. No additional training, no extra models to manage, no code changes. This breaks the autoregressive bottleneck: one token, one forward pass, repeat — where sequential decoding iterations increasingly dominate inference latency as generations get longer, and each pass spends most of its time loading weights from GPU memory. Unlike N-gram speculative decoding, which only fires on literal token repetition, the draft-model generalizes beyond literal repetition — proposing accurate candidates even when the exact sequence hasn’t appeared before. Best suited for agentic pipelines, long-form generation, and code completion. Supported target models: Gemma-4-31b-it, Kimi-K2.6, Qwen3.6-27B, GLM-5.1, GLM-5, DeepSeek-V3.2, MiniMax-M2.5. Link to blog post in first comment 👇

What's so special about 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸 𝗩𝟰 by DeepSeek AI? In the few weeks since launch, 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸-𝗩𝟰-𝗣𝗿𝗼 and 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸-𝗩𝟰-𝗙𝗹𝗮𝘀𝗵 have surged to the top of the most-used open-weight models on OpenRouter. Both are now live on 𝗙𝗿𝗶𝗲𝗻𝗱𝗹𝗶 𝗗𝗲𝗱𝗶𝗰𝗮𝘁𝗲𝗱 𝗘𝗻𝗱𝗽𝗼𝗶𝗻𝘁𝘀 with a 1M-token context window by default. 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸-𝗩𝟰-𝗙𝗹𝗮𝘀𝗵 is the one redrawing the performance-efficiency frontier: 🧬 284B total parameters and 13B active parameters per token 📈 91.6 LiveCodeBench, 88.1 GPQA Diamond, 86.2 MMLU-Pro — within two points of 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸-𝗩𝟰-𝗣𝗿𝗼 on most reasoning evals 🧠 At launch, on par with Claude Sonnet 4.6 (max reasoning) on the Artificial Analysis Intelligence Index 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸-𝗩𝟰-𝗣𝗿𝗼 is the ceiling above it: 📚 At launch, the highest open-weight world-knowledge score on record — 57.9 on SimpleQA-Verified 🤖 Frontier-grade agentic coding: 90.2 Apex Shortlist, 89.8 IMOAnswerBench, 80.6 SWE Verified 📝 Long-context evals at full reach: 83.5 MRCR 1M, 62.0 CorpusQA 1M Pick 𝗙𝗹𝗮𝘀𝗵 for high-throughput agents or 𝗣𝗿𝗼 when the workload deserves a higher ceiling. 𝗗𝗲𝗽𝗹𝗼𝘆 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸-𝗩𝟰-𝗙𝗹𝗮𝘀𝗵 👉 https://lnkd.in/gnjKKFG2 𝗗𝗲𝗽𝗹𝗼𝘆 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸-𝗩𝟰-𝗣𝗿𝗼 👉 https://lnkd.in/gtj3DuHK 𝗥𝗲𝗮𝗱 𝗼𝘂𝗿 𝗯𝗹𝗼𝗴 𝘁𝗼 𝗹𝗲𝗮𝗿𝗻 𝗺𝗼𝗿𝗲 𝗮𝗯𝗼𝘂𝘁 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹𝘀 👉 https://lnkd.in/gAjyZGFd