This briefing is based on vendor announcements, official blog posts, and press reports — we did not run these benchmarks ourselves.
OpenAI released the first benchmark results for its custom Jalapeño inference chip, and Google DeepMind piloted the world’s first double-blind AI evaluation. Both stories share one thread: every headline AI number comes with questions — who ran the test, what was compared, and can the score be trusted? We covered the lead-up to these announcements in yesterday’s briefing, and today we break down what each means for your tool choices.
OpenAI’s Jalapeño chip posts its first benchmark results
OpenAI’s first custom AI inference chip, co-developed with Broadcom, posted its first benchmark results at Hot Chips 2026 and on OpenAI’s engineering blog Tuesday. The numbers show big efficiency and speed gains over Nvidia’s previous-generation Blackwell systems — but they come with caveats you should understand before reading too much into them.
What happened and why it matters
Jalapeño is OpenAI’s first custom AI inference chip, co-developed with Broadcom and announced in October 2025. The first results were published August 25, 2026 on the OpenAI engineering blog and covered by TechCrunch. The chip matters because it’s designed specifically for serving AI models — the “inference” work that happens when you type a prompt into ChatGPT — rather than training them. If the numbers hold up, it could mean cheaper AI for everyone.
What was actually tested
The results were run on InferenceX, a public serving benchmark from SemiAnalysis. SemiAnalysis engineers verified the runs in person at OpenAI’s lab, but the numbers were generated by OpenAI — vendor self-reported and not independently reproduced, per SemiAnalysis InferenceX and TechCrunch.
Three open-weight models were tested: GPT-OSS 120B, DeepSeek R1 670B (MXFP4), and Kimi K2.5 1T (MXFP4). They were compared against Nvidia’s GB200 (1,200 W rated) and GB300 (1,400 W rated) Blackwell systems, per the OpenAI engineering blog.
Across all three models, Jalapeño showed 1.5-1.9x more AI work per watt at peak throughput, 1.7-3.6x lower end-to-end latency, and 2.1-4.1x higher performance on interactive low-latency workloads, per the OpenAI engineering blog.
Specific numbers: GPT-OSS 120B hit ~1.9x peak throughput (85,448 vs 44,960 mixed tokens/sec/kW) with latency of 1.03 s vs 1.80 s; DeepSeek R1 670B hit ~1.7x (19,641 vs 11,781) with latency of 1.65 s vs 5.99 s; Kimi K2.5 1T hit ~1.5x (18,195 vs 11,862) with latency of 1.56 s vs 5.31 s, all per the OpenAI engineering blog.
The chip is rated at 700 W but measured sustained at or below 550 W on tested workloads, built on TSMC N3P, with 13.4 petaFLOPs MXFP4 and 216 GiB HBM4 at 15.4 TB/s, per Tom’s Hardware and TechTimes.
There are important caveats. The comparison only covers Nvidia Blackwell — not Nvidia’s newer Vera Rubin systems, which per SemiAnalysis are roughly tied with Jalapeño on total cost of ownership per token. The comparison is also asymmetric: Vera Rubin used multi-token prediction while the Jalapeño InferenceX runs used single-token prediction. Additionally, SemiAnalysis’ AgentX agentic benchmark suite has not yet been run on Jalapeño. These details come from Tom’s Hardware and TechTimes.
Will AI chatbots get cheaper?
Probably, but not immediately. The chip isn’t inside ChatGPT yet — deployment starts in very small volumes at the end of 2026 and grows in 2027, per TechCrunch. Richard Ho, OpenAI’s head of hardware, called the results “a very, very significant performance advance over state of the art,” and Gen 2 is already “deep in development.”
The bottom line for beginners
Don’t change which tool you use because of this news — the chip isn’t inside ChatGPT yet. What it means is a price and speed trend to watch over the next year. Treat the numbers as vendor self-reported: OpenAI generated them, SemiAnalysis verified runs in person, and the comparison only covers Nvidia’s previous Blackwell generation, not Vera Rubin. If you’re comparing AI tools today, our Comparison Database breaks down how they score across Ease, Features, Performance, Docs, and Support.
FAQ
Is the Jalapeño chip already inside ChatGPT?
No. Deployment is very small volumes at the end of 2026, with more significant volume in 2027, per TechCrunch. Today’s ChatGPT runs on existing infrastructure, so this news doesn’t change which tool you should use right now.
Does Jalapeño beat Nvidia’s newest chips?
It beats Nvidia’s previous-generation GB200 and GB300 Blackwell systems on tested metrics — 1.5-1.9x more work per watt, per the OpenAI engineering blog. Against Nvidia’s newer Vera Rubin, SemiAnalysis says the systems are roughly tied on total cost of ownership per token, per TechTimes. The comparison is also asymmetric — Vera Rubin used multi-token prediction, Jalapeño didn’t.
Are the Jalapeño benchmark numbers trustworthy?
They’re vendor self-reported, not independently reproduced — OpenAI generated the numbers, and SemiAnalysis engineers verified the runs in person, per TechCrunch. The comparison covers only Nvidia Blackwell, not Vera Rubin, and the AgentX agentic benchmark suite hasn’t been run yet. Treat them as promising but incomplete.
DeepMind pilots the first double-blind AI evaluation
Google DeepMind published the world’s first double-blind evaluation of a proprietary frontier-class AI model. The setup lets an external evaluator test a model without seeing its weights, while the model owner can’t see the test prompts. It’s a big step toward trustworthy AI benchmarks.
What happened and why it matters
Published August 27, 2026 by William Isaac, Sol Messing, and Kristian Lum on the DeepMind blog, this is the first double-blind evaluation of a proprietary frontier-class AI model. The external evaluator cannot see the model’s weights, and Google cannot see the evaluator’s test prompts — cryptographic safeguards mean neither side hands over secrets. This matters because benchmark contamination is a documented problem: Xu et al. (2024) found leakage evidence in roughly half of 31 models tested, and Singh et al. (2025) documented one frontier lab testing 27 private model variants on Chatbot Arena and publishing only the top scorer.
How the secrecy trick works
The evaluation runs in a hardware-encrypted NVIDIA H100 secure enclave on Google Cloud Confidential Space, orchestrated with OpenMined’s PySyft. Data is encrypted in RAM at the hardware level, with remote attestation and an ephemeral enclave that leaves zero state remanence. Compute overhead is under 5%, per the DeepMind blog and technical report.
The model evaluated was Gemini 2.5 Flash Lite, a proprietary model. Two evaluations ran: private prompts from the MLCommons AILuminate corpus with AVERI and MLCommons, and private harmful-content elicitation prompts in Singapore’s context with the Singapore AI Safety Institute. Partners included OpenMined and MLCommons, per the DeepMind blog.
The pilot published no model scores — no safety rating, no pass/fail, no verdict on Gemini 2.5 Flash Lite. It’s a proof of the method, not a result. This builds on a 2024 OpenMined proof-of-concept with UK AISI and Anthropic that validated the architecture at small scale; this pilot is the first to protect two organizations’ proprietary assets simultaneously, per the technical report.
Can you trust AI benchmark scores?
Not always. Benchmark contamination is documented — models trained on test data inflate their scores, and some labs cherry-pick results. The DeepMind pilot is a major step toward fixing this, but it’s a method demonstration, not a verdict on any model. For now, the most trustworthy signals are independent third-party checks.
The bottom line for beginners
When you see a benchmark number, ask who ran the test before trusting it. Prefer models with independent third-party checks — community blind-vote arenas like Arena and independent testers like Artificial Analysis — and expect double-blind sealed-enclave evals to become the gold standard over the next couple of years. Our Comparison Database can help you weigh these factors side by side.
FAQ
What does double-blind mean for AI evaluation?
It means neither side sees the other’s secrets: the external evaluator can’t see the model’s weights, and the model owner can’t see the test prompts. Cryptographic safeguards in a hardware-encrypted enclave enforce this, per the DeepMind blog. It prevents both benchmark contamination and cherry-picking.
Which AI benchmarks can I trust today?
The most trustworthy options are independent third-party checks: community blind-vote arenas like Arena, independent testers like Artificial Analysis, and OpenRouter model rankings. The DeepMind double-blind pilot published no scores — it’s a method proof, not a result, per the technical report. Expect double-blind evals to become standard in the next couple of years.
Still choosing an AI tool?
Both stories point to one lesson: treat benchmark numbers with skepticism. The chip news is a future price trend, not a tool-changer yet; the DeepMind pilot is a trust framework, not a model verdict. Compare tools side-by-side in the Comparison Database. This story was produced by our automated pipeline — track what’s next at /cron-pipeline/ or see our roadmap.
Back to all posts
