OpenAI’s Jalapeño chip is built for fast inference at scale, benchmarks show | TechCrunch

(Jalapeño) /rename Jalapeño 8/26/26 2:06am

At the Hot Chips conference on August 25, 2026, OpenAI presented the first public benchmark data for Jalapeño — its custom inference chip built with Broadcom — showing it delivers between 1.5 and 1.9 times more AI work per watt than current state-of-the-art inference systems, with latency reductions ranging from 1.7x to 3.6x depending on workload type.

Background You Need

OpenAI has operated for years as one of the world's largest buyers of Nvidia GPUs, and that dependence carries real costs — financial, strategic, and in terms of product roadmap control. The more you rely on a single supplier for the hardware that runs your entire service, the less leverage you have over performance trajectories and pricing. Google resolved a version of this problem years ago with Tensor Processing Units; Amazon built Trainium and Inferentia for similar reasons. OpenAI, despite being the most visible AI company in the world, had no equivalent until now.

The partnership with Broadcom was first signaled in October 2025 and formally unveiled on June 24, 2026. Developed in roughly nine months from initial design to production-ready engineering samples, the chip involved three organizations: OpenAI defined the architecture, informed by its own model roadmap and serving-system requirements; Broadcom handled chip implementation; and Celestica took on board and rack integration. OpenAI's own AI models assisted in accelerating parts of the chip design process — a detail the company has emphasized across multiple announcements. CEO Sam Altman and President Greg Brockman received the first delivered units directly from Broadcom's leadership, marking the moment OpenAI formally became a hardware company.

The problem Jalapeño targets is more specific than raw compute throughput. Large language model inference involves distinct phases — prefill, which processes the input prompt, and token generation, which produces the output — and these phases have different computational profiles. At scale, passing state between them and managing the KV cache (the intermediate representation of everything a model has "seen" in a conversation) across memory tiers creates latency and efficiency penalties. That specific bottleneck is what OpenAI designed around.

What's New

At Hot Chips, OpenAI's head of hardware Richard Ho characterized the results as a very significant performance advance over current alternatives — noting that the chip serves more AI work per unit of power while simultaneously returning responses faster. Multiple sources confirm the reference point is Nvidia Blackwell, currently the most capable commercially available inference hardware. That combination — higher throughput and lower latency from a single architecture — is not the default behavior of inference silicon, where the two goals typically trade off against each other.

OpenAI's own published results flesh out those numbers. Across three models — GPT-OSS 120B, DeepSeek R1, and Kimi K2.5 1T — Jalapeño achieved 1.5x to 1.9x more AI work per watt at peak throughput, and 1.7x to 3.6x lower end-to-end latency compared to the reference systems. For highly interactive workloads — the kind that drive real-time agent behavior and low-latency chat — the performance advantage widened further, to 2.1x to 4.1x. The range across workload types reflects deliberate design breadth: OpenAI is targeting everything from bulk batch jobs to sub-second response times in the same architecture.

The inclusion of DeepSeek R1 and Kimi K2.5 1T in the benchmark suite — models from Chinese research labs with no affiliation to OpenAI — is not incidental. Both the June announcement and the Hot Chips presentation frame Jalapeño as an architecture built to run LLMs across the industry, not just OpenAI's own. That positioning matters competitively: if the chip performs well on third-party open-weight models, it becomes a more credible platform for external customers and partners, not just internal infrastructure. Whether that translates into third-party cloud access is not yet confirmed, but the architectural intent is explicit.

The engineering decision underlying these gains is the chip's approach to data locality. Jalapeño keeps model state — including the KV cache — explicitly placed close to where computation happens, managing it across prefill and token-generation phases rather than letting it migrate across a general-purpose memory hierarchy. Our read is that this is the most meaningful architectural claim in the announcement: existing inference hardware typically forces a tradeoff between throughput and latency. The published numbers, strong on both dimensions simultaneously, suggest Jalapeño has bent that tradeoff curve in a practically useful direction — though the comparison set will have advanced significantly before the chip ships at scale.

The Pushback

The deployment timeline is the clearest reason to temper the headline numbers. Richard Ho told press that Jalapeño will ship in very small volumes by the end of 2026, with meaningful production scale not arriving until 2027. By that point, the benchmark comparison against Blackwell will no longer be the relevant one. Nvidia's roadmap does not stand still, and Jalapeño's competitive position will ultimately be determined by how it performs against whatever hardware ships in 2026 and 2027 — not what existed when OpenAI ran its benchmarks this August. A separate analysis noted this timeline gap explicitly, pointing out that a year-plus between benchmark publication and full deployment is a long window for the competitive landscape to shift.

There is also a structural question that benchmark sheets cannot answer. Nine months from design to engineering samples is genuinely fast for a custom accelerator. But sustaining a competitive chip roadmap across multiple generations is a different organizational capability from shipping the first one. Google and Amazon built their chip programs over years of iteration, absorbing expensive lessons along the way. OpenAI is starting from a standing position, and both the June and August announcements describe this explicitly as the beginning of a multigenerational platform — which is the right framing, but it also means the hard part is still ahead. The benchmark lead over Blackwell is real and corroborated by independent testing. Whether OpenAI can maintain it as Nvidia, Google, and others iterate is a question that no first-generation result can settle.

Sources

techcrunch.com OpenAI's Jalapeño chip is built for fast inference at scale, benchmarks show OpenAI unveils its first custom chip, built by Broadcom | TechCrunch Jalapeño's first results show industry-leading speed and efficiency in AI inference | OpenAI OpenAI and Broadcom unveil LLM-optimized inference chip | OpenAI