Understanding JetStream 3: A Deep Dive into Modern Web Benchmarking

JetStream 3 is a major update to the cross-browser benchmark suite developed collaboratively by Google, Mozilla, and WebKit. This revamp addresses the evolving nature of web performance, especially in WebAssembly and large-scale applications. Below, we explore key questions about the changes, challenges, and innovations behind JetStream 3.

What is JetStream 3 and why was it created?

JetStream 3 is a next-generation benchmark suite designed to measure browser performance more accurately for modern web applications. It was built to replace JetStream 2, which had become outdated as web technologies evolved. The suite reflects a fundamental shift in how performance is evaluated, particularly for WebAssembly (Wasm) workloads. The goal is to ensure benchmarks stay relevant to real-world usage, such as Wasm in image decoders, UI frameworks, and libraries. By updating the benchmarks, engine developers can focus on optimizations that matter most to users today.

Why can't we just keep using existing benchmarks like JetStream 2?

Benchmarks naturally become outdated as new best practices emerge in web development. Over time, the most accessible optimizations get addressed, leading to diminishing returns and potentially narrowing the focus to specific test cases rather than general performance. JetStream 2, for example, was designed when WebAssembly was still maturing, and its measurement approach no longer reflects modern usage. Additionally, browser engines had optimized certain paths so well that benchmarks produced infinite scores (see next question), making the results meaningless. A refresh was needed to maintain relevance and drive meaningful improvements.

What was the 'infinity problem' with JetStream 2's WebAssembly tests?

The infinity problem emerged because JetStream 2 measured WebAssembly startup time in two phases: startup and runtime. As browser engines became extremely efficient at instantiating Wasm modules, startup times dropped dramatically—sometimes to zero milliseconds in the measurement due to Date.now() rounding down. The scoring formula was Score = 5000 / Time, so a zero time resulted in an infinite score. This was patched in JetStream 2.2 by clamping the score to 5000, but it signaled that the benchmark was obsolete. A startup time of zero in a microbenchmark doesn't reflect real-world scenarios where Wasm is critical for page loads.

How did JetStream 3 change the way WebAssembly is benchmarked?

JetStream 3 introduces a more nuanced approach to WebAssembly benchmarking, moving away from the two-phase startup/runtime model. Instead, it integrates Wasm workloads that reflect modern usage, such as libraries and UI components, where startup is not a separate isolated phase. The suite focuses on overall throughput and responsiveness, using more realistic scenarios. It also employs higher-precision timing to avoid the zero-time issue. By aligning with actual web application patterns, JetStream 3 encourages optimizations that benefit real users, not just synthetic tests.

What role did the WebKit team play in developing JetStream 3?

The WebKit team contributed significantly to JetStream 3, working alongside colleagues from Google and Mozilla. They focused on engineering improvements in JavaScriptCore, particularly for WebAssembly performance. WebKit aggressively optimized the startup path, which contributed to the infinity problem in JetStream 2. For JetStream 3, the team helped design benchmarks that could accurately measure performance without such artifacts. Their work included refining timing mechanisms and selecting workloads that represent real-world web applications, ensuring the suite drives genuine progress.

Why is it important for benchmarks to evolve with the web?

Benchmarks are one of the best tools for browser engine developers to drive performance, but they must stay current. As the web platform grows—with Wasm, advanced JavaScript, and heavy UI frameworks—benchmarks must reflect these changes. Outdated benchmarks can lead to optimization efforts that don't benefit users. For example, focusing solely on startup time when Wasm is used in critical paths for page loads would miss the bigger picture. Evolving benchmarks like JetStream 3 ensures that performance improvements are targeted where they matter most, keeping the web fast and responsive for everyone.

How will JetStream 3 impact future browser development?

JetStream 3 sets a new standard for performance measurement, prompting browser vendors to prioritize optimizations that align with modern usage. It encourages focus on Wasm in real-world scenarios, large-scale applications, and overall responsiveness rather than narrow microbenchmarks. This will likely lead to more efficient instantiation, better memory handling, and improved runtime performance. For developers, it means a more accurate picture of how their browsers perform with the applications they build. Ultimately, JetStream 3 helps ensure that the web remains a competitive platform for demanding applications like games, productivity tools, and creative software.