Monthly Archives: March 2013

Measuring Real User Experience with Site Speed Gauge

Many web performance testing service vendors, such as Keynote, offer last-mile testing data. In addition, many popular tools, such as WebPagetest, provide a convenient way to submit test cases so that you can collect large volumes of performance metrics. However, most of these services and tools are synthetic, meaning that they are not from real users. The testing agents in a computer performs the tests in a controlled environment.

The real world can be very different. A great number of parameters can affect the data dramatically. We can easily name a few such parameters: network connection speed, last mile speed in particular, browser type, computer CPU power. More importantly, the distribution of parameter values affects the results substantially. For example, we can get DSL and high-speed backbone test results from Keynote for a particular type of browser, but we don’t know exactly how many users have a network speed that’s comparable to the testing network’s speed. Other types of connection speeds that exist in the real world can also be missed in the tests. It’s difficult to put a weight on each of the tests and get an average that reflects the real world.

Several years back, at eBay we began building a framework called Site Speed Gauge to capture real-world, end-user site speed data. Site Speed Gauge has proven to be an effective way to understand real user experience with web performance, while also correlating site speed and the conversion of traffic to purchases. This framework is now the gold standard at eBay for monitoring site speed. Compared to synthetic tests with at most several thousand data points per day for a site page, we are getting millions of data samples for the same page. Large, statistically significant data sampling points help us to get stable trending and to identify real issues. This post provides an overview of how we use Site Speed Gauge to monitor and improve site performance as well as to collect meaningful business metrics.

The art of site speed measurement

We often hear questions such as, Do you know how fast or slow my web page is? Does it matter to end users? Depending on the source of the question, you’ll likely have a different answer. If a page takes 10 seconds to load and the person asking the question can wait that long, then the speed is fine. However, if I am an engineer, I probably would consider 10 seconds as too slow, and I’d want to know why the page is slow. Is the server response slow, or is the network connection slow? How long do I need to wait to see the first impression on the page? How long does it take to render the full page after getting the first byte of data? What is the speed on different types of browsers so I can tune the slow ones? If I am a business person, I would want to know how many users are affected by slow web performance; a distribution of performance data, such as the median or the 90th percentile, would be useful. Most importantly, everyone wants to know the correlation between site speed and the business metrics. Does site speed affect our company’s top line?

To answer these different questions, we will need various measurements and data analysis views. Throughout the design and enhancement of Site Speed Gauge, we have continually ensured that it is extensible to meet different needs. Site speed measurement is an art; you will experience its beauty when it exactly matches what any particular circumstance calls for.

How Site Speed Gauge works

The following diagram describes the 10 steps of Site Speed Gauge, from receiving a user request to reporting data:


For older browser versions, where the Navigation Timing object is not available, we mainly use JavaScript to capture the timers. Client-side timer measurements using JavaScript are relatively easy and accurate, as we can start the timer at the beginning of the page, and use the client machine’s clock to measure any points during the page rendering. The difficult measurement is the total end-to-end time, from when the user’s click initiates the page request through when the page completes with the browser onload event. We can use a cookie to store the timestamp when a user leaves the previous page, and measure end-to-end time when loading of the current page is completed. However, if the previous page is not from eBay—for example, if it is a third-party search or referral page—we will miss the end-to-end time metric.

Therefore, we instead use server-side timers for end-to-end time measurement. The beginning timestamp st1 is the time when an app server gets the request, and the end timestamp st2 is the time when another metrics collection server gets the site speed beacon request. We miss the URL request time for the real user’s page, but we compensate for this fact with the beacon request time. To handle the case of the app server’s clock and the metrics collection server’s clock not being in synch, we can use a time-synch service on the two machines. To provide sufficient accuracy, the synch service should return timestamps in milliseconds. Alternatively, we can use a single database timestamp to eliminate the time synch issue.

For the latest versions of browsers, we also send back measurements from the Navigation Timing object those browsers create. These measurements give us very useful information about client-side performance, such as DNS lookup time and network connection time. Through our analysis of the data, we have identified DNS and connection times as major sources of overhead if our datacenters and our clients are not located on the same continent.

Site Speed Gauge features

Currently, Site Speed Gauge supports these major features:

  • Key performance metrics are provided, such as total page end-to-end time, client-side rendering time, first byte time, DOM ready time, certain JavaScript execution times, server processing time, above-fold time, and graphical ads time. About 30 timers are available from the various stages of page processing.
  • Metrics are broken down by page, browser, device, international site, and client IP location. These breakdowns are in different selection dimensions when you query and view the data.
  • Data sampling is adjustable, from 100% sampling for low-traffic pages, to smaller percentages for high-traffic pages. For a heavily trafficked site like eBay, with billions of page views per day, big data processing and scaling requires controlling the sampling data size.
  • Through the gauge’s support for A/B testing, we can tie site speed to site feature changes. This ability is very useful for collecting business metrics correlation data; more on this in the next section.
  • In addition to collecting web performance data, we can plug in collection of other user behavior data:  user clicking, scrolling, and browser size data. We have built heap maps on top of the data to analyze user interactions with the pages.
  • We can plug in other browser performance objects, such as the Navigation Timing object available in new browser versions. As described previously, this capability enables us to get more visibility into the network layer, such as DNS lookup and network connection times.
  • Site Speed Gauge also supports other client-side capturing, such as JavaScript error capturing. Without this capability, we would be flying blind, unaware of problems until getting complaints from end users.

Integration with A/B testing

Tracking the trending of site speed helps us to identify site feature rollout issues, and to monitor site speed improvements we can achieve from various optimizations. In addition, we can run different versions of a web page at the same time and compare the site speed of each version. The resulting data enables us to correlate business metrics with site speed in a precise way. One of the characteristics of eBay’s business is seasonality; if we can simultaneously monitor site speed and business metrics for a page with seasonal or other variations, we can build meaningful correlations.

To enable such analysis, we have integrated Site Speed Gauge with eBay’s A/B testing platform. In this platform, business metrics are collected and analyzed based on testing and control groups; a user session id identifies which group a user belongs to. We use the same session id to collect site speed as well as business metrics. Once a user is sampled for site speed, all pages viewed by the same user are sampled so that we have all site speed and business metrics data for this user.

Several years ago, we ran two versions of the eBay search page. The versions had the same features but different implementations of the page, one our classic search and the other a new implementation. We collected data on site speed as well as buyer purchases per week (PPW, a business metric related to conversion of traffic to purchases), and over time we found a strong correlation between site speed and PPW. As the chart below shows, the correlation is not linear, starting from 10% site speed and 1 % PPW, increasing to 35% site speed and 5% PPW. Our interpretation of this result is that a small change in page speed might not have much impact, but a large page speed change can have a noticeable effect on end users; for example, a large reduction in site speed can cause user activity to drop, or even abandonment of the site, thus affecting conversions.


As we established the correlation between site speed and PPW, business people and engineers alike began vigorously emphasizing site speed when they designed and implemented features. Now, the engineering culture at eBay is that site speed and optimization are part of the design, implementation, and rollout of features. Many important changes to the site first go through A/B testing to ensure that we’re maximizing site speed as well as business impacts. In addition, the Site Operations team uses dashboards and alerts to monitor site speed 24×7.

Data processing and visualization dashboards and tools

Although we typically collect 10% of site speed sampling data for high-traffic pages, these pages still generate large amounts of beacon data. A batch job running at certain time intervals performs site speed data processing and aggregation in ways that meet the different needs of various consumers of site speed data.

Site Speed Gauge provides dashboards and analytical tools that support data visualization in highly configurable ways. For example, we can select daily results, hourly trending on the 50th percentile, or hourly trending on the 90th percentile. We can view data on a specific eBay page, for a particular international or US site. We can see the site speed for various A/B testing groups.

Here is an example of the site speed breakdown by browser:


And here is an example of the browser usage breakdown:


For those who are interested in network latency and the impacts of a CDN, they can view results by geographic location. We process the data based on client IP address to support this feature. The image below shows a heat map of site speed for the United States. A picture is worth a thousands words; the heat map helps people identify page speed visually.



At eBay, Site Speed Gauge helps us to identify user web performance issues as well as to monitor site speed improvement efforts. Its abilities to collect real-world data and correlate business metrics provide powerful tools in eBay’s highly trafficked consumer-oriented web site. We built extensibility into Site Speed Gauge. In the past several years, we have enhanced Site Speed Gauge to support different needs, and we expect to continue enhancing it in the future.

RaptorJS – An End-to-End JavaScript Toolkit for Building Adaptive Modules and UI Components

RaptorJS is an open-source toolkit for building JavaScript modules and UI components that function on the server and in the browser. RaptorJS promotes modularity, which is crucial to building reasonably complex HTML web applications that are maintainable, easily testable, and optimized. The central goal of RaptorJS is to enable applications to provide the best user experience.

As an eBay project, RaptorJS was designed to be extremely efficient and lightweight. Rather than being a monolithic framework, RaptorJS embraces a modular design; it is intended to work alongside existing JavaScript libraries—not replace them.

RaptorJS provides support for the following features:

  • Object-oriented JavaScript: Defining JavaScript modules, classes, mixins, and enums based on the Asynchronous Module Definition (AMD) syntax
  • Resource optimization: Resource minification, bundling, compilation, checksums, and CDN integration for optimal delivery of JavaScript modules and UI components to the browser
  • Packaging: Defining dependencies for JavaScript modules and UI components based on simple package.json files, with support for optional and environment-specific dependencies
  • Client-side asynchronous package loading: Downloading packages of JavaScript and CSS resources asynchronously after the initial page load
  • Server-side JavaScript module loading: Loading JavaScript modules in multiple server-side JavaScript environments, including Node and Rhino
  • HTML templating: Using the same extensible templating language on both the server and in the browser to build UI components and web pages
  • Widgets: Automatic binding of JavaScript behavior to DOM nodes associated with rendered UI components—whether the UI component was rendered on the server or in the browser

The following sections describe each of these features in more detail.

Object-oriented JavaScript

Using the RaptorJS library, you can define namespaced modules, classes, mixins, and enums. This library employs a clean and easy-to-understand syntax based on the AMD syntax.

To make building modular JavaScript applications even easier, RaptorJS provides an AMD implementation that extends AMD in backwards-compatible ways. The syntax is very close to pure JavaScript, but fills gaps in the existing JavaScript language. Now you can start creating modular code that is easy to maintain without waiting for a new version of JavaScript or switching to an alternative language.

Resource optimization

The RaptorJS Optimizer is a server-side tool to build optimized web pages by bundling, compiling, and minifying page dependencies. This tool makes managing complex JavaScript and CSS dependencies almost effortless while helping you produce extremely optimized web pages.

Unlike other optimizers, the RaptorJS Optimizer does not require that you write your code a certain way. Simply tell the optimizer which modules your page depends on, and it will take care of the rest—including writing optimized JavaScript and CSS bundles to disk and generating the HTML that is required to include those bundles. Since every application has different requirements, the RaptorJS Optimizer gives you full control over how resources are combined and where to include resources on your HTML pages.

All dependencies are described using RaptorJS package.json files, which are easy to maintain and allow you to leverage the full power of the RaptorJS Packaging system.


RaptorJS extends the popular package.json format so that it is better suited for packaging up JavaScript modules and UI components to be delivered to the browser. RaptorJS packages are used by the RaptorJS server-side module loader, as well as to optimally deliver only the required code to the browser. RaptorJS imposes no requirements on the code that is packaged, so you can continue to write code however you prefer. Furthermore, RaptorJS packages are extensible and support any type of dependencies (JavaScript, CSS, LESS, Raptor Templates, etc.).

Here are some of the use cases this packaging model supports:

  • You can create separate RaptorJS packages for JavaScript modules and UI components.
  • JavaScript modules and UI components can be delivered and loaded differently based on the target environment. For example, you can package UI components so that CSS code sent to mobile devices is different from the CSS code sent to desktop browsers.
  • You can make dependencies explicit—including which dependencies are optional or environment-specific.

Client-side asynchronous package loading

RaptorJS includes a lightweight AMD-compatible package/module loader that enables JavaScript and CSS resources to be downloaded asynchronously after the initial page load. This asynchronous package loader works in conjunction with the RaptorJS Optimizer to efficiently download resources from the server.

Server-side JavaScript module loading

RaptorJS provides a server-side module loader that integrates with multiple server-side JavaScript environments, including Node and Rhino. The server-side module loader reads modules’ package files to determine which code to load based on the target environment.

The RaptorJS AMD module loader integrates seamlessly with the Node module loader. In addition, RaptorJS provides a fully compliant CommonJS and AMD module loader for Rhino.

You can load JavaScript modules in multiple server-side JavaScript environments, and you can load JavaScript modules differently in Node and Rhino server-side JavaScript environments.

HTML templating

Raptor Templates is a new templating language that elegantly blends powerful templating directives with HTML tags. Unlike most other templating languages, Raptor Templates also enables you to use custom tags to embed high-level UI components into templates.

Raptor Templates is XML-based so that the XML structure of an HTML document can be used to its full advantage, making it simpler for you to write easily readable templates and to provide optimal output. Raptor Templates includes a compiler that converts HTML templates into native and optimized JavaScript code. Equally effective in both the browser and on the server, no other templating language makes it so effortless to produce easily readable templates that are blazingly fast with such a miniscule footprint.

Use Raptor Templates wherever you want to produce HTML.


The RaptorJS Widget Framework simplifies the creation of web applications built using UI components. This lightweight framework provides a mechanism for automatically attaching JavaScript behavior to the DOM subtrees associated with rendered UI components—regardless of whether the UI components were rendered on the server or in the web browser.

The RaptorJS Widget Framework does not dictate how a client-side widget is implemented. Furthermore, there is no complex class inheritance hierarchy (only a few mixins that get applied to all initialized widgets).

UI components that use Raptor Templates to render their view will benefit from the bindings that allow widgets to be attached to rendered HTML elements. During the rendering of an HTML template, the widget framework keeps track of which widgets have been rendered and which HTML element each widget is bound to. As a result, widgets can be automatically and efficiently initialized without having to rely on the CPU-intensive task of scanning the final DOM tree to discover rendered widgets.


Lightweight, optimized, modular, portable, and designed for scale, RaptorJS simply makes JavaScript better. To view demos and get started, visit

Kudos to Patrick Steele-Idem (@psteeleidem) of eBay’s presentation platform team, who built RaptorJS.

— Neeraj Khanna (@nk94555), presentation engineer at eBay

Fine-Tuning the eBay Technical Infrastructure: A New Methodology

I’ve been with eBay Inc. since 2009, and even just in the past four years, I’ve witnessed a significant change in the growth and importance of data centers, and the way we, as an industry, measure their efficiency. The Green Grid’s Power Usage Effectiveness (PUE) metric – as well as the water and carbon equivalents of the same metric – have spurred organizations to track and report on these measurements, demonstrating greater data center efficiency for their businesses and in many cases even sparking competition between companies.

The PUE standard for data center efficiency has brought immeasurable benefit to the industry; however, it’s only one piece of a larger puzzle that transcends data centers and energy efficiency. In fact, if anything has become clear over the past several years, it’s that technology decisions are inextricably linked to business performance. We can see more clearly now than ever before that our designs, purchases and operating decisions have real, tangible effects on the key indicators that are important to running a business: cost, performance, environmental impact and, ultimately, revenue. If we believe this to be true – and I certainly do – then we can no longer make choices regarding our technical infrastructure  in a piecemeal, isolated, component-by-component manner that ignores the larger context of the decision. And yet the way we analyze and optimize the efficiency of our technical infrastructure – our very ability to measure our value to business leaders and stakeholders – hasn’t, until now, connected these indicators in a meaningful way.

At eBay Inc., we’ve spent the past two years developing a new approach to address this gap. Today, we’re releasing the Digital Service Efficiency methodology, or DSE for short. As you’ll see below, DSE is essentially a dashboard for a company’s technical ecosystem – the data centers, compute equipment and software that combine to deliver its digital services to consumers. Much like a dashboard in a car, DSE offers a straightforward approach to measuring the overall performance of technical infrastructure across four key business priorities: performance, cost, environmental impact and revenue. Similar in spirit to the miles-per-gallon (MPG) measurement for cars, DSE enables executives and technical professionals to see the results of how their “engine” performed with real customer consumption – or, rather, how the engine performed as our users drove it. Like its vehicular counterpart, the DSE dashboard equips decision-makers to see the results of their technical infrastructure choices to date (i.e., what MPG they achieved with their design and operations), and serves as the flexible tool they need when faced with making new decisions (i.e., what knobs to turn to achieve maximum performance across all dimensions). Ultimately, DSE enables balance within the technology ecosystem by exposing how turning knobs in one dimension affects the others.

Here’s what it looks like:

In the case of, what you see is how we’ve combined all of our customer-initiated requests into two distinct services: buy and sell. DSE measures how many business transactions are completed per kilowatt-hour of electric power consumed for these services. Or put another way, it shows how much energy is consumed to deliver each transaction. Transactions divided by kilowatt-hours reveals the performance of each service. Once these two variables are established, a family of interconnected metrics can be derived to provide further insights for optimizations — for example, the cost per transaction, tonnes of CO2e generated per transaction and revenue created per transaction. See more of these details by clicking around the dashboard for yourself.

With this information, we’re able to see and do things that we simply weren’t equipped for previously. As an example, by using the DSE dashboard, some of our software engineers saw that by slightly decreasing the memory allocated for an application in a pool of servers, they could remove 400 servers from the pool. This insight helped us eliminate nearly a megawatt of power consumption and avoid spending more than $2 million to refresh the servers. This simple software tweak helped us lower power consumption, decrease costs, and increase system performance, ultimately increasing our revenue per transaction. And that’s just one example.

Thinking bigger-picture and longer-term, DSE is enabling us to project forward and build tangible plans for what we as a business need to do today to realize the results we’re looking to achieve tomorrow. In essence, we’re able to balance and tune our technical infrastructure to optimize how effectively it’s driving core business performance. And as business priorities fluctuate over time – envision a scenario in which, say, federal or state climate change legislation passes in a couple of years and we’re forced to be even more aggressive in the way we address our environmental impacts – we have the flexibility to model the various choices and trade-offs we’ll need to face, from an IT infrastructure and operations perspective, so that we can help propel the business forward to achieve its goals.

By releasing DSE, and our corresponding data, our intent is to establish a baseline for 2012 and then to set productivity and efficiency goals for 2013. These are our goals for the year ahead:

  • Increase transactions per kWh by 10%
  • Reduce cost per transaction by 10%
  • Reduce carbon per transaction by 10%

We’ll be updating and reporting out on all of the numbers in the dashboard and on our progress toward our improvement goals each quarter. As we continue on our course to fine-tune our own technical infrastructure, we hope to encourage other companies – particularly those that deliver their services digitally – to explore the DSE methodology and see how it can help expose the business value and performance of their own technical infrastructure. And while each company is different in terms of the services they provide and the variables that can be tuned to balance the cost, performance and environmental impact of delivering those services, DSE provides a framework for how other companies can begin.

Our DSE solution paper is just the beginning of what we hope will be an exciting new chapter in how we, as an industry, think about measurement and value. By sharing our methodology and progress, we at eBay Inc. hope to start that conversation. And as other companies are inspired to make their technical infrastructure as efficient and productive as possible, together we as an industry can drive the digital economy to even greater heights.

To comment or continue the conversation, feel free to email