eBay Tech Blog

Want to hear what some of the world’s most talented testers think about the changing face of software and software testing? Then come to the Conference of the Association for Software Testing (CAST), where you’ll also get a chance to talk with these testers and explore your own thoughts and ideas.

CAST is put together by testers, for testers (and those interested in testing). This year, it takes place in Madison, Wisconsin, August 26-28. The presenters are among the best practitioners from around the globe, and many attendees travel thousands of miles specifically for this conference. eBay will have a strong presence, including a keynote by Quality Engineering Director Jon Bach and a track presentation by Ilari Henrik Aegerter, manager of quality engineering in Europe.

Unlike many testing conferences, at CAST a third of each presentation is reserved for a “threaded” question-and-answer session, in which the audience uses colored cards to indicate new questions or questions related to the current thread. With this format, you can satisfy your curiosity, raise doubts, and make presenters defend their positions. That includes the keynote speakers. The conference also includes a test lab where you can get hands-on and try out ideas you might have heard about, see how other testers test, and share your own experience. You’ll find testers hanging out in the hallways having in-depth discussions and challenging each other to demonstrate their testing skills. Everything about the environment is designed to support testers who want to excel at their craft.

The theme for CAST 2013 is “Old Lessons applied and new lessons learned: advancing the practice and building a foundation for the future.”  The technology we work with changes at a rapid pace. Some testing practices stand the test of time; others become obsolete and irrelevant as the technology changes around them.

If this conference sounds like something you’d like to be a part of, then I urge you to register.

http://www.associationforsoftwaretesting.org/conference/cast-2013

I’d love to see you there.

- Ben Kelly, eBay EUQE team and a content chair for CAST 2013

{ 0 comments }

As discussed in a previous post, earlier this year we unveiled the Digital Service Efficiency (DSE) methodology, our miles-per-gallon (MPG) equivalent for viewing the productivity and efficiency of our technical infrastructure across four key areas: performance, cost, environmental impact, and revenue. The goal of releasing DSE was to provide a transparent view of how the eBay engine is performing, as well as spark an industry dialogue between companies, big and small, about how they use their technical infrastructure and the impact it’s having on their business and the planet. In the past month, we’ve been excited to see – and participate in – the resulting dialogue and conversation.

When we shared DSE externally for the first time, we also set a number of performance goals for 2013 and committed to disclosing updates on our progress on a quarterly basis. Today, we’re pleased to share our first such quarterly update with year-over-year comparisons and analysis. This post provides highlights of our findings as well as links to the DSE dashboard and other details.

Dash_Q1-600x339

Q1 metrics summary

Here’s where we stand on the progress toward 2013 performance and efficiency goals:

  • Our transactions per kWh increased 18% year over year. The growth of eBay Marketplaces and continued focus on driving efficiencies have contributed to this increase.
  • Our cost per transaction decreased by 23% in Q1 alone, already exceeding our initial goal.
  • Our carbon per transaction showed a net decrease of 7% for the quarter. As we’re still on track for our Utah Bloom fuel cell installation to go live this summer, we expect this number to continue to decrease and contribute significantly to our 10% carbon reduction goal for the year, even with our projected business growth.

Recognizing that our dynamic infrastructure changes each quarter, we’re confident that we’re on track for our 10% net gain across performance, cost, and environmental impact for the year.

Trends

We’ve seen a few other interesting trends as well:

The New eBay: Over the past year we’ve added numerous new features to our site. Last fall, we rolled out our feed technology, which makes the shopping experience on eBay.com more personal to each of our individual users. On the backend, with this curated site personalization, we’ve seen a jump in our transaction URLs as the eBay feed technology increases infrastructure utilization. This is one example of our customers’ site use driving a more productive work engine.

Continued Growth and Efficiency: As you can see on the dashboard, we had a significant spike in the number of physical servers powering eBay.com – a 37% increase year over year. The rise is in direct response to increased customer demand and personalization for our users. And while we added a significant number of new servers, we were able to limit our increase in power consumption to just 16% (2.69 MW). Compared to the first quarter of last year, we’ve reduced the power consumed by an average of 63 watts per server; this is a direct result of our efforts to run more energy-efficient servers and facilities.

Running a Cleaner Engine: As eBay aspires to be the leading global engine for greener commerce, we’re continually focused on integrating cleaner and renewable energy into our portfolio. In March, our Salt Lake City data center’s solar array came online and though it’s relatively small, it increased our owned clean energy powering eBay.com by 0.17%. Our active projects with fuel cells and renewable energy sourcing will continue to increase this value through the year.

Continuing to Refine Our Approach

When we first announced DSE, one of our top priorities was continued transparency into our metrics, calculations, and lessons learned. Along those lines, the greater transparency has also sparked internal conversations at eBay about how our server pools are categorized.

We organize each server into one of three categories: “buy,” “sell,” or “shared.” “Buy” and “sell” serve the pages and APIs directly for customer traffic, which count as our transactions (URLs); “shared” is the backend support equipment, which does not receive external traffic.

When we released the 2012 DSE numbers we reported 7.3 trillion URLs or transactions in the “buy” and “sell” groups. As we rolled up the Q1 2013 numbers, we found that some internally facing servers were grouped into “buy” and “sell.” We moved them to the appropriate “shared” group. While this did not change overall server counts or power consumption, it did decrease the 2012 transactions count coming from the external servers to 4.3 trillion. We also moved some server groups from “sell” to “buy” to be more consistent with our business model (unlike before, when “buy” and “sell” were relatively more equal). We felt it was important to stay strict with our methodology, and so we have retroactively updated the 2012 baseline numbers to ensure that our year-over-year results were standard and consistent.

Based on lessons learned in the Q1 2013 work, we’ve fine-tuned our methodology as follows:

  1. We’ll look only at those server pools that receive external web traffic in order to ensure that we’re accurately speaking to the year-over-year comparisons for “buy” and “sell” – all other server pools not receiving external web traffic will be considered “shared.”
  2. We’re now measuring Revenue per MW hour (as opposed to Revenue per MW), as this metric represents total consumption per quarter and year instead of quarterly averages. Likewise, we’ve decided to measure CO2e per MW hour instead of CO2e per MW.

Conclusion

With the Q1 2013 results under our belt, we’re happy with the progress we’ve made to fine-tune our technical infrastructure and the DSE metric. We still have more to do, though, and we’ll be sure to keep you updated along the way.

You can find the full Q1 2013 DSE results at http://dse.ebay.com, and click through the dashboard yourself to see the full cost, performance, and environmental impact of our customer “buy” and “sell” transactions.

Have a question or comment? Leave a note in the comments below or email dse@ebay.com and we’ll be sure to get back to you.

- Sri Shivananda, Vice President of Platform, Infrastructure and Engineering Systems, eBay Inc.

 

 

{ 2 comments }

For the most part, eBay runs on a Java-based tech stack. Our entire workflow centers around Java and the JVM. Considering the scale of traffic and the stability required by a site like ebay.com, using a proven technology was an obvious choice. But we have always been open to new technologies, and Node.js has been topping the list of candidates for quite some time. This post highlights a few aspects of how we developed eBay’s first Node.js application.

Scalability

It all started when a bunch of eBay engineers (Steven, Venkat, and Senthil) wanted to bring an eBay Hackathon-winning project called “Talk” to production. When we found that Java did not seem to fit the project requirements (no offense), we began exploring the world of Node.js. Today, we have a full Node.js production stack ready to rock. 

We had two primary requirements for the project. First was to make the application as real time as possible–i.e., maintain live connections with the server. Second was to orchestrate a huge number of eBay-specific services that display information on the page–i.e., handle I/O-bound operations. We started with the basic Java infrastructure, but it consumed many more resources than expected, raising questions about scalability for production. These concerns led us to build a new mid-tier orchestrator from scratch, and Node.js seemed to be a perfect fit.

Mindset

Since eBay revolves around Java and since Java is a strongly typed static language, initially it was very difficult to convince folks to use JavaScript on the backend. The numerous questions involved ensuring type safety, handling errors, scaling, etc. In addition, JavaScript itself (being the world’s most misunderstood language) further fueled the debate. To address concerns, we created an internal wiki and invited engineers to express their questions, concerns, doubts, or anything else about Node.js.

Within a couple of days, we had an exhaustive list to work on. As expected, the most common questions centered around the reliability of the stack and the efficiency of Node.js in handling eBay-specific functionality previously implemented in Java. We answered each one of the questions, providing details with real-world examples. At times this exercise was eye-opening even for us, as we had never considered the angle that some of the questions presented. By the end of the exercise, people understood the core value of Node.js; indeed, some of the con arguments proved to be part of the beauty of the language.

Once we had passed the test of our peers’ scrutiny, we were all clear to roll.

Startup

We started from a clean slate. Our idea was to build a bare minimum boilerplate Node.js server that scales; we did not want to bloat the application by introducing a proprietary framework. The first four node modules we added as dependencies were express, clusterrequest, and async. For data persistence, we decided on MongoDB, to leverage its ease of use as well as its existing infrastructure at eBay. With this basic setup, we were able to get the server up and running on our developer boxes. The server accepted requests, orchestrated a few eBay APIs, and persisted some data.

For end-to-end testing, we configured our frontend servers to point to the Node.js server, and things seemed to work fine. Now it was time to get more serious. We started white-boarding all of our use cases, nailed down the REST end points, designed the data model and schema, identified the best node modules for the job, and started implementing each end point. The next few weeks we were heads down–coding, coding, and coding.   

Deployment

Once the application reached a stable point, it was time to move from a developer instance to a staging environment. This is when we started looking into deployment of the Node.js stack. Our objectives for deployment were simple: Automate the process, build once, and deploy everywhere. This is how Java deployment works, and we wanted Node.js deployment to be as seamless and easy as possible.

We were able to leverage our existing cloud-based deployment system. All we needed to do was write a shell script and run it through our Hudson CI job. Whenever code is checked in to the master branch, the Hudson CI job kicks off. Using the shell script, this job builds and packages the Node.js bundle, then pushes it to the deployment cloud. The cloud portal provides an easy user interface to choose the environment (QA, staging, or pre-production) and activate the application on the associated machines.

Now we had our Node.js web service running in various stable environments. This whole deployment setup was quicker and simpler than we had expected.  

Monitoring

At eBay, we have logging APIs that are well integrated with the Java thread model as well as at the JVM level. An excellent monitoring dashboard built on top of the log data can generate reports, along with real-time alerts if anything goes wrong. We achieved similar monitoring for the Node.js stack by hooking into the centralized logging system. Fortunately for us, we had logging APIs to consume. We developed a logger module and implemented three different logging APIs:

  1. Code-level logging. This level includes logging of errors/exceptions, DB queries, HTTP service calls, transaction metadata, etc.
  2. Machine-level logging. This level includes heartbeat data about CPU/memory and other OS statistics. Machine-level logging occurs at the cluster module level; we extended the npm cluster module and created an eBay-specific version.
  3. Logging at the load balancer level. All Node.js production machines are behind a load balancer, which sends periodic signals to the machines and ensures they are in good health. In the case of a machine going down, the load balancer fails-over to a backup machine and sends alerts to the operations and engineering teams.

We made sure the log data formats exactly matched the Java-based logs, thus generating the same dashboards and reports that everyone is familiar with.

One particular logging challenge we faced was due to the asynchronous nature of the Node.js event loop. The result was that the logging of transactions was completely crossed. To understand the problem, let’s consider the following use case:  The Node process starts a URL transaction and issues a DB query with an async callback. The process will now proceed with the next request, before the DB transaction finishes. This being a normal scenario in any event loop-based model like Node.js, the logs are crossed between multiple URL transactions, and the reporting tool shows scrambled output. We have worked out both short-term and long-term resolutions for this issue.

Conclusion

With all of the above work completed, we are ready to go live with our Hackathon project. This is indeed the first eBay application to have a backend service running on Node.js. We’ve already had an internal employee-only launch, and the feedback was very positive–particularly on the performance side. Exciting times are ahead!

A big shout-out to our in-house Node.js expert Cylus Penkar, for his guidance and contributions throughout the project. With the success of the Node.js backend stack, eBay’s platform team is now developing a full-fledged frontend stack running on Node.js. The stack will leverage most of our implementation, in addition to frontend-specific features like L10N, management of resources (JS/CSS/images), and tracking. For frontend engineers, this is a dream come true; and we can proudly say, “JavaScript is EVERYWHERE.”

Senthil Padmanabhan & Steven Luan
Engineers @ eBay

{ 55 comments }

Many web performance testing service vendors, such as Keynote, offer last-mile testing data. In addition, many popular tools, such as WebPagetest, provide a convenient way to submit test cases so that you can collect large volumes of performance metrics. However, most of these services and tools are synthetic, meaning that they are not from real users. The testing agents in a computer performs the tests in a controlled environment.

The real world can be very different. A great number of parameters can affect the data dramatically. We can easily name a few such parameters: network connection speed, last mile speed in particular, browser type, computer CPU power. More importantly, the distribution of parameter values affects the results substantially. For example, we can get DSL and high-speed backbone test results from Keynote for a particular type of browser, but we don’t know exactly how many users have a network speed that’s comparable to the testing network’s speed. Other types of connection speeds that exist in the real world can also be missed in the tests. It’s difficult to put a weight on each of the tests and get an average that reflects the real world.

Several years back, at eBay we began building a framework called Site Speed Gauge to capture real-world, end-user site speed data. Site Speed Gauge has proven to be an effective way to understand real user experience with web performance, while also correlating site speed and the conversion of traffic to purchases. This framework is now the gold standard at eBay for monitoring site speed. Compared to synthetic tests with at most several thousand data points per day for a site page, we are getting millions of data samples for the same page. Large, statistically significant data sampling points help us to get stable trending and to identify real issues. This post provides an overview of how we use Site Speed Gauge to monitor and improve site performance as well as to collect meaningful business metrics.

The art of site speed measurement

We often hear questions such as, Do you know how fast or slow my web page is? Does it matter to end users? Depending on the source of the question, you’ll likely have a different answer. If a page takes 10 seconds to load and the person asking the question can wait that long, then the speed is fine. However, if I am an engineer, I probably would consider 10 seconds as too slow, and I’d want to know why the page is slow. Is the server response slow, or is the network connection slow? How long do I need to wait to see the first impression on the page? How long does it take to render the full page after getting the first byte of data? What is the speed on different types of browsers so I can tune the slow ones? If I am a business person, I would want to know how many users are affected by slow web performance; a distribution of performance data, such as the median or the 90th percentile, would be useful. Most importantly, everyone wants to know the correlation between site speed and the business metrics. Does site speed affect our company’s top line?

To answer these different questions, we will need various measurements and data analysis views. Throughout the design and enhancement of Site Speed Gauge, we have continually ensured that it is extensible to meet different needs. Site speed measurement is an art; you will experience its beauty when it exactly matches what any particular circumstance calls for.

How Site Speed Gauge works

The following diagram describes the 10 steps of Site Speed Gauge, from receiving a user request to reporting data:

10steps

For older browser versions, where the Navigation Timing object is not available, we mainly use JavaScript to capture the timers. Client-side timer measurements using JavaScript are relatively easy and accurate, as we can start the timer at the beginning of the page, and use the client machine’s clock to measure any points during the page rendering. The difficult measurement is the total end-to-end time, from when the user’s click initiates the page request through when the page completes with the browser onload event. We can use a cookie to store the timestamp when a user leaves the previous page, and measure end-to-end time when loading of the current page is completed. However, if the previous page is not from eBay—for example, if it is a third-party search or referral page—we will miss the end-to-end time metric.

Therefore, we instead use server-side timers for end-to-end time measurement. The beginning timestamp st1 is the time when an app server gets the request, and the end timestamp st2 is the time when another metrics collection server gets the site speed beacon request. We miss the URL request time for the real user’s page, but we compensate for this fact with the beacon request time. To handle the case of the app server’s clock and the metrics collection server’s clock not being in synch, we can use a time-synch service on the two machines. To provide sufficient accuracy, the synch service should return timestamps in milliseconds. Alternatively, we can use a single database timestamp to eliminate the time synch issue.

For the latest versions of browsers, we also send back measurements from the Navigation Timing object those browsers create. These measurements give us very useful information about client-side performance, such as DNS lookup time and network connection time. Through our analysis of the data, we have identified DNS and connection times as major sources of overhead if our datacenters and our clients are not located on the same continent.

Site Speed Gauge features

Currently, Site Speed Gauge supports these major features:

  • Key performance metrics are provided, such as total page end-to-end time, client-side rendering time, first byte time, DOM ready time, certain JavaScript execution times, server processing time, above-fold time, and graphical ads time. About 30 timers are available from the various stages of page processing.
  • Metrics are broken down by page, browser, device, international site, and client IP location. These breakdowns are in different selection dimensions when you query and view the data.
  • Data sampling is adjustable, from 100% sampling for low-traffic pages, to smaller percentages for high-traffic pages. For a heavily trafficked site like eBay, with billions of page views per day, big data processing and scaling requires controlling the sampling data size.
  • Through the gauge’s support for A/B testing, we can tie site speed to site feature changes. This ability is very useful for collecting business metrics correlation data; more on this in the next section.
  • In addition to collecting web performance data, we can plug in collection of other user behavior data:  user clicking, scrolling, and browser size data. We have built heap maps on top of the data to analyze user interactions with the pages.
  • We can plug in other browser performance objects, such as the Navigation Timing object available in new browser versions. As described previously, this capability enables us to get more visibility into the network layer, such as DNS lookup and network connection times.
  • Site Speed Gauge also supports other client-side capturing, such as JavaScript error capturing. Without this capability, we would be flying blind, unaware of problems until getting complaints from end users.

Integration with A/B testing

Tracking the trending of site speed helps us to identify site feature rollout issues, and to monitor site speed improvements we can achieve from various optimizations. In addition, we can run different versions of a web page at the same time and compare the site speed of each version. The resulting data enables us to correlate business metrics with site speed in a precise way. One of the characteristics of eBay’s business is seasonality; if we can simultaneously monitor site speed and business metrics for a page with seasonal or other variations, we can build meaningful correlations.

To enable such analysis, we have integrated Site Speed Gauge with eBay’s A/B testing platform. In this platform, business metrics are collected and analyzed based on testing and control groups; a user session id identifies which group a user belongs to. We use the same session id to collect site speed as well as business metrics. Once a user is sampled for site speed, all pages viewed by the same user are sampled so that we have all site speed and business metrics data for this user.

Several years ago, we ran two versions of the eBay search page. The versions had the same features but different implementations of the page, one our classic search and the other a new implementation. We collected data on site speed as well as buyer purchases per week (PPW, a business metric related to conversion of traffic to purchases), and over time we found a strong correlation between site speed and PPW. As the chart below shows, the correlation is not linear, starting from 10% site speed and 1 % PPW, increasing to 35% site speed and 5% PPW. Our interpretation of this result is that a small change in page speed might not have much impact, but a large page speed change can have a noticeable effect on end users; for example, a large reduction in site speed can cause user activity to drop, or even abandonment of the site, thus affecting conversions.

fig2_revised

As we established the correlation between site speed and PPW, business people and engineers alike began vigorously emphasizing site speed when they designed and implemented features. Now, the engineering culture at eBay is that site speed and optimization are part of the design, implementation, and rollout of features. Many important changes to the site first go through A/B testing to ensure that we’re maximizing site speed as well as business impacts. In addition, the Site Operations team uses dashboards and alerts to monitor site speed 24×7.

Data processing and visualization dashboards and tools

Although we typically collect 10% of site speed sampling data for high-traffic pages, these pages still generate large amounts of beacon data. A batch job running at certain time intervals performs site speed data processing and aggregation in ways that meet the different needs of various consumers of site speed data.

Site Speed Gauge provides dashboards and analytical tools that support data visualization in highly configurable ways. For example, we can select daily results, hourly trending on the 50th percentile, or hourly trending on the 90th percentile. We can view data on a specific eBay page, for a particular international or US site. We can see the site speed for various A/B testing groups.

Here is an example of the site speed breakdown by browser:

browser_site_speed

And here is an example of the browser usage breakdown:

browser_usage

For those who are interested in network latency and the impacts of a CDN, they can view results by geographic location. We process the data based on client IP address to support this feature. The image below shows a heat map of site speed for the United States. A picture is worth a thousands words; the heat map helps people identify page speed visually.

geographic

Conclusion

At eBay, Site Speed Gauge helps us to identify user web performance issues as well as to monitor site speed improvement efforts. Its abilities to collect real-world data and correlate business metrics provide powerful tools in eBay’s highly trafficked consumer-oriented web site. We built extensibility into Site Speed Gauge. In the past several years, we have enhanced Site Speed Gauge to support different needs, and we expect to continue enhancing it in the future.

{ 3 comments }

RaptorJS is an open-source toolkit for building JavaScript modules and UI components that function on the server and in the browser. RaptorJS promotes modularity, which is crucial to building reasonably complex HTML web applications that are maintainable, easily testable, and optimized. The central goal of RaptorJS is to enable applications to provide the best user experience.

As an eBay project, RaptorJS was designed to be extremely efficient and lightweight. Rather than being a monolithic framework, RaptorJS embraces a modular design; it is intended to work alongside existing JavaScript libraries—not replace them.

RaptorJS provides support for the following features:

  • Object-oriented JavaScript: Defining JavaScript modules, classes, mixins, and enums based on the Asynchronous Module Definition (AMD) syntax
  • Resource optimization: Resource minification, bundling, compilation, checksums, and CDN integration for optimal delivery of JavaScript modules and UI components to the browser
  • Packaging: Defining dependencies for JavaScript modules and UI components based on simple package.json files, with support for optional and environment-specific dependencies
  • Client-side asynchronous package loading: Downloading packages of JavaScript and CSS resources asynchronously after the initial page load
  • Server-side JavaScript module loading: Loading JavaScript modules in multiple server-side JavaScript environments, including Node and Rhino
  • HTML templating: Using the same extensible templating language on both the server and in the browser to build UI components and web pages
  • Widgets: Automatic binding of JavaScript behavior to DOM nodes associated with rendered UI components—whether the UI component was rendered on the server or in the browser

The following sections describe each of these features in more detail.

Object-oriented JavaScript

Using the RaptorJS library, you can define namespaced modules, classes, mixins, and enums. This library employs a clean and easy-to-understand syntax based on the AMD syntax.

To make building modular JavaScript applications even easier, RaptorJS provides an AMD implementation that extends AMD in backwards-compatible ways. The syntax is very close to pure JavaScript, but fills gaps in the existing JavaScript language. Now you can start creating modular code that is easy to maintain without waiting for a new version of JavaScript or switching to an alternative language.

Resource optimization

The RaptorJS Optimizer is a server-side tool to build optimized web pages by bundling, compiling, and minifying page dependencies. This tool makes managing complex JavaScript and CSS dependencies almost effortless while helping you produce extremely optimized web pages. 

Unlike other optimizers, the RaptorJS Optimizer does not require that you write your code a certain way. Simply tell the optimize

r which modules your page depends on, and it will take care of the rest—including writing optimized JavaScript and CSS bundles to disk and generating the HTML that is required to include those bundles. Since every application has different requirements, the RaptorJS Optimizer gives you full control over how resources are combined and where to include resources on your HTML pages.

All dependencies are described using RaptorJS package.json files, which are easy to maintain and allow you to leverage the full power of the RaptorJS Packaging system.

Packaging

RaptorJS extends the popular package.json format so that it is better suited for packaging up JavaScript modules and UI components to be delivered to the browser. RaptorJS packages are used by the RaptorJS server-side module loader, as well as to optimally deliver only the required code to the browser. RaptorJS imposes no requirements on the code that is packaged, so you can continue to write code however you prefer. Furthermore, RaptorJS packages are extensible and support any type of dependencies (JavaScript, CSS, LESS, Raptor Templates, etc.).

Here are some of the use cases this packaging model supports:

  • You can create separate RaptorJS packages for JavaScript modules and UI components.
  • JavaScript modules and UI components can be delivered and loaded differently based on the target environment. For example, you can package UI components so that CSS code sent to mobile devices is different from the CSS code sent to desktop browsers.
  • You can make dependencies explicit—including which dependencies are optional or environment-specific.

Client-side asynchronous package loading

RaptorJS includes a lightweight AMD-compatible package/module loader that enables JavaScript and CSS resources to be downloaded asynchronously after the initial page load. This asynchronous package loader works in conjunction with the RaptorJS Optimizer to efficiently download resources from the server.

Server-side JavaScript module loading

RaptorJS provides a server-side module loader that integrates with multiple server-side JavaScript environments, inclu

ding Node and Rhino. The server-side module loader reads modules’ package files to determine which code to load based on the target environment.

The RaptorJS AMD module loader integrates seamlessly with the Node module loader. In addition, RaptorJS provides a fully compliant CommonJS and AMD module loader for Rhino.

You can load JavaScript modules in multiple server-side JavaScript environments, and you can load JavaScript modules differently in Node and Rhino server-side JavaScript environments.

HTML templating

Raptor Templates is a new templating language that elegantly blends powerful templating directives with HTML tags. Unlike most other templating languages, Raptor Templates also enables you to use custom tags to embed high-level UI components into templates.

Raptor Templates is XML-based so that the XML structure of an HTML document can be used to its full advantage, making it simpler for you to write easily readable templates and to provide optimal output. Raptor Templates includes a compiler that converts HTML templates into native and optimized JavaScript code. Equally effective in both the browser and on the server, no other templating language makes it so effortless to produce easily readable templates that are blazingly fast with such a miniscule footprint.

Use Raptor Templates wherever you want to produce HTML.

Widgets

The RaptorJS Widget Framework simplifies the creation of web applications built using UI components. This lightweight framework provides a mechanism for automatically attaching JavaScript behavior to the DOM subtrees associated with rendered UI components—regardless of whether the UI components were rendered on the server or in the web browser.

The RaptorJS Widget Framework does not dictate how a client-side widget is implemented. Furthermore, there is no complex class inheritance hierarchy (only a few mixins that get applied to all initialized widgets).

UI components that use Raptor Templates to render their view will benefit from the bindings that allow widgets to be attached to rendered HTML elements. During the rendering of an HTML template, the widget framework keeps track of which widgets have been rendered and which HTML element each widget is bound to. As a result, widgets can be automatically and efficiently initialized without having to rely on the CPU-intensive task of scanning the final DOM tree to discover rendered widgets.

Conclusion

Lightweight, optimized, modular, portable, and designed for scale, RaptorJS simply makes JavaScript better. To view demos and get started, visit http://raptorjs.org

Kudos to Patrick Steele-Idem (@psteeleidem) of eBay’s presentation platform team, who built RaptorJS.

— Neeraj Khanna (@nk94555), presentation engineer at eBay

{ 0 comments }

I’ve been with eBay Inc. since 2009, and even just in the past four years, I’ve witnessed a significant change in the growth and importance of data centers, and the way we, as an industry, measure their efficiency. The Green Grid’s Power Usage Effectiveness (PUE) metric – as well as the water and carbon equivalents of the same metric – have spurred organizations to track and report on these measurements, demonstrating greater data center efficiency for their businesses and in many cases even sparking competition between companies.

The PUE standard for data center efficiency has brought immeasurable benefit to the industry; however, it’s only one piece of a larger puzzle that transcends data centers and energy efficiency. In fact, if anything has become clear over the past several years, it’s that technology decisions are inextricably linked to business performance. We can see more clearly now than ever before that our designs, purchases and operating decisions have real, tangible effects on the key indicators that are important to running a business: cost, performance, environmental impact and, ultimately, revenue. If we believe this to be true – and I certainly do – then we can no longer make choices regarding our technical infrastructure  in a piecemeal, isolated, component-by-component manner that ignores the larger context of the decision. And yet the way we analyze and optimize the efficiency of our technical infrastructure – our very ability to measure our value to business leaders and stakeholders – hasn’t, until now, connected these indicators in a meaningful way.

At eBay Inc., we’ve spent the past two years developing a new approach to address this gap. Today, we’re releasing the Digital Service Efficiency methodology, or DSE for short. As you’ll see below, DSE is essentially a dashboard for a company’s technical ecosystem – the data centers, compute equipment and software that combine to deliver its digital services to consumers. Much like a dashboard in a car, DSE offers a straightforward approach to measuring the overall performance of technical infrastructure across four key business priorities: performance, cost, environmental impact and revenue. Similar in spirit to the miles-per-gallon (MPG) measurement for cars, DSE enables executives and technical professionals to see the results of how their “engine” performed with real customer consumption – or, rather, how the ebay.com engine performed as our users drove it. Like its vehicular counterpart, the DSE dashboard equips decision-makers to see the results of their technical infrastructure choices to date (i.e., what MPG they achieved with their design and operations), and serves as the flexible tool they need when faced with making new decisions (i.e., what knobs to turn to achieve maximum performance across all dimensions). Ultimately, DSE enables balance within the technology ecosystem by exposing how turning knobs in one dimension affects the others.

Here’s what it looks like:

In the case of ebay.com, what you see is how we’ve combined all of our customer-initiated requests into two distinct services: buy and sell. DSE measures how many business transactions are completed per kilowatt-hour of electric power consumed for these services. Or put another way, it shows how much energy is consumed to deliver each transaction. Transactions divided by kilowatt-hours reveals the performance of each service. Once these two variables are established, a family of interconnected metrics can be derived to provide further insights for optimizations — for example, the cost per transaction, tonnes of CO2e generated per transaction and revenue created per transaction. See more of these details by clicking around the dashboard for yourself.

With this information, we’re able to see and do things that we simply weren’t equipped for previously. As an example, by using the DSE dashboard, some of our software engineers saw that by slightly decreasing the memory allocated for an application in a pool of servers, they could remove 400 servers from the pool. This insight helped us eliminate nearly a megawatt of power consumption and avoid spending more than $2 million to refresh the servers. This simple software tweak helped us lower power consumption, decrease costs, and increase system performance, ultimately increasing our revenue per transaction. And that’s just one example.

Thinking bigger-picture and longer-term, DSE is enabling us to project forward and build tangible plans for what we as a business need to do today to realize the results we’re looking to achieve tomorrow. In essence, we’re able to balance and tune our technical infrastructure to optimize how effectively it’s driving core business performance. And as business priorities fluctuate over time – envision a scenario in which, say, federal or state climate change legislation passes in a couple of years and we’re forced to be even more aggressive in the way we address our environmental impacts – we have the flexibility to model the various choices and trade-offs we’ll need to face, from an IT infrastructure and operations perspective, so that we can help propel the business forward to achieve its goals.

By releasing DSE, and our corresponding data, our intent is to establish a baseline for 2012 and then to set productivity and efficiency goals for 2013. These are our goals for the year ahead:

  • Increase transactions per kWh by 10%
  • Reduce cost per transaction by 10%
  • Reduce carbon per transaction by 10%

We’ll be updating and reporting out on all of the numbers in the dashboard and on our progress toward our improvement goals each quarter. As we continue on our course to fine-tune our own technical infrastructure, we hope to encourage other companies – particularly those that deliver their services digitally – to explore the DSE methodology and see how it can help expose the business value and performance of their own technical infrastructure. And while each company is different in terms of the services they provide and the variables that can be tuned to balance the cost, performance and environmental impact of delivering those services, DSE provides a framework for how other companies can begin.

Our DSE solution paper is just the beginning of what we hope will be an exciting new chapter in how we, as an industry, think about measurement and value. By sharing our methodology and progress, we at eBay Inc. hope to start that conversation. And as other companies are inspired to make their technical infrastructure as efficient and productive as possible, together we as an industry can drive the digital economy to even greater heights.

To comment or continue the conversation, feel free to email dse@ebay.com.

{ 2 comments }

Complex ideas can be conveyed with just a single image. That’s the notion of the common expression, “A picture is worth a thousand words.”

To provide rich user experiences, today’s web sites are incorporating lots of images. In most cases, these images are high resolution and have larger dimensions than in the recent past. Images contribute more to page weight than any other resource; according to httparchive.org, they represent 62% of the weight on an average web page.

Of course, as the use of images has increased so has the challenge of serving them without adversely affecting the user experience. This post addresses some of the most effective techniques for optimizing the perceived and real load times of image-rich web pages.

WebP

WebP is a new image format from Google with a 25-34% smaller file size compared to JPEG images. Currently, Chrome and Opera are the only browsers that support WebP. According to wikipedia, Chrome has a 36% share and Opera a 1% share of browser usage.

This test from webpagetest.org compares the page load time of WebP vs. JPEG. The test has one page with 50 images in the WebP format, and another page with the same 50 images in the JPEG format. Because the WebP page had to download fewer bytes (474484 vs. 757228), it completes loading much earlier compared to the JPEG page.

If you track your web site’s browser usage stats and find that Chrome/Opera users are a sizable chunk, using WebP images will improve the page load time for these users.

Progressive JPEG

A simple or “baseline” JPEG file is stored as one top-to-bottom scan of the image—whereas the progressive JPEG format divides the file into a series of scans. When a progressive JPEG loads, the user sees a low-quality image at first, but gradually sees the image at its full quality. Most of the latest browsers support progressive JPEGs. They improve the perceived loading speed when compared to baseline JPEGs, as this webpagetest.org test demonstrates.

Try progressive JPEGs for your above-the-fold images, and do A/B testing to see the impact on your site’s user experience.

Lazy loading

The most popular image optimization technique, widely used across the web, is lazy loading. This technique involves loading only a subset of the images initially, and loading the rest of the images on the scroll event or after the window on-load event. Using the on-scroll event to lazy-load images will significantly improve the page load time.

This technique might require the use of JavaScript. The only issue I see with lazy loading is that if the user loads a page through the browser’s back or forward button, JavaScript usage might cause some lag before the lazy-loaded images appear.

Image embedding (Data URI / Base64)

Each HTTP request exacts a price. Another popular technique for improving page load time is to minimize the number of HTTP requests. To improve the perceived load time, you can embed above-the-fold images using the Data URI scheme (base64). With this technique, images can be delivered as part of the page’s main payload; the HTTP request can be eliminated completely. Consider, for example, this <IMG> tag, which uses an HTTP request to display Right arrow (a right arrow):

<IMG src="http://pics.ebaystatic.com/aw/pics/help/infohubs/images/imgNaviArrowRgt_12x12.gif">

The following <IMG> tag uses the Data URI scheme to embed the image in the page:

<IMG src="data:image/gif;base64,R0lGODlhDAAMANUmACxw0RZVtClvzhxevA1OqApFoQQ/mxFMqARBnApFnxVWsipxzRZXsyFmwRxfugVIoSNlwx1fvStv0BZWsAA9lAA8lApHoA9PpwNAmSduyilsxx5dvCRlwxxevhRXsiNkwidsyRFOqSFmxQ5Nqidsyxpfuv///wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACH5BAEAACYALAAAAAAMAAwAAAY6QJPQJBkajQDB8ShYAJZDUkYDgpo+EE5DBO2UNgNHZKnwBBiBwXIUOhAmUMujcLFiEAmrqWLQmyhQQQA7">

The Data URI approach has two drawbacks:  the main payload of the page is increased, and the images aren’t cached by the browser. To address the caching issue, you could deliver Data URIs via an external JavaScript or CSS file; but you might not see the same perceived speed improvement that you would if you embedded the images directly in the page.

Sprites

Sprites are another way to minimize the number of HTTP requests. With this technique, you “stitch” together different smaller images and display the stitched image on the page, using CSS background styling to show different parts of the image. Sprites are easier to implement in cases where fairly static sets of images are to be displayed. Applying this technique to dynamically constructed pages—such as a search results page, where images to be displayed can vary substantially over time—could be challenging. The stitching process would need to be on demand and very fast. Moreover, if you’re using a content delivery network (CDN), you might lose the CDN benefits, because the stitched images might not be in the CDN cache servers and so the requests would come to your app servers instead.

CDN

By serving all static resources for a web page through a CDN, you can significantly reduce the actual page load times. Instead of the static content being requested from your servers, it would be delivered from CDN cache servers that are closer to the end user. Using the same images across all of your web pages would increase the CDN cache hits.

Domain sharding

Traditionally, web browsers place a limit on the number of simultaneous connections they can make to one domain. To increase the number of simultaneous connections, you can use multiple domains to serve the resources. You would have to pay for extra DNS lookup time for these extra domains, but the benefits outweigh this overhead. Domain sharding is applicable to all external resources—not just the images.

For mobile use cases, domain sharding might not be optimal, because the connection overhead for mobile browsers exceeds the benefit of having additional simultaneous connections.

Image dimensions

Specifying the width and height of every image in the <IMG> tag or in the CSS avoids unnecessary reflows and repaints. For example:

<IMG src=” http://thumbs.ebaystatic.com/d/l225/m/mHpX0W1OpQlEaydbGwCWuOg.jpg” width=”208px” height=”126px”>

This practice improves the rendering time and provides the user with a much smoother rendering experience.

Image scaling

If you need a smaller image, scale the bigger image and create the smaller one. Avoid using width and height specifications to scale images on the fly—a practice that would require downloading unnecessary extra bytes, thereby increasing page loading time.

Here is an example of poor practice:

<IMG src=”bigimage_width300_height300.gif” width=”100px” height=”100px”>

Instead, generate a 100 x 100 image, and use it:

<IMG src=”resized_bigimage_width100_height100.gif” width=”100px” height=”100px”>

The generated image will be smaller in size.

Image pre-fetching

If you can guess the user’s next move, you can pre-fetch some images (maybe the above-the-fold images) for the next page in the idle time or after the current page is loaded. This technique would load the next page faster, since the pre-fetched images would come from the browser’s cache.

Conclusion

To provide rich content, images are king. The optimization techniques discussed in this post are widely used across the web. By adopting all or some of them, you can substantially improve the perceived and real page load times for your site. Personally, I am thrilled about WebP and progressive JPEGs, and I hope more browser providers will come on board to support WebP in the future.

{ 7 comments }

Becoming a World-Class Tester

by Ilari Henrik Aegerter on 01/31/2013

in Testing

 A tester is somebody who knows that things can be different.

That is Jerry Weinberg’s description of a world-class tester, which reminds us that testing is about critical thinking and imagination. It is about not believing anything at face value, but maintaining an investigative mindset.

Testing is questioning a product in order to evaluate it.

That definition of the activity of software testing, from James Bach, includes one of the most important aspects of testing: information discovery and delivery. Software testing is about revealing the unknown.

If quality means value to some person who matters (as Weinberg has defined it), then testing is in the service of discovering anything that threatens this value. To provide this service, a tester needs to be able to do a variety of things, such as asking good questions, commenting on requirements, and interacting with a product in such a way that it yields information that’s useful.

My definition of a world-class tester is a person who is able to rapidly discover highly relevant information about the product, who makes the most use of any resource that is available to him/her, and who has the respect of people involved in a project. It is a person who can be trusted.

So, given these definitions, what skill and mindset dimensions should you invest in to become a world-class tester? What areas do you need to be good at in order to avoid being discredited as a “mouse-click-monkey”?

Technical proficiency

At the higher levels of management, software testing is often seen as only a technical discipline. They think your effectiveness as a tester depends on your ability to write code that does the testing for you. But there are many aspects of software testing that are simply not automatable. Humans are incredibly skilled at thinking and adapting to new information that arises based on context, and since software is a system of ever-changing functions and states, sometimes it is more efficient to do exploration. Automation is the opposite of that, since it involves scripted procedures.

However, automation can free the time of testers to do more exploration, which allows discovery. Anything that is either highly repetitive/boring or that needs to be run over and over again, is a good candidate for automation. Automation is very valuable in these areas.

Developing powerful and maintainable automation is difficult. I am convinced that a mediocre software developer is not able to live up to the requirements for such a job. If you want to do useful test automation, you’d better be an excellent developer in addition to having other testing skills.

Here are some more skills and mindsets that I believe will help you:

Investigative curiosity

If you think that testing is a necessary evil – something to get out of the way until you can do something really cool – you are in danger of missing important problems. World-class testers are constantly exploring the capabilities and limitations of their products while maintaining a healthy skepticism about those products’ capabilities.

If you want to give yourself a fancy title, think of yourself as a “compulsive empiricist,” guided by irresistible impulses to find the truth about what’s happening. World-class testing means following the urge to get to the bottom of things, not giving up until you have experienced enough value for yourself. When you see a tester shying away from unverified assumptions, that’s a good indication they take their job seriously.

Observational skills

If you want to observe something, you need to notice it first. Observation is the ability to notice important or significant details. While testing the product you need to be able to observe situations that are unusual.

Who would be a good teacher in the domain of observation? Go for a walk with a five-year-old and see what grabs his/her attention: a little crack in the pavement, a beautiful flower in a garden, a snail slowly making its way on a wall. Children are natural observers; they are interested in everything that happens around them. Their channels are wide open and they observe with joy.

Condition yourself to keep your channels open. Emulate child-like behavior. Train yourself in general awareness. Look closer. Slow down. Observe.

Communication skills

You will encounter circumstances where you have to defend your findings. You will have to convince people to take action, so you’d better know how to explain why your findings are relevant.

World-class testers actually practice describing their testing. They know their explanations need to go way beyond “Yeah, I tested it, trust me.” If you can describe your testing, your credibility as someone who knows what he/she is doing will grow. One way to get better is to use the Heuristic Test Strategy Model when you explain what you are doing. Or, even better, you will come up with your own models.

As you tell your story, remember that communication is a two-way street. World-class testers follow up on answers they get from their questions and are alert to new information that comes up in conversation.

Writing skills and bug advocacy

The richest communication happens when you are able to talk with somebody face to face. You observe facial expressions, you can react to indicators of confusion, and misunderstandings can be clarified right away. Not so with writing. The channel is narrower and more prone to misunderstandings. Clarifications need more time and effort. 

Bug advocacy often goes along with writing skills. Your ability to describe the problem accurately and understandably influences the probability of the right bugs getting fixed. Not every reported bug should be fixed, but the ones that should had better not escape because of poor wording.

A bug report is a persuasive document that needs to influence the right people to act. Free your bug reports from irrelevant information and don’t make them come across like a tabloid headline screaming at the reader.

You might want to keep a record of your past bug reports. Which ones produced confusion? Could you have written them differently? Would a change of wording have resulted in better understanding?

Domain knowledge

As a tester you are not just comparing specifications with the implementation. There are many other references, such as user expectation, consistency with comparable products, or the history of your product, that you can use to identify problems. Domain knowledge is also one of the references you can develop. Talk to your customers. Have a chat with the marketing guys. Spend a day with Customer Support. The better you know what is important for your business, the better your judgment will be.

Willingness to learn

You are a knowledge worker. Knowledge is not static, especially not in the technical domain. Constant learning is essential in order to become better at what you do. A situation where humans are interacting with software is a situation where software is interacting with humans. It is helpful to try to understand how both work. I recommend exploring the social sciences (cognitive psychology, social interaction theories, etc.) as well as increasing your technical education.

In my opinion, every tester who wants to become really good will profit from participating in the excellent BBST (Black Box Software Testing) courses. Each course takes four weeks, and you will spend 15-25 hours each week on intensive hands-on exercises. You can find more information about these courses on the website of the Association for Software Testing.

Another course I highly recommend is RST (Rapid Software Testing), which focuses on context, exploration, and thinking to find better bugs. It is currently taught by James Bach, Michael Bolton, and Paul Holland. A skillful search on the engine of your choice will lead you to the right place.

Social skills

Testers who constantly whine about their job and how horrible everybody else is get my blood boiling. Self-victimization is poisonous. Sometimes you just have to let go. If you cannot do that, you will be perceived as a sourpuss. If the crowd goes for a Friday evening beer, you won’t be invited. Imagine what influence that has on people’s willingness to collaborate with you.

Humor

Humor helps. Among many other things, it helps to sustain your sanity. As a tester, you often live in a stressful environment. Smiling and laughter reduce the damaging effects of stress hormones. It helps you to stay focused on what you want to be doing: testing.

Practice

And above all, you need to practice your skills. If you don’t practice, you can’t reach the higher levels of proficiency. While you practice, observe yourself, find areas for improvement, and make all of your actions deliberate and self-reflective.

Conclusion

Aiming to become a world-class tester does not mean reaching a destination, but rather being on a constant journey into new territories. The exploration never ends–which is exhilarating, and makes software testing so rewarding.

Happy testing!

{ 19 comments }

Until fairly recently, a single point of failure (SPOF)—a condition that can bring down an entire system—has been predominantly associated with the backend world of server clusters, networking, and operations. However, with the increase in third-party widgets and the complexity of modern web applications, the web community is increasingly aware that a frontend SPOF can also bring down an entire site. As a result of a frontend SPOF, the user sees an entirely or partially blank page, or experiences delayed rendering of the page.

This post describes a tool developed at eBay, called SPOFCheck, which has been open-sourced and is being integrated into engineering life cycles not only at eBay but globally.

Background

The awareness of frontend SPOF has increased tremendously among engineers, thanks to Steve Souders’ initial research on this topic, as well as recent articles and blogs emphasizing its importance; see, for example, Bjorn Kaiser’s post about third-party content causing SPOFs. Numerous utilities and plugins exist that can detect possible SPOF vulnerabilities in a web application, most notably webpagetest.org, the Chrome plugin SPOF-O-Matic, and the YSlow 3PO extension.

At eBay, we wanted to detect SPOF at a very early stage, during the development lifecycle itself. Such early detection would require an additional hook in our automated testing pipeline. The solution we developed and open-sourced is SPOFCheck, a simple tool that works on our test URLs and produces SPOF alerts. The tool is integrated with our secondary jobs, which run daily automations on a testing server where a development branch is deployed. When SPOFCheck detects possible vulnerabilities, engineers receive SPOF alerts and act accordingly. As a result, SPOFs can be contained within the development cycle, rather than sneaking into staging or production.

Command-line interface

SPOFCheck is a command-line interface (CLI) that runs on Node.js. Source code, syntax, and usage documentation are available in GitHub. Running SPOFCheck involves simply specifying the output directory. Command-line options include output format (XML that most CI servers can parse, XML to be consumed by other utilities, or text), console-only output, and the rules (checks) to be run. By default, SPOFCheck runs the following rules:

  1. Always load third-party external scripts asynchronously in a non-blocking pattern.
  2. Load application JS in a non-blocking pattern or towards the end of the page.
  3. Attempt to inline the @font-face style; ensure that the font files are compressed and cacheable.
  4. Ensure that the font files are compressed, cached, and small in size.
  5. Ensure that inlined @font-face styles are not preceded by a SCRIPT tag (which causes SPOFs in Internet Explorer).

Extensibility

New rules can be easily added, either by pushing them to the rules array or by calling SPOFCheck’s registerRules function.

At eBay, we’ve built a thin web client to provide SPOFCheck as a service. eBay engineers can create multiple SPOF projects, which can be triggered on demand or configured to run on a recurring schedule. Project owners designate the teams or individuals who receive SPOFCheck results.

Acknowledgements

Thanks to Steve Souders, whose work (cited above) is the source of SPOFCheck’s default rules.

SPOFCheck reuses code logic from the GitHub projects spof-o-matic and 3po. The design and packaging of the tool is based on csslint, thanks to Nicholas Zakas and Nicole Sullivan.

Thanks also to perfplanet, who featured SPOFCheck in their 2012 performance calendar.

{ 2 comments }

Tomcat/Jasper Performance Tuning

by Sheldon Shao on 01/04/2013

in Software Engineering

JavaServerPages (JSP) is productivity-enhancing technology for building dynamic web pages. With more than 10 years of history, it is popular and mature. JSP 2.0 includes new features that make the code cleaner, such as the Expression Language (EL), the tag file, and simple tags. However, these new features can cause some performance issues.

eBay uses JSP technology to build web pages. More specifically, we use Geronimo 3 as the web container and Geronimo’s JSP engine, Tomcat/Jasper. To handle eBay’s huge volume of requests, the JSP engine must be powerful and must provide exceptionally high performance. Because Tomcat/Jasper shows some weakness in performance when using hundreds of EL expressions and tags, we have focused much effort on profiling and optimizing Jasper to make JSP more efficient. Most of the performance solutions and patches that we have submitted to the Tomcat community have been accepted. As a result, Tomcat users will soon be able to benefit from our work.

This blog post discusses the main issues we have identified and addressed.

EL resolving

When a JSP page includes a large number of EL expressions, EL resolving becomes a bottleneck for page rendering. The following JProfiler screen shot shows a case where 40% of the Java processing time is for EL resolving; the whole JSP page costs 1,865 ms, while the EL resolving costs 722 ms.

This profiling enables us to identify the heavy methods from Jasper:

 

Looking into the code, we found that every expression node needs to call CompositeELResolver.getValue when trying to resolve the value. The bottleneck is that CompositeELResolver contains at least seven EL resolvers:  ImplicitObjectELResolver, MapELResolver, ResourceBundleELResolver, ListELResolver, ArrayELResolver, BeanELResolver, and ScopedAttributeELResolver. The method getValue tries to return the value from each of these resolvers one by one, until one of them returns a not-null value. This logic results in many unnecessary resolver calls.

We proposed an enhancement:  a customized CompositedELResolver in Jasper that avoids many unused resolver calls. This change, accepted in Tomcat 7.0.33, saves 10% of the EL resolving time. The Bugzilla URL for this issue is https://issues.apache.org/bugzilla/show_bug.cgi?id=53896 (The issue was filed by the Geronimo team, who tried to resolve the issue for us).

We’ve filed some related Tomcat bugs as well, such as Bug 53867 (also accepted in Tomcat 7.0.33) and Bug 53869.

JSP compilation

JSP compilation is a very heavy process that includes JSP reading, JSP parsing, Java generation and compilation, and class loading. We have achieved many optimizations in this logic as well.

JspReader was the focus of our initial performance-tuning work related to compilation. JspReader.mark is called too many times during JSP and tag files compilation. The following screen shot illustrates the excessive number of JspReader.mark invocations and the time they consume.

 

By using the JspReader buffer to reduce the number of invocations, JSP reading performance can improve fourfold. Tomcat 7.0.31 includes this enhancement; for more details, see https://issues.apache.org/bugzilla/show_bug.cgi?id=53713.

EL code generation

Even with multiple optimizations, EL evaluation is still a big hotspot. It is not only time-intensive but also CPU-expensive. EL resolving is the root cause. Therefore, if we can bypass EL resolving in some cases, we can improve performance.

The idea is to do code generation for simple EL expressions, such as ${model}, ${model.test}, ${empty model}, and ${model.value++}. Such expressions can be interpreted as simple Java code. We have identified three kinds of EL expressions that can use code generation:

  • Simple EL expression containing only one part of the expression — for example, ${model}. Here is an example of the generated code for this kind of expression :
this.getJspContext().findAttribute("model")
  • EL expression with two nodes where the type of the first node is specified by a tag file attribute—for example, ${model.location}, where model is specified by the following directive in the tag file:
<%@ attribute name="model" required="true" type="com.ebay.raptor.search.pres.common.model.results.ItemModel" %>

   Here is the generated code for this example:

(getModel() != null ? getModel().getLocation() : null)
  • EL expression with logic or arithmetic where the value part can be generated. For example:
${(intlExpansion eq 'true' && not empty model.location) || sortType==7}

   The generated code for this example looks like this:

(org.apache.jasper.runtime.ELRuntimeUtil.equals(getIntlExpansion(), "true")&&(! org.apache.jasper.runtime.ELRuntimeUtil.isEmpty((getModel() != null ? getModel().getLocation() : null))))

We have proposed an extensible ELInterpreter solution to Tomcat and implemented an interpreter for these three kinds of EL expressions. After the extensible interpreter has been accepted, we will contribute the code to Tomcat as well. This customized interpreter improves total page performance by 10%. Application developers will be able to inject their own interpreter to replace the default JSPUtil.interpreterCall.

For more about this enhancement, see https://issues.apache.org/bugzilla/show_bug.cgi?id=54239

JSTL code generation

Tag library performance is another hotspot for page rendering. Among the problems are the numerous method calls in the standard generated code and the high number of context switches. Every tag needs to call doStartTag and doEndTag.

Jasper has a code-generation solution for JSTL, which is based on the TagPlugin interface. Although eBay leverages the implementations in Jasper, there are many bugs. Here are some of the fixes we’ve contributed:

Issue

Bug

Status

Fixed version

Bug in JSTL tagPlugin “Out”

Bug 54011

Fixed

7.0.33

Bug in JSTL tagPlugin “Set”

Bug 54012

Fixed

7.0.33

Configurable tagPlugins.xml

Bug 54240

In-Progress

 

NPE in BodyContentImpl

Bug 54241

In-Progress

 

Bug in ForEach

Bug 54242

In-Progress

 

Bug in When and Choose

Bug 54314

In-Progress

 

Summary

Tomcat/Jasper is eBay’s chosen JSP solution. However, few solutions can fully satisfy eBay’s daunting performance requirements, and Tomcat/Jasper is no exception. Our efforts have identified four major performance issues:  EL resolution, JSP compilation, EL code generation, and JSTL code generation. Working with the Tomcat community, we’ve made substantial progress on some of these issues, and more enhancements and bug fixes are underway.

{ 4 comments }

Copyright © 2011 eBay Inc. All Rights Reserved - User Agreement - Privacy Policy - Comment Policy