The eBay Motors engineering team took up an initiative to revisit some of the legacy JavaScript code shared across various pages, optimizing them to leverage the latest advancements in HTML5. Our main focus areas were user interactions and animations in which the old JavaScript code was lacking in performance and sturdiness. When we completed the exercise and demoed the upgraded experience to our product folks, the feedback we got was “SLEEK“. This post highlights five of those JavaScript techniques that we think made a difference.

1. requestAnimationFrame over setInterval: Using the new requestAnimationFrame API for building JavaScript-based animations is a more optimized and efficient approach when compared to the traditional timers and intervals. To quickly summarize, this API offloads the timer calculations for when to do the next UI/DOM style changes to the browser, rather than the developer deciding when to repaint the screen. We grepped our code base and replaced all applicable occurrences of setTimeout and setInterval with requestAnimationFrame; as a result, the animations were a lot smoother. This approach also saves battery life when content is viewed on mobile devices. As expected, the requestAnimationFrame API is not supported in all browsers, so we used polyfill, provided by Paul Irish, as shown below.

    // shim layer with setTimeout fallback
    window.requestAnimFrame = (function(w){
      return  w.requestAnimationFrame       ||
              w.webkitRequestAnimationFrame ||
              w.mozRequestAnimationFrame    ||
              w.oRequestAnimationFrame      ||
              w.msRequestAnimationFrame     ||
              function( callback ){
                w.setTimeout(callback, 1000 / 60);
              };
    })(window);

2. insertAdjacentHTML over innerHTML: The insertAdjacentHTML API is a fine-grained and optimized version of the super-popular innerHTML. Since we specify the insert position, insertAdjacentHTML does not re-parse the element it is being used on and avoids the extra step of serialization, making it much faster than direct innerHTML manipulation. This approach is very effective in scenarios where we keep appending markup to a page or module, such as for the daily deals feed and for endless scroll. A simple JSPerf test result shows that insertAdjacentHTML is 100% faster than innerHTML. Surprisingly, the browser support for this API has been there for a very long time (Firefox started supporting it in version 8), and the helper function is pretty straightforward:

    // helper function to append content for a given element
    var appendContent = function(){
        // Closure to hold the insertAdjacentHTML API support
        var insertAdjacentSupported = document.createElement('div').insertAdjacentHTML;

        return function(elem, content) {
            if(insertAdjacentSupported) {
                elem.insertAdjacentHTML('beforeend', content);
            } else {
                elem.innerHTML = elem.innerHTML + content;
            }
        };
    }();

3. if-else over try-catch: Try-catch blocks provide an efficient mechanism to handle exceptions and unforeseen runtime errors. However, they impose a performance penalty in JavaScript, especially when used in iterations or recursive functions. More details can be found at dev.opera.com and O’Reilly’s site. To address this issue, we scanned our code base, particularly looking for performance-critical functions. We found a few occurrences where a try-catch was inside a long-running iterator loop, and also where the probability of the thread entering the catch block was high. The try-catch was replaced with simple if-else conditions, which took care of all error handling, and the resulting code was much more efficient.

Using try-catch:

    // jsonResponse comes from some web service
    var i, l = jsonResponse.length, item, offer, binPrice, bidPrice;
    for(i = 0; i < l; i++) {
        item = jsonResponse[i];
        // Some code
        // ...
        try {
            offer = item.offer;
            binPrice = offer.bin;
            bidPrice = offer.bid;
        } catch(e) {
            offer = {err: "Offer not found"};
        }
        // Some more code
        // ...
    }

Using if-else:

    // jsonResponse comes from some web service
    var i, l = jsonResponse.length, item, offer, binPrice, bidPrice;
    for(i = 0; i < l; i++) {
        item = jsonResponse[i];
        // Some code
        // ...
        offer = item.offer;
        if(offer) {
            binPrice = offer.bin;
            bidPrice = offer.bid;
        } else {
            offer = {err: "Offer not found"};
        }
        // Some more code
        // ...
    }

4. XMLHttpRequestUpload over server polling: Using browsers to upload files (photos, PDFs, other documents, etc.) has become a very common use case, and one of our applications had this requirement. To keep users informed during the upload process, we wanted to show a real-time progress meter with accurate percentages. One (non-Flash) way of simulating this AJAX-based upload is to create a hidden iFrame and submit the main form, which holds the input file element targeted to the iFrame. Following the form submit, the server has to be polled periodically to retrieve the upload percentage for the progress meter. Not only is this approach a hack, but it also consumes huge amounts of server and client resources.
As a savior, the XMLHttpRequest object in modern browsers has the capability of uploading files (as byte streams) using the send method, and also has this amazing XMLHttpRequestUpload attribute. The upload attribute has a couple of associated events, the most important being the progress event. The progress event handler receives the total number of bytes to transfer and the number of bytes transferred so far, from the event's total and loaded fields. With this information, end users receive updates of real-time progress in the most efficient and sturdy way. Here is a quick preview of the API usage:

    var uploadAJAX = function(serverURL) {
        var xhr = new XMLHttpRequest(),
            fileElement = document.getElementById("file"),
            fileObj = fileElement[0], // For demo, taking only the first file in the file list
            progressMeter = getProgressMeterComponent(); // retrieve the progress meter UI component 

        xhr.upload.onprogress = function(e){
            if (e.lengthComputable){
                var percentComplete = Math.round(e.loaded / e.total * 100);
                progressMeter.update(percentComplete); // Updates the progress meter UI component with the given percentage
                }
        };    

        xhr.open("POST", serverURL, true);
        xhr.setRequestHeader("X-Requested-With", "XMLHttpRequest");
        xhr.setRequestHeader("X-File-Name", encodeURIComponent(fileObj.name));
        xhr.setRequestHeader("Content-Type", "application/octet-stream");
        xhr.send(fileObj.file);
    };

We have open-sourced the HTML5 image uploader utility in github. For browsers that do not support this feature, the application falls back to the hidden iFrame approach.

5. CSS3 over JavaScript: The final optimization is to AVOID JavaScript as much as possible for animations (JavaScript is great but...) and to leverage the modern CSS3-based transitions. The new-age CSS comes with a ton of great animatable properties that can replace most of the basic animations currently implemented in JavaScript. The main advantages are ease of use, the browser doing most of the work, the leveraging of machine hardware if necessary, and above all an elegant and smooth visual touch to the transition (which is nearly impossible to achieve with JavaScript). We started changing our animations from simple tab switches (check it out) to complex 3D carousels (check out the beta version) - all with CSS, which netted a great user experience. For older browsers, we just stopped doing animations.

This entire re-engineering process also helped us develop a workflow for similar upgrades, thus enabling our code base to iterate at the same speed as do browser innovations. We can always hope that the need to use polyfills will be reduced in the near future.

Senthil Padmanabhan
Engineer @ eBay

Mark Twain is quoted as having said, “Some German words are so long that they have a perspective.”

Although eBay users are unlikely to search using fearsome beasts like “rindfleischetikettierungsüberwachungsaufgabenübertragungsgesetz”, which stands for the “beef labeling supervision duties delegation law”, we do frequently see compound words in our users’ queries. While some might look for “damenlederhose”, others might be searching for the same thing (women’s leather pants) using the decompounded forms “damen lederhose” or “damen leder hose”. And even though a German teacher would tell you only “damenlederhose” or “damen lederhose” are correct, the users’ expectation is to see the same results regardless of which form is used.

This scenario exists on the seller side as well. That is, people selling on eBay might describe their item using one or more of these forms. In such cases, what should a search engine do? While the problem might seem simple at first, German word-breaking – or decompounding, as it is also known – is not so simple.

How to find compound synonyms?

Writing a program to figure out how to break a German compound word for one of the world’s largest online marketplaces poses challenges:

• The word could have been created using a product name, and new product names can appear all the time.
• Syntactically, the word can be split into many valid components, but not all splits are useful or make sense. For instance, “aktivkohlefiltermagnetventil”, meaning “active carbon filter magnetic valve” (a valve used in water treatment plants), can be split at least three different ways, all of which are comprised of valid dictionary words:
  • aktiv (active) kohlefilter (carbon filter) magnetventil (magnetic valve) or
  • aktiv (active) kohle (carbon) filter (filter) magnetventil (magnetic valve) or
  • aktiv (active) kohle (carbon) filter (filter) magnet (magnetic) ventil (valve)

  
  However, the only acceptable split that is synonymous to the compound word “aktivkohlefiltermagnetventil” is “aktivkohlefilter magnetventil”.

• Some strings look like they might be compound words, but in fact are not. For instance, a tokenizer might look at the word “beiden” (“both”) and decide it’s made up of the two separate words “bei” (“at”) and “den” (“the”), but that would be incorrect.
• There is also the problem of how small or large each of the splits can be. Consider the three forms below; which are most useful?
• granitpflastersteine (granite paving stones)
• granit pflastersteine (granite cobblestones)
• granit pflaster steine (granite paving stones)
• A smaller concern is the cost of processing possible splits online, i.e., after the user types in her search query.

Dictionary-based approaches have a tendency to split German compound words into all possible morphologically valid forms. The best splits are hard to algorithmically determine.

Wisdom of the crowds

Faced with these challenges, we turned to our users for help. We can learn a lot simply by watching millions of German users listing millions of items, and performing hundreds of millions of searches. Essentially, whenever users changed from a compound to a decompounded form of a query (or vice versa), we asked, did their subsequent actions on our site indicate that they were pleased with the change? First, we mined our logs to find instances where users changed their search terms by breaking up or joining consecutive adjacent terms. For instance, people typed in “damen schuhe” and then changed it to “damenschuhe”. We analyzed how many and what kind of results we were showing for either search, and how those users liked those results (based on whether they clicked on any of those results). Based on how frequently users were making such changes in their search terms, we were able to collect millions of candidate compound words and their decompounded forms. We did the same for item titles, as sellers often include multiple forms of compound words in order to make sure buyers can find their items. To ensure a good user experience, we then filtered out words without relevant inventory, model numbers, and single character splits. This whittled the set of synonymous compound word pairs to a few hundred thousand.

Since we now had a large number of compound words and their decompounded forms, we were able to determine if a user’s query matched items listed with any of its compounded or decompounded forms. Since the compound words are precomputed, the lookups are blazingly fast. Further, the list of words is based on real-live user behavior data, which allows us to use the most “sensible” compounds/decompounds.

The bottom line

Being able to find items listed with all common compounded/decompounded word forms is valuable and easy to measure. We simply split our user population in two. We provide the existing search experience to some users, and the new experience to the others. The new compound word synonym experience, although largely unnoticed by most users, was indeed improved, enabling buyers to find – and subsequently buy – more of the items they were looking for. This was measurable in terms of direct revenue. In this case, buyer, seller, and eBay interests are well aligned. Buyers bought more, sellers sold more, and eBay revenue increased.

Brian Johnson, Prathyusha Senthil Kumar, Ashok Mallya

eBay has been exploring the potential benefits of incorporating peer-to-peer transport technologies into our network topology. We’ve begun to use BitTorrent for index distribution, and we’ll soon use it for build distribution as well. In this post, we discuss our experience with BitTorrent and the problem of package distribution within the enterprise environment. Typical enterprise requirements are for replicating packages of medium to large size (a few megabtyes to several gigabytes) across a large number of target nodes, with high transfer reliability and often within stringent time lines. Use cases having such requirements include the following:

• Rollout of versioned software packages
• Fan-out of data packages across distributed grids for downstream processing

About BitTorrent

BitTorrent (BT) is a popular peer-to-peer (P2P) transport protocol that is commonly used across the Internet to transfer packages such as audio, video files, and data sets. According to a comprehensive 2008-2009 study of Internet traffic in eight regions of the world, BitTorrent traffic represents not only a lion’s share of all P2P traffic, but also a significant portion (20-30% or more) of all Internet traffic. BT works well due to the distributed nature of the transfers among the nodes; since a single server, or a few servers, do not need to transfer to all nodes across the grid, BT is efficient for distributing large packages to a large number of nodes.

BT splits the packages into pieces that are shared between the peers. A tracker maintains and provides a list of peers that are interested in a particular package. A peer first acts as a client. Once it receives a full piece, it acts as a server for that piece. The peer discovers other peers via one or more trackers. Information about trackers, pieces, and their checksums is stored within a “.torrent” file, which is typically a fraction of the overall file size. Before BT transfers can be initiated, the “.torrent” file is distributed to the peers using HTTP, SCP, or other client/server means. The BT specification and protocol details are described at theory.org.

The P2P technique, particularly the BitTorrent protocol, is well studied. Two key strategies for efficient BT content replication have been found to be piece selection and peer selection (see the 2006 paper, Rarest First and Choke Algorithms Are Enough). BT uses rarest first as the piece selection strategy, and the choke algorithm as the peer selection strategy. The rarest first strategy, as the name suggests, involves peers targeting the rarest piece for transfer. The choke strategy is based on fairness, speed, and reciprocation of uploads.

The content distribution problem has also been studied from a theoretical perspective (see Optimal Scheduling of Peer-to-Peer File Dissemination). Optimal distribution choices, comparing performance using a client/server mechanism versus P2P transfers, are described below.

where:

N = Number of nodes
R = Number of pieces that the package is split into
t = Time to transfer a complete package from one node to another

BitTorrent within the enterprise environment

BitTorrent has been used to a limited extent in large-scale web services to distribute files. Facebook uses it for file sharing, and Twitter for code deployments.

The BitTorrent protocol is designed for Internet scale with slow, unreliable, far-away and greedy clients in the presence of ISP throttling. BT-style distributions are desirable in enterprise environments, where large data sets of increasing size need to be transferred to a large set of nodes. In some ways, the problem of data transfers (fan-out) within a data center is simpler with reliable, managed, fast, and low-latency nodes. However, the demands of predictability, scalability, manageability, and efficiency are higher in such environments.

We now turn to optimization options, on top of regular BT transfers, for improving efficiency and predictability

Configuration

The general BitTorrent configuration is designed for Internet scale with slow and unreliable clients. This configuration can be adapted to work better in the enterprise environment with homogenous clients. The upload and download parameters and piece sizes can be adapted to improve transfer rates and reliability in an enterprise environment. The tit-for-tat strategy as a part of the choke algorithm, throttling, and encryption of payload may not be needed in enterprise environments.

Peer selection

When distributing pieces between peers, it is efficient to pick peer nodes the closest to each other. In enterprise environments, the server topology may be available beforehand and easily leveraged in the algorithm to pick nearer neighbors. We find that closeness of IP address can often be used as a reasonable approximation.

Cross-data-center transfers

When distribution topology extends across multiple data centers, it is useful to limit cross-data-center transfers due to bandwidth constraints–limited and/or expensive bandwidth.

A naïve approach to reducing cross-data-center transfers is to perform them in two steps. The first step is to transfer a package to one or a few nodes in each data center. The second step is to initiate BT transfers within each data center such that package transfer takes place between the initial nodes and the rest of the nodes in that data center—in effect, one BT transfer session per data center. This two-step scheme increases the overall transfer time. Peer selection based on closeness of IP address can be an effective way of conducting transfers in a single step, with all nodes participating in the torrent session.

Package transfer steps

When the source package is in HTTP, HDFS, or custom storage, BT distribution requires sequential execution of steps as follows:

Before initiating the BT session to download a package, the additional steps that need to be executed are downloading from the package store, generating the “.torrent” file (assuming it was not previously generated), and seeding the package. Each step requires the sequential reading or writing of the data. It is common to find that the time taken by these steps is comparable to actual download time.

As multiples of these steps need to be performed in sequence, the best way to simplify the distribution is to avoid some of the steps altogether.

Web seeding

HTTP-based web servers are popularly used to host packages. Clients connect to one or a handful of these servers to download a package. Even when using BT, it is common for the initial “.torrent” file to be downloaded from an HTTP server. The “.torrent” file contains package meta-information, including the location of trackers and piece-by-piece checksums.

Web seeding is an attempt to host packages on web servers so that they can be used for seeding. This technique avoids the need for a separate BT seeding process, as web servers can double up as seeders for hosted content.

BitTorrent web seeding has two distinct and incompatible specifications. The BitTornado specification is based on a custom URL scheme to download specific pieces. The GetRight specification relies on the basic HTTP 1.1 download mechanism using byte serving (byte range queries).

We find that the HTTP byte-serving method is more convenient, as it simply requires hosting packages behind an HTTP 1.1-compatible server. We recommend hosting both the “.torrent” file and packages on the same server. The “.torrent” file can be created at the same time that the source package is generated, thereby avoiding the need for a separate step to create it later during BT transfer. In this scheme, BT seeders are not used; instead, BT clients use HTTP range queries to download pieces from a web server. Once one or a few peers have a piece, the distribution of that piece takes place between peers without a request to the HTTP server. Note that this scheme requires BT clients to support both the BT and HTTP protocols.

Here is a summary of the benefits of using web seeding:

• HTTP server-hosted packages can be used for both HTTP and BT transport. Web seeding allows the HTTP packages to be transferred using the BT protocol; when distributing to a few nodes, HTTP transport works well.
• BT seeding requires the additional step(s) of downloading the package (if the seeder is different from the HTTP package store), and then seeding it before the BT transfer can be initiated. As explained above, web seeding avoids these steps and saves time.
• When we have a large number of packages, any of which can be downloaded using BT, serving via BT seeding requires an always-on seeder—which means having a separate seeding process (a process or thread per package). Even when transfers are inactive, these processes or threads need to be running. By contrast, HTTP transport does not require any activity when package download is not occurring.
• With traditional BT seeding, the seeder periodically communicates with the tracker on its status and progress, so that tracker logs can be mined for network utilization, the time taken by nodes, and other package transfer details. If the BT seeding session is left active to support any future BT downloads, it continues communicating with the tracker whether or not there is any download activity. The result is communication overhead and cluttered logs. Web seeding avoids these disadvantages because it does not involve communicating with the tracker.

HDFS-based seeding

Hadoop and HDFS are becoming increasingly popular in enterprise environments to store large data files. In HDFS, large data files are split into several blocks, copies of which are stored on multiple nodes to provide reliability and fail-over capability (default block size is 64 MB).

Distribution for HDFS-hosted packages can borrow ideas from the web-seeding approach, as HDFS supports retrieval of arbitrary portions of packages. The “.torrent” file can be generated in a parallel manner, with piece checksums being computed as a map/reduce task. In order for the map/reduce task to work efficiently, the BT piece size can be chosen such that one or multiple BT pieces equate to a single HDFS block. This strategy helps with torrent generation and BT distribution, as BT pieces wouldn’t spawn across HDFS blocks.

The use of HDFS-based seeding has the following advantages (some of which are similar to those of web seeding):

• The source package may be stored across HDFS nodes. Downloading it on a single seeder node requires download and aggregation of HDFS blocks before BT seeding. This step can be avoided with HDFS-based seeding.
• BT seeding traverses through the entire package, validating checksums, before seeding. The HDFS-based seeding approach both avoids a sequential read of a large package and provides an always-on seeder.
• Using traditional BT seeding with large HDFS packages requires keeping multiple versions of the packages on seeder nodes—and consequently extra bookkeeping, space management, and cleanup. HDFS seeding avoids this overhead.

Dealing with deltas

When clients have an earlier version of a package with some overlapping content, efficiency improvements can be achieved by downloading less of that package. The rsync algorithm is one of the commonly used techniques for dealing with package deltas. This algorithm uses a combination of strong and weak checksums to determine the deltas. A modified scheme, called zsync, is used for delta computation in a distributed manner. The zsync scheme works nicely with HTTP 1.1 and byte range queries. We suggest a combination of BitTorrent and zsync as a way to transfer packages with some overlapping content.

As clients may have different versions of a package, each client identifies and reuses common parts of the package by employing small “.zsync” files that contain weak and strong checksums. When using zsync in combination with the BitTorrent protocol, a client can download the modified portion from other peers if peers contain the piece. If other peers do not contain the piece, the client can default to web seeding and use HTTP 1.1 byte range to download that piece from source.

The amount of overlap between package versions would depend on the package format as well as the changes compared to the earlier version of the package. The package producers are likely to know whether such overlap is expected. The presence of a “.zsync” file can indicate whether a previous version of the package should be scanned to determine overlap. In the absence of a “.zsync” file, the BT client can fall back to the case where no common parts are shared with the previous version.

BitTorrent test results

To optimize transfer time, we ran a series of experiments using a small cluster of 16 nodes. The results discussed below are captured in a data center environment with fast connectivity (1 Gbps) between nodes with low latency (less than a few milliseconds). We used BitTornado for our experiments. The peer nodes are homogenous, running the same BT client version with identical configurations.

This first table shows the actual and theoretical times taken to transfer a 30 GB package as the number of nodes is varied:

The theoretical times are computed using the formulas described earlier with t = 4 minutes and R = 3840. The actual time taken is the time for all nodes to receive the full package.

The theoretical client/server time shows a linear increase as the number of nodes is increased. The theoretical P2P best time is constant at 4 minutes. The data shows that the actual transfer times are better than the theoretical client/server times when the number of nodes increases beyond 3-4. While the actual time taken increases as the number of nodes increases, the rate of growth decreases.

The ratio between theoretical best and actual times indicates the efficiency in transfer compared to the theoretical best. The ratio, and thus efficiency, decreases as the number of nodes increases—going from 36% to 24% as the number of nodes increases from 1 to 16. The low efficiency suggests potential improvements with an optimized BitTorrent implementation or use of other P2P schemes.

The following table shows the time taken by the different steps in downloading and transferring the 30 GB package to a 16-node cluster:

The time for actual package transfer (step 4) is comparable to the combined time of the three prior steps. Web seeding or HDFS-based seeding would avoid steps 1 and 3. Step 2 can be performed during package creation or as a map/reduce task for HDFS packages. Because steps 1 through 3 are executed sequentially, HTTP- or HDFS-based seeding can achieve substantial gains for package transfer.

The following graph shows the amount of piece sharing that happens in the swarm of 16 nodes. The data depicted is captured from tracker logs at the end of a BT transfer.

The download figures are not very interesting, as each peer needs the entire package. Thus, download is 100% for peers while 0% for the seeder.

The amount of upload varies between the seeder and the peers. The seeder needs to upload all pieces at least once, and thus its upload would be 100% or more. The results show seeder upload of slightly more than 100%, while peer upload ranged from 75% to 125%. Thus, all peers participated in uploads such that the original seeder did not upload a disproportionately high amount of the package bytes.

Summary

For the package fan-out problem in enterprise environments, BitTorrent-based distribution is a good solution. The problem of package transfers is simpler in enterprise environments, with their fast, reliable networks and homogenous clients. Web seeding and HDFS-based seeding can be used to avoid some of the steps and increase the speed of package transfers. Deltas across package versions can be dealt with effectively by using an approach that combines zsync and BitTorrent mechanisms.

In the first part of this blog post, we outlined the problem of rapid development setup as well as the approaches that we have used at eBay to address the problem. We will now describe how we set up projects in the Eclipse IDE, which is well adopted at eBay for Java applications.

Let’s start by looking at two Eclipse project types – source and binary – before we describe dynamic projects, which convert between source and binary.

Source project

A source project is a standard Eclipse project created by clicking File -> New Project. A source project includes these characteristics:

• Source elements
• Dependencies to other projects or libraries (binary bundles)
• Builders that convert the source elements to binaries

With source projects, an SCM system typically provisions the source elements. Project metadata is created and optional pre-build steps are executed before compiling the source elements. The most time-consuming steps tend to be source downloads and compilation.

Binary project

The binary project is not a common Eclipse project type. Eclipse does not have a wizard to create a binary project, but it does provide this option in cases where an existing plugin or feature is imported into the workspace. A binary project has these characteristics:

• No source elements; instead, libraries with attached source
• Dependencies to other projects or libraries
• No-op builders and no source authorship

In most other ways, binary projects share the capabilities of source projects, including the ability to browse, debug, and execute code. As there are no source elements to provision and compile, the setup of binary projects is generally much quicker compared to setup of source projects.

Dynamic project (project level)

The advantage of a binary project is that there is no source to download or compile, as we consume compiled source; the obvious disadvantage is that it doesn’t provide the source authorship capability that is available with source projects. We therefore came up with a third type of project — what we call a dynamic project — which can convert a binary project into a source project and vice versa.

Let’s look at the workflow for a project-level conversion. The developer first sets up the environment with binary projects to browse, run, and debug the application. When ready to make changes to a project by editing the source, the developer simply selects the appropriate binary bundle and converts it to source.

The system fetches the source elements, compiles them, and brings them as a source project into the workspace. The system also adjusts dependencies, so that all other projects that were depending on the binary project now depend on the source project. Conversion from source to binary is accomplished in a reverse manner.

Dynamic project (source level)

At the project level, the dynamic project acts as either a binary or a source project. When we have a limited number of source elements, project-level conversion from binary to source works fine. In contrast, when a project contains a large number of elements but only a small set of those elements require authorship, conversion of individual source elements is useful. In this case, the project becomes a hybrid: it is neither a binary nor a source project. Only the source elements that need to be modified are converted from binary to source.

When the first source elements are converted, a source project is created. It contains only the converted source elements, with a dependency to the binary bundle. Due to specific tooling for dynamic projects, Eclipse exports the compiled .class elements before the libraries and binary bundles. The class loader picks the first matching class, which is the one from the source element in the workspace. In this manner, any source element modifications get reflected in the application.

Summary

Quick developer setup is very desirable yet challenging in environments with a large code base and a high velocity of changes. In web applications affording the flexibility of frequent rollouts, the number of feature releases is usually large (hundreds per month in eBay’s case), while the number of changes per feature is often limited (for eBay, hundreds of source elements).

Processes and tools help provide an environment that supports quick developer setup. We have described our journey in providing such an environment through the use of binary bundles, hierarchical repositories, and a combination of incremental source changes with binary bundles. We have described the challenges that these approaches pose as well as the approaches and ideas we have developed to overcome them.

Large development teams with a sizable code base have two conflicting needs: code sharing and rapid developer setup (agility). Teams who are working on different features need to modify common projects and source elements. While code ownership helps, members of the same team invariably need to touch the same components across different features – or even the same features! Additionally, as ownership itself changes over time, developers other than the original authors need to enhance features or resolve issues.

The problem facing large development teams stems from more than source-level conflicts and SCM code merges or promotions – which, in our experience, are well bounded. The more fundamental problem has to do with enabling teams to set up their development environments rapidly while working with a code base that is not only large but has a high velocity of changes.

Between the need for code sharing and the need for developer setup agility, code sharing often receives priority as it avoids duplicating code and reinventing the wheel. In this two-part post, we discuss our experience with providing rapid developer setup.

The size of the problem

We will first consider quantitative aspects of the problem before looking at our journey to solutions. The problem involves both the number of changes and the velocity of the changes. The eBay code base consists of hundreds of thousands of source elements (with total lines of code in the tens of millions). The applications – which comprise web, services, messaging, and batch style – number in the thousands. It is common for an application to use tens of thousands of source elements. The elements themselves are contributed by different teams and are shared with other applications.

Code base changes take the form of new features, which require modifying, adding, and (sometimes) removing source elements. eBay is a fast-paced environment with hundreds of teams; every month, we roll out hundreds of features. The following chart shows the number of source elements that features touch.

The X-axis is a log-like scale indicating the number of elements modified. The Y-axis shows the percentage of features matching each bucket. The data distribution is fairly normal, with a median of 100 to 200 source elements. Compared to the overall code base size, the per-feature change is less than 1% of applications. However, with hundreds of features rolled out per month, these changes compound quickly.

In addition, the code base dynamically changes across features. As modifications for individual features are validated and rolled out, later features pick up modifications from earlier features as they themselves are validated and rolled out.

Setup time

The time involved in setting up the development environment consists of the following activities:

1. Determining the target set of source elements (can be for multiple applications)

2. Provisioning (downloading) of the target set of elements

3. Setting up the development environment in an IDE

4. Compiling source elements such as Java source code

Although we find that practices like service orientation, binary dependencies, and code ownership help minimize code-level coupling, quick setup challenges still exist in large and fast-paced environments. The following sections describe eBay’s own evolution toward providing a faster and more reliable development environment.

Binary bundles

An application typically has several thousand source elements. Instead of provisioning and compiling all of those elements, the setup problem is simplified by consuming many of the elements in binary form. Already-compiled binary bundles diminish setup time by reducing the number of source elements that need downloading and compiling. The release stream tags the binary bundles that are released; different features can consume the bundles based on the release stream.

Feature teams can consume some projects as source while consuming many others in binary form. Source projects are required only where code authorship is required. As source projects take longer to set up, teams are encouraged to use them only for the projects that they expect to modify. The rest is consumed as binary bundles, to which the source is attached.

The binary bundle approach has clear advantages in reducing the number of source elements that need to be dealt with in the development environment. With this approach, the release streams are a vehicle for contributing changes that can be consumed later by other features. This methodology has also been adopted in the open-source world – for example, with the Apache Maven build tool and with increasingly popular binary repositories.

One of the challenges with the binary bundle approach is managing the bundles per release stream. This approach also requires the development team to identify the set of bundles that need authorship and thus need to be consumed as source. Furthermore, storing binary bundles at the release-stream level does not provide feature-level isolation. These shortcomings are discussed next.

Binary bundles with feature isolation and round-tripping

Feature-level isolation is required because features are developed in parallel and have overlapping code areas. In addition, features might be in early stages of development or might be developed in an experimental manner; changes from such features, in the form of binary bundles, should not be shared with others. With feature isolation, modified bundles are not shared until they have gone through appropriate validations and rollout. In other words, we desire to limit sharing when features are actively under development, and to encourage sharing after those changes have been rolled out.

To provide feature isolation, a common repository with release and snapshot versions does not suffice, as that approach is meant to encourage sharing. We use a per-feature binary repository to achieve feature isolation.

As individual features change only a small set of bundles, a complete binary repository per feature leads to duplication of binary bundles. We use hierarchical binary repositories to avoid duplication of bundles and to improve efficiency. Only bundles that are modified by features are placed in the feature repository. The rest of the bundles are obtained from higher-level repositories such as those from the release stream. This approach is similar to hierarchical management of source code in an SCM system (trunk -> branch 1 -> branch 2). In addition to the hierarchical binary repository, the local cache of these bundles is stored in a hierarchical manner to encourage sharing of common bundles across different features.

Another aid to rapid developer setup is dynamic conversion of binary bundles to source and vice versa. This conversion ability enables development to start before all of the projects that need source authorship have been identified. Furthermore, it encourages teams to start with a small set of source projects and to consume most (or even all!) in binary form. When the team realizes the need for source authorship, the appropriate projects are converted from binary to source. During conversion, the system determines source elements, provisions (downloads) them, compiles them, and appropriately adjusts project dependencies.

Once the authoring is completed, the projects can be converted back to binary form. This reverse conversion picks up binary bundles with the recent changes that have been made to those bundles.

The main challenge with this approach is managing the hierarchical repository, including timely updates as feature changes are committed. The hierarchical repository itself needs to point to different release streams as a feature is rebased. This approach still involves numerous source elements if the number or size of projects is large.

Binary bundles with source element changes

As discussed earlier in this post, most of our feature changes consist of hundreds of source elements. We find that these changes occur across tens of different projects. The chart below summarizes project size and usage in our environment.

The X-axis identifies the number of source elements in a project. The green bars show the percentage of projects that match a given bucket, and the red bars show the distribution of projects that are used as source. The median project size is about 50 to 100 elements, and we have a small number of projects that have more than 1000 elements. However, while small in number, the large projects get used disproportionately, to the extent where the largest of them are used significantly more compared to their counts.

We find that projects don’t start out being large; rather, some grow huge because many features use them and contribute to them. As these projects grow by contribution, many features consume them in source form. Due to their size, these source projects drastically increase development setup time.

To avoid such issues, we use various approaches: splitting large projects into several small projects, decoupling applications from common code areas, creating meta-information (DSL) instead of compiled source code, etc. These are all valid ways of addressing the problem. In our experience, it is easier to start with an approach that incorporates these concepts rather than changing to such an approach later, because of the pervasive nature of changes that touch large numbers of projects.

Another way of achieving rapid setup in the presence of large projects is by using source element changes in combination with binary bundles. When a project needs source authorship, instead of having all elements come in source form, we include a small set of the elements that are modified or that need authorship. The rest of the elements continue to come from binary bundles. We achieve this approach by setting the classpath such that compiled source elements take precedence over the elements coming in binary form.

The challenges with this approach include setting up the development environment while incorporating others’ feature changes, and dealing with non-source elements that are used for prebuilds. Given the limited number of source elements that change by feature, we are starting to adopt this approach in a limited fashion and are finding that it has potential to work well.

In the second part of this post, we will describe how we use the Eclipse IDE to optimize working with both source elements and binary bundles.

We are happy to announce ql.io – a declarative, evented, data-retrieval and aggregation gateway for HTTP APIs. Through ql.io, we want to help application developers increase engineering clock speed and improve end user experience. ql.io can reduce the number of lines of code required to call multiple HTTP APIs while simultaneously bringing down network latency and bandwidth usage in certain use cases.

ql.io consists of a domain-specific language inspired by SQL and JSON, and a node.js-based runtime to process scripts written in that language. Check out ql.io on Github for the source and http://ql.io for demos, examples, and docs.

Why ql.io

HTTP based APIs – some call them services – are an integral part of eBay’s architecture. This is true not just for eBay, but for most companies that use the Web for content and information delivery. Within eBay’s platform engineering group, we noticed several pain points for application developers attempting to get the data they need from APIs:

• Most use cases require accessing multiple APIs – which involves making several network round trips.
• Often those API requests have interdependencies – which requires programmatic orchestration of HTTP requests – making some requests in parallel and some in sequence to satisfy the dependencies and yet keep the overall latency low.
• APIs are not always consistent as they evolve based on the API producers’ needs – which makes code noisier in order to normalize inconsistencies.

We found that these issues have two critical impacts: engineering clock speed and end user experience.

• Engineering clocks slow down because developers need to account for dependencies between API calls, and to arrange those calls to optimize overall latency. Implementing orchestration logic involves multi-threaded fork-join code, leads to code bloat, and distracts from the main business use case that the developer is striving to support.
• End user experience suffers due to high bandwidth usage as well as the latency caused by the number of requests and the processing overhead of non-optimized responses from APIs.

The goal of ql.io is to ease both pain points:

• By using a SQL- and JSON-inspired DSL to declare API calls, their interdependencies, forks and joins, and projections, you can cut down the number of lines of code from hundreds of lines to a few, and the development time from entire sprints to mere hours. Using this language, you can create new consumer-centric interfaces that are optimized for your application’s requirements.
• You can deploy ql.io as an HTTP gateway between client applications and API servers so that ql.io can process and condense the data to just the fields that the client needs. This helps reduce the number of requests that the client needs to make as well as the amount of data transported to clients.

A quick taste

Here is one of the typical examples of ql.io usage. It shows how ql.io can transform the experience of a developer getting the data needed to paint the UI in a native application.

prodid = select ProductID[0].Value from eBay.FindProducts where
    QueryKeywords = 'macbook pro';
details = select * from eBay.ProductDetails where
    ProductID in ('{prodid}') and ProductType = 'Reference';
reviews = select * from eBay.ProductReviews where
    ProductID in ('{prodid}') and ProductType = 'Reference';

return select d.ProductID[0].Value as id, d.Title as title,
    d.ReviewCount as reviewCount, r.ReviewDetails.AverageRating as rating
    from details as d, reviews as r
    where d.ProductID[0].Value = r.ProductID.Value
    via route '/myapi' using method get;

This script uses three API calls (in this case, all offered by eBay) to get four fields of products that match a keyword. The result is provided via a new HTTP resource with URI http://{host}:{port}/myapi. See the guide to build this example yourself, or copy and paste the above script into ql.io’s Web Console to see it in action.

While we are still working on various benchmarks, we want to share some early results on developer productivity and end user benefits. One of the teams at eBay recently migrated an application that relies solely on eBay’s APIs to get the data needed to paint its UI. The first diagram below shows the request-response traces before migrating to ql.io.

The code related to these API calls was about 2800 lines long. The diagram below shows the request-response traces after migrating API access to ql.io.

This effort brought the code down to about 1200 lines, in addition to reducing the number of requests from 18 to 5 and the data size from 274k to 91k. In this experiment, latency drop is not significant as the client application was using broadband and some of the APIs used were slow APIs.

How to use ql.io

ql.io is not intended to replace frameworks that are currently used to build HTTP APIs. API producers can continue to use existing frameworks to offer interfaces that are generic and broadly reusable. ql.io comes into play when a consumer of APIs wants to implement consumer-specific aggregation, orchestration, and optimizations. In other words, while existing frameworks continue to support “producer-controlled” interfaces, you can use ql.io to create “consumer-controlled” interfaces.

We are building ql.io with flexible deployment in mind. Depending on where the network costs are felt, you can deploy ql.io closer to API servers, closer to users on the edge, or even on front-end machines.

Deploying closer to API servers

The primary usage of ql.io is to run it as a gateway at the reverse proxy tier, potentially between your load balancers and API servers.

Deploying closer to client applications

A secondary usage is to deploy ql.io closer to client applications on the edge.

Edge-side deployment can further reduce network costs for client applications by pushing API orchestration closer to those applications. Where API servers are globally distributed and the best place for aggregation may be closer to client applications, edge-side deployment may yield significant gains. If you are a developer using third-party APIs, you can follow the same pattern and deploy ql.io on your own closer to your applications.

Deploying on the front end

Our choice of Javascript and node.js for building ql.io provides an additional deployment option: front-end applications built on node.js can use ql.io programmatically.

Why node.js

Early on, one of the critical choices that we had to make was the software stack. We had two choices: Should we go with the proven Java stack that has full operational support within eBay? Or should we choose a stack like node.js with its excellent support for async I/O, but which was not yet proven when we started the project? Moreover, very few companies had operational experience with node.js. This was not an easy choice to make. In our deliberations, we considered the following systemic qualities, in their order of importance:

• Performance and scalability for I/O workloads.  Of workloads performed during script execution, a significant percentage  is I/O bound. CPU loads are limited to in-memory tasks like joining and projections. Blocking I/O was out of the equation for supporting such workloads.
• Operability. We need to be able to monitor the runtime, know what is going on, and react quickly when things go wrong. Furthermore, integrating with eBay’s logging and monitoring tools is a prerequisite for bringing in a new technology stack.
• Low per-connection memory overhead. Since script execution involves some slow and some fast APIs, we need the stack to remain stable as the number of open connections increases.
• Dynamic language support. This consideration had two parts. We wanted to build ql.io very quickly in a very small team with low code-to-execution turn-around times. This approach helps us iterate rapidly in the face of bugs as well as new use cases. In addition, we wanted application developers to be able to extend ql.io’s processing pipeline with small snippets of code.

After some analysis and prototyping, we chose Javascript as the language and node.js as the runtime stack. Here are some highlights of our experience so far:

• Javascript and node.js allowed us to iterate very rapidly. Though we were initially concerned about finding the right tools and libraries, the node.js ecosystem proved sufficient for us to build as complex a system as ql.io.
• We were able to tune a regular developer-quality Ubuntu workstation to handle more than 120,000 active connections per node.js process, with each connection consuming about 2k memory. We knew we could go further with the number of connections; although we did not spend the time to go beyond, this gave us the confidence to proceed with node.js.
• ql.io’s core engine does automatic fork-join of HTTP requests by using compile-time analysis of scripts. Node’s evented I/O model freed us from worrying about locking and concurrency issues that are common with multithreaded async I/O.
• We did pay some operationalization tax while we prepared the ql.io and node.js stack for integration with eBay’s monitoring and logging systems. This was a one-time penalty.

What’s next

We’re not done with ql.io yet, and we want to continue to develop ql.io in the open. Go to ql.io on Github and http://ql.io to find more about ql.io, try it out, discuss it, and participate.

Historical user click patterns on search result pages are considered a great resource for algorithms that attempt to learn to rank search results. This ranking method is a well-studied problem in the field of Information Retrieval (IR). See Mike Mathiesonʼs excellent blog post, Using Behavioral Data to Improve Search, for more background on this subject.

If learn-to-rank algorithms could explicitly observe which results were deemed relevant by most users and which were not, we would have open access to usersʼ minds. Alas, we live in the real world where only user clicks can be observed; their thought patterns are forever hidden, for us to infer and speculate upon.

What are click models?

Click models are mathematical models that attempt to do just that: Describe a typical userʼs decision process as he or she interacts with the search results page, so that we may infer said userʼs judgments on the relevance and irrelevance of specific search results.

Take, for example, the following scenario:

The user searched for “Meaning of life”. A search result page (SRP) with 50 results was served back. The user then clicked on result #2, and we never heard from him again.

Consider the following two explanations:

1. The user looked at the SRP, read the snippet returned for result #1, then ignored it as irrelevant. The user then moved to result #2, read the snippet, found it attractive, clicked through to the page, found the meaning of life in there, then stopped the search, satisfied with what he found.
2. The user glanced at the SRP, chose result #2 randomly, read the snippet, found it somewhat relevant, clicked-through to the page, and found it completely irrelevant. Then his phone rang and he abandoned the search.

According to the first explanation, result #2 was relevant to the userʼs search. According to the second explanation, it wasnʼt. Both explanations (and many others) are possible. But are they equally likely?

A click model helps us assign mathematical probabilities to every such explanation, enabling us to use millions of scenarios to infer the likelihood of relevance for every search result against every search.

Commonly Used Click Models

The simplest class of click models is called position models. Position models assume that search results have a probability of being examined by the user that decays with the position of the result within the page. A click depends on a result being examined and deemed relevant, so that P(click) = P(examined) * P(relevant), and P(examined) is a decaying function of rank.

Cascade models are another class of click models, where the user is assumed to examine the results sequentially: starting from the top, clicking on the first relevant result examined, and stopping the search immediately. Here the probability of a click depends on the relevance of a result, as well as the irrelevance of all previous results. This model doesnʼt account for abandoned searches, or searches with multiple clicks.

Recent work has shown that using a dynamic bayesian network (DBN) to model general web search clicks outperforms both position models and cascade models [Olivier Chapelle, A dynamic bayesian network click model for web search ranking, Proceedings of the 18th International World Wide Web Conference (WWW), 2009]. The DBN model assumes that users examine items in an SRP starting at position 1 and working downwards:  skipping items that appear irrelevant and clicking on items that appear relevant, until either abandoning the search or landing on a satisfactory page following a click from the SRP. The user is assumed to leave the search as soon as the first satisfactory page is presented. SRP clicks are observed in the logs, but all other variables in the bayesian network are hidden.

Figure 1 below illustrates the concept. The boolean variables inside the box relate to user behavior, whether observable or not, and the boolean variables below the box relate to intrinsic properties of URLs: a_u denotes how click-attractive a URL snippet is, and s_u denotes how satisfactory the page really is once you land on it following a search.

Here, clicks are the only observable events. All other events are to be inferred using expectation maximization (EM) guided by the causality links that the model assumes between boolean variables. The causality links are denoted in the figure by arrows.

A parameter of the model is the user persistence rate γ, which is the probability that a user would continue to examine results after landing on an irrelevant page.

Why We Need Something New

DBN and other web search click models are not well suited for explaining user behavior at eCommerce sites such as eBay. Compared to users of general web search, shoppers at eBay are unlikely to be satisfied by landing the first relevant result. Shopping and web browsing are radically different IR exercises. For shopping, the items in the SRP are competing for the shopperʼs business, and the shopper is in a process of arbitration, where he/she must choose to award business to zero or one winning item.

Unlike web surfers, web shoppers (think referees) are expected to sample seemingly relevant items until they feel satisfied that they have come across a representative sample of good deals as well as bad deals. Only then can a shopper make an informed decision that an item is a winner. In this setting, a good deal is only good relative to other deals that are not as good. For example, if the top ten results are equally good deals then weʼd expect the shopper to keep scrolling down until a satisfactory number of bad deals are encountered. That is in sharp contrast to the general web browsing behavior assumed in DBN.

Proposed Click Model

The following model builds upon DBN, with a focus on eCommerce user behavior.

In this model, SRPs list items on sale rather than webpage URLs. Every listing L is assumed to have two independent intrinsic properties: a_L and g_L. The a_L property denotes how attractive the listing looks in the SRP, which influences clicks; and g_L denotes how much of a good shopping deal the listing really is, which depends on the price, item details, seller details, shipping details, etc. Both variables are latent – that is, cannot be observed directly; however, they are assumed to influence the behavior of users on the site, and therefore can be inferred (hopefully) by analyzing mountains of click data.

Notice how the model still contains E_i (examine) events, and how, just like the DBN model, the user is assumed to examine results sequentially top to bottom. The proposed model also contains A_i (attracted enough to click) events; but S_i (satisfied) events are replaced with G_i (good deal) events, which denote that the user found the listing to be a “good deal” over all. We introduce a new event S (success event) to denote a successful shopping experience. At eBay, an S event translates into a bid or buy-it-now action. Because S events are directly observable, this model enables learning from clicks, bids, and purchases simultaneously.

Upon finding a good deal, the user performs an S event with probability f(G), which depends not only on the goodness of the current listing, but on the goodness of all previously examined listings. This is how the model accounts for users browsing multiple good and bad deals before arriving at an S decision. In the simplest case, f(G) can be the sum of good deals encountered, but can be generalized to the form:

f(G) = αΣ(good deals) + βΣ(bad deals)

Where α and β are model parameters.

This modeling framework, if properly applied to eCommerce click logs, would greatly enhance the learnability of ranking algorithms from click data.

In the first part of this blog posting, I talked about how to estimate a buyer’s propensity to purchase an auction over a fixed price item. I gave the Empirical Bayes formula f = (a+k)/(a + b+ n) for the auction propensity f which is a compromise between the buyer’s shopping history k/n and the propensity of his peer group a/(a+b). In this posting I will explain where the formula comes from, and how to compute the numbers a and b.

Since the method is called Empirical Bayes, it won’t surprise you to learn that it uses Bayes Theorem

The specific form I’ll use is

In this formula, p is the probability of buying an auction, n is the number of purchases that a shopper has made, and k is the number of those purchases that were auctions. The left-hand side is what I’d like to know: the probability (propensity) of an auction given that the buyer has previously purchased k/n auctions. As usual, the reason to use Bayes formula is that it relates the unknown left-hand side to the computable right-hand side.

The pleasure of Bayes formula is the first term on the right-hand side. This is the textbook formula we’ve seen before:

The pain of Bayes formula is the second term, the prior Pr(p). This is the estimate of p before (prior to) learning there were k/n auction purchases. Much time has been spent debating what value to use for the prior. This is where empirical Bayes comes in. I will use the data itself to estimate Pr(p). I do it by assuming that Pr(p) follows a Beta probability distribution

and all I need to do is specify the two parameters a and b. Why this distribution? It’s general enough to fit most data, but simple enough to make explicit calculations possible. I’ll do those calculations in a moment. The following figure from Wikipedia shows the first point. By varying the parameters a and b (α and β in the figure) you can get a large range of distributions.

Now for the details of how to compute a and b. The probability of purchasing k/n auctions is

If I don’t know a precise value for p, but do know it varies via some probability distribution Prob(p) then p_k is an average over those p, expressed as the integral

As I mentioned above, I’ll assume that Prob(p) is a beta distribution

Now plug this value of Prob(p) into the formula for p_k which I’ve written as p_k(a, b) to emphasize the dependence on a and b.

If you’re following along in detail, you’ll see that I did some simplifications after the plug-in. The important thing is this: the value of p_k depends on the known k, n and the unknown a, b, and my job is to pick those a, b so that

is minimized. In other words, I want to make the computed p_k as close as possible to the observed fraction, which I’ve written with a hat. For example p₂ (a,b) is the computed fraction of people in the peer group that bought 2 auctions assuming the prior is Beta(a, b), while p₂ with a hat is the observed fraction of users who bought two auctions.

Here’s an illustration. Suppose the peer group is users who’ve purchased 21 items. Then the best values of a and b are 1.16 and 2.22 which gives this beta distribution:

Once I have a and b I can compute the p_k (a,b) and see how closely they approximate

The match is excellent. The red predicted curve has only two parameters and yet it has a very good fit to all 22 points.

Now that I’ve explained how to compute a and b, I can explain where the formula f = (a+k)/(a + b+ n) comes from. Use Bayes formula

and substitute the beta distribution B_a,b for Pr(p) as follows:

I’ve been a little pedantic and used â to show that it is an estimate of a derived from the peer group (and similarly for b). The calculations above show that when you plug the formula for B_a,b(p) into Bayes formula you get out another beta distribution! This is the explanation for my earlier statement that beta was chosen to make explicit calculations possible.

My goal is to estimate the auction propensity for a buyer who has purchased k/n auctions. What I have so far is a probability distribution for this propensity, specifically a beta distribution. If I want a single probability, the natural choice is the average of that beta distribution. The average of Beta(a,b) is a/(a+b). So if I want to get a single number for the auction propensity, I take the average of

which is (a+k)/(a+b+n). This is the formula that I have used many times in this posting.

So much for the calculations. I want to end by discussing whether personalization is desirable. Some have argued that it isn’t a good idea to show users only what they’ve seen before. I think these arguments are often attacking a straw man. A good personalization system doesn’t eliminate all variety, it merely tailors it. Suppose our Mr. X has purchased 20 items, all of them at fixed price. But his friend Ms. Y has done the opposite: all her purchases were made by auction. I find it hard to believe that anyone would seriously argue that X and Y should see the same search results. Just as bad as showing X a pile of auctions would be to never show him any auctions. A good personalization system should be nuanced. Methods like those shown in this posting provide principled estimates of user’s preferences. But how these preferences are used requires some finesse.

In this blog post and a succeeding one, I will discuss personalization at eBay. Specifically, I’ll talk about how search can be personalized. First I’ll investigate whether buyers actually differ in their preferences for the items they’d like to see in their search results. Next I’ll discuss what preferences have the widest variation from buyer to buyer. Then I’ll talk about how to implement personalization, and finally I’ll explain the theory that underpins the implementation. Some people argue that personalization is actually not desirable. I’ll address that at the end of my last posting.

To make this concrete, I’ll focus on what eBay calls format: is the item being offered for sale via an auction, or is it a fixed-price item with a buy-it-now button? Of course some users buy more auctions than others. But does this indicate a preference? Perhaps not. Perhaps users simply search for the best deal, and if a user buys more auctions it’s not because they prefer auctions, but simply because they happened to purchase a string of items where the best deal was an auction.

The following graph suggests that the “just looking for the best deal” theory is not tenable.

Users who bought 1 item in the past year did not have much taste for auctions. They only purchased an auction 25% of the time. On the other hand, heavy buyers who purchased at least 600 items in the past year bought mostly auctions – over 70% of their purchases were auctions. So it seems that users do have different preferences. Infrequent buyers prefer fixed-price items, while frequent buyers prefer auctions.

But before discarding the “looking for best deal” hypothesis, I want to consider some alternate possibilities. First is the question you should always ask – is the data statistically significant? Given how consistently the curve moves upwards, it seems unlikely that we’re seeing an artifact due to noise. But I can check this rigorously. I’m measuring a binary variable – one that takes on two values, auction or fixed-price. Statistics textbooks tell us that the noise (as estimated by the standard error) is √ pq/n . The leftmost point represents 14 million buyers. So p=.25, q=.75 and n≈14000000. The noise is about 0.0001. I can be confident that the 25% is very accurate.

Here’s a more subtle alternate explanation. Suppose everyone has similar auction preferences. But suppose infrequent buyers tend to purchase electronics, which has fewer auctions, and frequent buyers are more likely to buy antiques, which have a high fraction of auctions. So the curve might be consistent with buyers having similar auction preferences, and simply be a reflection of the type of items that different buyers purchase.

I can test this alternate explanation by focusing on a single category, for example Jewelry and Watches. This is a category with a high proportion of auction listings.

The plot shows the same pattern as before: infrequent buyers prefer fixed price, heavy buyers prefer auctions. The curve is shifted upwards, because Jewelry is a high-auction category. But there’s a clear difference in preferences. The same pattern holds in a low-auction category like Computers & Networking.

The curve is shifted downward, but again more experienced buyers have a stronger preference for auctions. So to summarize, the most likely explanation of the graphs I’ve shown is that some buyers have a much stronger preference for auctions than other buyers.

But I want to dig deeper. Suppose I look at buyers with similar buying experience, for example all those who bought 8 items in the past year. Do they have different format preferences?

To answer that question, I need to be more precise about what it means for users to have similar format preferences. I’ll assume that buyers chose an auction with probability p. One way that might happen is that just before buyers make a purchase, they mentally throw dice in their head, and depending on the outcome say “I feel like an auction today”, or “I feel like buying fixed price today”. OK, that’s not too realistic, but there are other scenarios where it will appear that buyers are choosing an auction with probability p. For example their decision might depend on the inventory available, and that could vary randomly. Or more likely, their decision depends on factors that I don’t have available, such as how soon they want to receive the item (for an auction, you have to wait for bidding to end). In any case, imagining that buyers have some propensity for auctions that is captured as a probability is a simple but plausible model.

A consequence of this model is that I can predict how many auctions a user will buy. Recall that I’m focusing on those who bought 8 auctions. Assuming that all buyers have the same auction propensity p, the chance that k of the 8 purchases are an auction is the familiar formula In this formula n = 8. In Jewelry & Watches, about 75% of the purchases are auctions, so p = 0.75. The top histogram below labeled Predicted was generated using this formula.

Most users are predicted to buy 6/8 = 0.75 auctions. But many are predicted to buy 8 auctions. In our model they buy an auction with probability p, and so there’s random variation. They buy 6 on the average, but some buy 8 and others only 2. The bottom histogram gives the actual numbers. They are nothing like the prediction. This suggests that the hypothesis of all users have the same auction preference is false. Here’s another example in the Computers & Networking category.

Again, the count of auctions vs. fixed price is nothing like what would be predicted if all users had a similar auction preference. Here’s a summary of my reasoning. I focused on users with similar buying experience by looking at those who bought 8 items in the past year. I observed how many bought 0/8 auctions, how many bought 1/8, etc. If all buyers had similar auction preferences, that preference would be 75% (in Jewelry), and I could predict how many would buy 0/8, 1/8, etc. But the predicted values and observed values are totally different, so I conclude that buyers do not have similar auction preferences.

Let me move on to a new topic. Users have widely varying preferences for auctions. Are there other attributes of items that vary widely? I can repeat the analysis above to find out. I will examine 3 different attributes: item condition (new vs. used), whether the seller had an eTRS (eBay Top-Rated Seller) rating, and whether the item had free shipping. In each case I will compare the predicted and observed plots, and observe by how much they differ. First item condition:

The two histograms are very different, so condition appears to be a good attribute for personalization. Next is a graph for items sold by trusted sellers.

This time the histograms are similar. True, the tails are a bit larger in the observed histogram, but there is nowhere near the variation there was for item condition. Finally, here’s free shipping.

The observed histogram is not too different from what would be expected if everyone had a 44% preference for eTRS. So I conclude that personalizing search results on format and condition will probably be much more useful than personalizing on eTRS or free shipping.

Now that I’ve established that buyers have different preferences, I need to figure out how to estimate a buyer’s personal preferences. I will use a technique called Empirical Bayes. Like all things Bayes, it has some controversy. Brad Efron, the former chair of the Stanford Statistics department has said:

The suggestion here, made explicit in the final section, is that after 50 years of underuse, we are poised for an avalanche of empirical Bayes applications

Meanwhile across the Bay, the late statistician David Blackwell, who was chairman of the Berkeley statistics department, has been quoted as such:

He [David Blackwell] noted that he didn’t believe in empirical Bayes and showed that it didn’t make sense when applied to a single inference.

I will ignore the controversy and plunge ahead. First, here’s an extremely simple way to implement personalization: If a user has bought more than 10 auctions, I will assume I have enough information to estimate their auction preference, and compute it as k/n where k is the number of auction purchases and n is the total number of purchases. Otherwise I make no assumptions. This is clearly unsatisfying. Why 10? Do I really know nothing if they’ve purchased 9 items, but suddenly at 10 items I believe their purchase history is the whole story? As a rule of thumb, hard cutoffs like this do not perform as well as a smooth transition. And empirical Bayes is a perfect tool for eliminating the hard cutoff. It gives a principled way of estimating a buyer’s propensity (probability) of purchasing an auction. In the next posting, I’ll explain how to use Empirical Bayes to derive an estimation formula. For now, I’ll just explain how the formula applies to a typical buyer, Mr. X.

The first step is to aggregate buyers who are similar to Mr. X. This peer group is then summarized by two numbers, a and b. I’ll explain how to compute them later. These numbers encode the aggregate estimate of the auction propensity, f_A = a/(a+b). I write this as f, since this probability estimates the fraction of auctions. The next step is to record X’s purchases. If he has made n purchases, and k were in auction format, then his personal estimate is f_P = k/n. Empirical Bayes gives a compromise between these two numbers, via the formula f = (a+k)/(a + b+ n). I’ll next check whether this formula is reasonable.

Suppose there were no purchases, so that k = n = 0. Then the formula becomes f = (a+k)/(a + b+ n) = a/(a + b) which is the aggregate estimate. This makes sense. On the other extreme, suppose X has made many purchases so that k and n are large. Then the Bayes estimate becomes f = (a+k)/(a + b+ n) ≈ k/n which is the user’s personal preference. So again the formula makes sense: it gives the aggregate value initially, and then as X’s purchases increase, it becomes more like his personal value.

I’ll now work through an example. Suppose I’ve bought 8 items in the Computers & Networking category in the past year. I’ll use as my peer group the buyers who have also purchased 8 items in Computers & Networking. The a, b numbers are a=0.95, b=4.64. These numbers encode the behavior of my peer group, and I can use them to compute the average fraction of auctions purchased by my peers via , f_A = a/(a+b) = .95/(.95 + 4.64) = .17. The empirical Bayes formula says that my auction propensity is f = (a+k)/(a + b+ n) = (.95+k)/13.59. Here’s a plot of what the formula predicts for different values of k, the number of purchases that were auctions.

The empirical Bayes estimate is a compromise between the aggregate value and my personal value. Here’s the plot with n = 4 and n = 16 added to n = 8.

As n gets larger, the Empirical Bayes estimate gets closer and closer to the personal estimate. Here’s a second example in the Jewelry & Watches category. Again with 8 purchases, the numbers a and b are a = 0.75, b = 0.80 and so f_A = a/(a+b) = .75/(.75 + 40.8) = .48. The plot is below:

Again, the Empirical Bayes estimate is between the aggregate and personal. You might be wondering why the aggregate isn’t more simply represented by a single number f_A, rather than decomposing it into f_A = a/(a+b). The reason is that a and b encode not just the average number of auctions in your peer group, but also the variation in that group. When a and b are large, buyers in the peer group cluster closely near f_A (left plot below). When a and b are small, buyers have a large spread (right plot).

This is consistent with f = (a+k)/(a + b+ n). When a and b are large the left graph shows that the peer group is very consistent in their auction preference, and so we should be skeptical of a user whose personal preference is very different from f_A. And indeed, the formula f = (a+k)/(a + b+ n) shows that k and n have small influence. When a and b are small the right graph applies. It shows that peer group has no strong preference, and so k and n are more important.

I’ve actually already shown you an example of this. In Computers & Networking, a and b are large, and the Bayes compromise is roughly midway between the personal and aggregate. But in Jewelry & Watches, a and b are small and the compromise is much closer to the personal line – see below.

In my next posting I will explain where the formula f = (a+k)/(a + b+ n) comes from, and how to compute the numbers a and b.

Our eBay Motors homepage has been around for more than 5 years, and we needed to give it a facelift. One of our goals was to make it faster: we wanted the new page to render twice as fast compared to the old one.

We started by running the YSlow and PageSpeed plug-ins and implementing the recommendations. Much better already, as you can see below!

You can run the PageSpeed tests yourself against the old and new homepages:

• Old: http://pagespeed.googlelabs.com/pagespeed/#url=www.motors.ebay.com&mobile=false
• New: http://pagespeed.googlelabs.com/pagespeed/#url=www.ebay.com_2Fmtr&mobile=false

We wanted to do more. Here are the three things we learned along the way:

1. Progressive rendering via multiple flushes is a great way to increase perceived performance of a page because the user starts seeing content at the earliest possible time. Using just two chunks– one comprising all content above the fold, and the second everything below the fold — can give you great improvements. Our data indicates that 70% of our users are viewing our pages with a screen resolution of 1024×768, so we can divide the page as shown below (click on images to see enlarged versions).

This approach makes it possible to do very specific site speed optimizations. The first chunk above the fold needs to be optimized for fast server-side processing. By adding smart server caching mechanism, the number of backend HTTP calls needed to retrieve all the data for the above-fold content can be reduced significantly. In contrast, all JavaScript loading, parsing, and binding can be moved below the fold. As a result, a user can already see page content while below-the-fold content and JavaScript behavioral characteristics are still being delivered.

2. Data URI for Static Images: Reduce the number of HTTP requests for static images (both sprited and individual) by leveraging the Data URI scheme. The new Motors homepage includes all static images as base64-encoded data URIs in CSS instead of as external URLs. (For IE7 and other browsers that don’t support Data URI, you need to have a CSS fallback to include external URLs.) Because CSS files are cached in the browser, you also get the benefit of image caching – similar to using external image URLs – in addition to reduced HTTP calls.

3. CSS Lint is an open-source tool similar to JS Lint used to identify CSS coding problems. From the wide variety of configurable rules that CSS Lint offers, you can pick a subset based on your application’s needs. Due to the various options available in the CSS Lint source, integration with your CI  build system should be pretty straightforward. Look for improvement opportunities in the following areas:
   • removing unused and redundant code
   • using efficient CSS selectors
   • writing reusable and maintainable code

   After running CSS Lint on our code base and fixing the reported problems, we improved page rendering by about 200 milliseconds — a big win for us! And now that CSS Lint is a part of our build system, we are sure to catch issues before they become a problem in production. We highly recommend this tool for CSS-heavy websites.

What’s next? With all the performance optimizations, we were able to achieve our goal in our first release. The new page is indeed twice as fast as the old one! We are now testing ideas for the next iteration. One promising approach is displaying images using div tags with the background-image style, rather than using img tags with the src attribute. After reading about this technique we experimented with it in one of our image-heavy projects, and the results were positive. Some tests showed page speed improved by as much as 23%; however, we didn’t see a difference for pages with few images. We are going to test the technique on our image-heavy Motors homepage in one of our upcoming releases.

Go ahead, try out the new eBay Motors homepage! We would love to get your feedback.

Senthil Padmanabhan
Engineering Lead & Site Speed Evangelist