eBay Tech Blog

At eBay, we take site speed very seriously and are always looking for ways to allow developers to create faster-loading web apps. This involves fully understanding and controlling how web pages are delivered to web browsers. Progressive HTML rendering is a relatively old technique that can be used to improve the performance of websites, but it has been lost in a whole new class of web applications. The idea is simple: give the web browser a head start in downloading and rendering the page by flushing out early and multiple times. Browsers have always had the helpful feature of parsing and responding to the HTML as it is being streamed down from the server (even before the response is ended). This feature allows the HTML and external resources to be downloaded earlier, and for parts of the page to be rendered earlier. As a result, both the actual load time and the perceived load time improve.

In this blog post, we will take an in-depth look at a technique we call “Async Fragments” that takes advantage of progressive HTML rendering to improve site speed in ways that do not drastically complicate how web applications are built. For concrete examples we will be using Node.js, Express.js and the Marko templating engine (a JavaScript templating engine that supports streaming, flushing, and asynchronous rendering). Even if you are not using these technologies, this post can give you insight into how your stack of choice could be further optimized.

To see the techniques discussed in this post in action, please take a look at the accompanying sample application.


Progressive HTML rendering is discussed in the post The Lost Art of Progressive HTML Rendering by Jeff Atwood, which was published back in 2005. In addition, the “Flush the Buffer Early” rule is described by the Yahoo! Performance team in their Best Practices for Speeding Up Your Web Site guide. Stoyan Stefanov provides an in-depth look at progressive HTML rendering in his Progressive rendering via multiple flushes post. Facebook discussed how they use a technique they call “BigPipe” to improve page load times and perceived performance by dividing up a page into “pagelets.” Those articles and techniques inspired many of the ideas discussed in this post.

In the Node.js world, its most popular web framework, Express.js, unfortunately recommends a view rendering engine that does not allow streaming and thus prevents progressive HTML rendering. In a recent post, Bypassing Express View Rendering for Speed and Modularity, I described how streaming can be achieved with Express.js; this post is largely a follow-up to discuss how progressive HTML rendering can be achieved with Node.js (with or without Express.js).

Without progressive HTML rendering

A page that does not utilize progressive HTML rendering will have a slower load time because the bytes will not be flushed out until the complete HTML response is built. In addition, after the client finally receives the complete HTML it will then see that it needs to download additional external resources (such as CSS, JavaScript, fonts, and images), and downloading these external resources will require additional round trips. In addition, pages that do not utilize progressive HTML rendering will also have a slower perceived load time, since the screen will not update until the complete HTML is downloaded and the CSS and fonts referenced in the <head> section are downloaded. Without progressive HTML rendering, a server/client waterfall chart might be similar to the following:

Single Flush Waterfall Chart

The corresponding page controller might look something like this:

function controller(req, res) {
            function loadSearchResults(callback) {
            function loadFilters(callback) {
            function loadAds(callback) {
        function() {
            var viewModel = { ... };
            res.render('search', viewModel);

As you can see in the above code, the page HTML is not rendered until all of the data is asynchronously loaded.

Because the HTML is not flushed until all back-end services are completed, the user will be staring at a blank screen for a large portion of the time. This will result in a sub-par user experience (especially with a poor network connection or with slow back-end services). We can do much better if we flush part of the HTML earlier.

Flushing the head early

A simple trick to improve the responsiveness of a website is to flush the head section immediately. The head section will typically include the links to the external CSS resources (i.e. the <link> tags), as well as the page header and navigation. With this approach the external CSS will be downloaded sooner and the initial page will be painted much sooner as shown in the following waterfall chart:

Flush Head Waterfall Chart

As you can see in the chart above, flushing the head early reduces the time to render the initial page. This technique improves the responsiveness of the page, but it does not significantly reduce the total time it takes to make the page fully functional. With this approach, the server is still waiting for all back-end services to complete before flushing the final HTML. In addition, downloading of external JavaScript resources will be delayed since <script> tags are placed at the end of the page (assuming you are following best practices) and don’t get sent out until the second and final flush.

Multiple flushes

Instead of flushing only the head early, it is often beneficial to flush multiple times before ending the response. Typically, a page can be divided into multiple fragments where some of the fragments may depend on data asynchronously loaded from various back-end services while others may not depend on any asynchronously loaded data. The fragments that depend on asynchronously loaded data should be rendered asynchronously and flushed as soon as possible.

For now, we will assume that these fragments need to be flushed in the proper HTML order (versus the order that the data asynchronously loads), but we will also show how out-of-order flushing can be used to further improve both page load times and perceived performance. When using “in-order” flushing, fragments that complete out of order will need to be buffered until they are ready to be flushed in the proper order.

In-order flushing of async fragments

As an example, let’s assume we have divided a complex page into the following fragments:

Page diagram

Each fragment is assigned a number based on the order that it appears in the HTML document. In code, our output HTML for the page might look like the following:

    <title>Clothing Store</title>
    <!-- 1a) Head <link> tags -->
        <!-- 1b) Header -->
    <div class="body">
            <!-- 2) Search Results -->
        <section class="filters">
            <!-- 3) Search filters -->
        <section class="ads">
            <!-- 4) Ads -->
        <!-- 5a) Footer -->
    <!-- 5b) Body <script> tags -->

The Marko templating engine provides a way to declaratively bind template fragments to asynchronous data provider functions (or Promises). An asynchronous fragment is rendered when the asynchronous data provider function invokes the provided callback with the data. If the asynchronous fragment is ready to be flushed, then it is immediately flushed to the output stream. Otherwise, if the asynchronous fragment completed out of order then the rendered HTML is buffered in memory until it is ready to be flushed. The Marko templating engine ensures that fragments are flushed in the proper order.

Continuing with the previous example, our HTML page template with asynchronous fragments defined will be similar to the following:

    <title>Clothing Store</title>
    <!-- Head <link> tags -->
        <!-- Header -->
    <div class="body">
            <!-- Search Results -->
            <async-fragment data-provider="data.searchResultsProvider"

                <!-- Do something with the search results data... -->
                    <li for="item in searchResults.items">

        <section class="filters">

            <!-- Search filters -->
            <async-fragment data-provider="data.filtersProvider"
                <!-- Do something with the filters data... -->

        <section class="ads">

            <!-- Ads -->
            <async-fragment data-provider="data.adsProvider"
                <!-- Do something with the ads data... -->

        <!-- Footer -->
    <!-- Body <script> tags -->

The data provider functions should be passed to the template as part of the view model as shown in the following code for a sample page controller:

function controller(req, res) {
            searchResultsProvider: function(callback) {
                performSearch(req.params.category, callback);

            filtersProvider: function(callback) {

            adsProvider: function(callback) {
        res /* Render directly to the output HTTP response stream */);

In this particular example, the “search results” async fragment appears first in the HTML template, and it happens to take the longest time to complete. As a result, all of the subsequent fragments will need to be buffered on the server. The resulting waterfall with in-order flushing of async fragments is shown below:

In-order Flush Waterfall Chart

While the performance of this approach might be fine, we can enable out-of-order flushing for further performance gains as described in the next section.

Out-of-order flushing of async fragments

Marko achieves out-of-order flushing of async fragments by doing the following:

Instead of waiting for an async fragment to finish, a placeholder HTML element with an assigned id is written to the output stream. Out-of-order async fragments are rendered before the ending <body> tag in the order that they complete. Each out-of-order async fragment is rendered into a hidden <div> element. Immediately after the out-of-order fragment, a <script> block is rendered to replace the placeholder DOM node with the DOM nodes of the corresponding out-of-order fragment. When all of the out-of-order async fragments complete, the remaining HTML (e.g. </body></html>) will be flushed and the response ended.

To clarify, here is what the output HTML might look like for a page with out-of-order flushing enabled:

    <title>Clothing Store</title>
    <!-- 1a) Head <link> tags -->
        <!-- 1b) Header -->
    <div class="body">
            <!-- 2) Search Results -->
            <span id="asyncFragment0Placeholder"></span>
        <section class="filters">
            <!-- 3) Search filters -->
            <span id="asyncFragment1Placeholder"></span>
        <section class="ads">
            <!-- 4) Ads -->
            <span id="asyncFragment2Placeholder"></span>
        <!-- 5a) Footer -->

    <!-- 5b) Body <script> tags -->

    // Small amount of code to support rearranging DOM nodes
    // Unminified:
    // https://github.com/raptorjs/marko-async/blob/master/client-reorder-runtime.js

    <div id="asyncFragment1" style="display:none">
        <!-- 4) Ads content -->

    <div id="asyncFragment2" style="display:none">
        <!-- 3) Search filters content -->

    <div id="asyncFragment0" style="display:none">
        <!-- 2) Search results content -->


One caveat with out-of-order flushing is that it requires JavaScript running on the client to move each out-of-order fragment into its proper place in the DOM. Thus, you would only want to enable out-of-order flushing if you know that the client’s web browser has JavaScript enabled. Also, moving DOM nodes may cause the page to be reflowed, which can be visually jarring to the user and result in more client-side CPU usage. If reflow is an issue then there are tricks that can be used to avoid a reflow (e.g., reserving space as part of the initial wireframe). Marko also allows alternative content to be shown while waiting for an out-of-order async fragment.

To enable out-of-order flushing with Marko, the client-reorder="true" attribute must be added to each <async-fragment> tag, and the <async-fragments> tag must be added to the end of the page to serve as the container for rendered out-of-order fragments. Here is the updated <async-fragment> tag for the search results fragment:

<async-fragment data-provider="data.searchResultsProvider"

The updated HTML page template with the new <async-fragments> tag is shown below:

    <title>Clothing Store</title>
    <!-- Head <link> tags -->

    <!-- Body <script> tags -->


In combination with out-of-order flushing, it may be beneficial to move <script> tags that link to external resources to the end of the first chunk (before all of the out-of-order chunks). While the server is busy preparing the rest of the page, the client can start downloading the external JavaScript required to make the page functional. As a result, the user will be able to start interacting with the page sooner.

Our final waterfall with out-of-order flushing will now be similar to the following:

Out-of-order Flush Waterfall Chart

The final waterfall shows that the strategy of out-of-order flushing of asynchronous fragments can significantly improve the load time and perceived load time of a page. The user will be met with a progressive loading of a page that is ready to be interacted with sooner.

Additional considerations

HTTP Transport and HTML compression

To allow HTML to be served in parts, chunked transfer encoding should be used for the HTTP response. Chunked transfer encoding uses delimiters to break up the response, and each flush results in a new chunk. If gzip compression is enabled (and it should be) then flushing the pending data to the gzip stream will result in a gzip data frame being written to the response as part of each chunk. Flushing too often will negatively impact the effectiveness of the compression algorithm, but without flushing periodically then progressive HTML rendering will not be available. By default, Marko will flush at the beginning of an <async-fragment> block (in order to send everything that has already completed), as well as when an async fragment completes. This default strategy results in efficient progressive loading of an HTML page as long as there are not too many async fragments.

Binding behavior

For improved usability and responsiveness, there should not be a long delay between rendered HTML being displayed to the user in the web browser and behavior being attached to the associated DOM. At eBay, we use the marko-widgets module to bind behavior to DOM nodes. Marko Widgets supports binding behavior to rendered widgets immediately after each independent async fragment, as illustrated in the accompanying sample app. For immediate binding to work, the required JavaScript code must be included earlier in the page. For more details, please see the marko-widgets module documentation.

Error handling

It is important to note that as soon as a byte is flushed for the HTTP body, then the response is committed; no additional HTTP headers can be sent (e.g., no server-side redirects or cookie-setting), and the HTML that has been sent cannot be “unsent”. Therefore, if an asynchronous data provider errors or times out, then the app must be prepared to show alternative content for that particular async fragment. Please see the documentation for the marko-async module for additional information on how to show alternative content in case of an error.


The Async Fragments technique allows web developers to maximize the benefits of progressive HTML rendering to produce web pages that have improved actual and perceived load times. Developers at eBay have found the concept of binding HTML template fragments to asynchronous data providers easy to grasp and utilize. In addition, the flexibility to support both in-order and out-of-order flushing of async fragments makes this technique applicable for all web browsers and user agents.

The Marko templating engine is being used as part of eBay’s latest Node.js stack to improve performance while also simplifying how pages are constructed on both the server and the client. Marko is one of a few templating engines for Node.js and web browsers that support streaming, flushing, and asynchronous rendering. Marko has a simple HTML-based syntax, and the Marko compiler produces small and efficient JavaScript modules as output. We encourage you to try Marko online and in your next Node.js project. Because Marko is a key component of eBay’s internal Node.js stack, and given that it is heavily documented and tested, you can be confident that it will be well supported.

Patrick Steele-Idem is a member of eBay’s platform team who enjoys writing open-source software and improving how web applications are built. He is the author of RaptorJS, a suite of open-source front-end power tools that are being used within and outside eBay. You can follow Patrick on Twitter at @psteeleidem.


Diversity in Search

by David Goldberg on 11/26/2014

in Search Science

This post is about how to rank search results, specifically the list of items you get after doing a search on eBay.


One simple approach to ranking is to give a score to each item. For concreteness, imagine the score to be an estimate of the probability that a user will buy the item. Each item is scored individually, and then the items are presented in score order, with the item most likely to be purchased at the top.

This simple approach makes a lot of sense. But there are some situations where it can work poorly. For example, suppose that there are 40 results, 20 for Sony and 20 for Panasonic. And suppose that all the Sony items have a score that is just a tiny bit better than any of the Panasonic items. Then the first 20 results will be Sony, giving the incorrect impression that we only have Sony. It would be better to show at least one Panasonic near the top, preferably high enough to be “above the fold”. In other words, in this case I want to take diversity into account. An ultra-simple method to handle this case would be to add a small amount of random noise to each score. But I think you can see that this is somewhat of a kludge, and doesn’t address the problem of balancing score and diversity head-on.

Solutions in the literature

One of the first principled methods for diversity goes back 15 years to the paper of Jaime Carbonell and Jade Goldstein, The use of MMR, diversity-based reranking for reordering documents and producing summarization, in Proceedings of SIGIR ’98. Their idea is a simple one. Find a way to measure the similarity of items, where presumably two Sony items would be more similar than a Sony item and a Panasonic one. Then trade off the score and the similarity. In more detail, they propose placing items on the results page one step at a time. The first step places the item with the highest score, and records its index in the set of placed items I. The second step examines every unplaced item and compares each one’s score with its similarity to the item i (where I={i}). If P(j) is the score of j (P for probability) then each item j is evaluated using what they call marginal relevance: λP(j)−(1−λ)S(i,j). The marginal relevance is higher when the score P is better, and it is higher when the similarity to i is less. The parameter λ determines the relative importance of these two numbers. The item chosen for the second slot is the one with maximum marginal relevance (MMR):

    \[ MMR = \mbox{argmax}_{j \notin I}\left(\lambda P(j) - (1 - \lambda)S(i, j)\right)\]

The MMR item is placed on the results page, and its index is added to I. In general the algorithm computes

    \[MMR = \mbox{argmax}_{j \notin I}\left(\lambda P(j) - (1 - \lambda)\max_{i \in I}S(i, j)\right)\]

and places the resulting item on the results page as well as adding its index in I.

This method seems an improvement over adding random noise, but is unsatisfying for several reasons. Once it places an item, it is reluctant to place a second similar item. This reluctance may be fine for web search where you only want to see one instance of each kind of page, but is not so good for e-commerce where you’d like to compare multiple items. Also, there are almost certainly several dimensions of similarity (brand, cost, new vs. used, etc.), and it’s not clear how to combine them all into a single similarity function.

Rakesh Agrawal et. al. have a paper Diversifying search results from WSDM ’09 that doesn’t address any of those objections, but does address a different problem: how do you pick λ? They propose an algorithm that doesn’t have an arbitrary tuning parameter. They suppose that each item is in a category and that the demand (for a fixed query) of each category is known. This approach maps well with eBay search, since we know the category demand for our top queries. How to use this info? They imagine that users who issue query Q are diverse—that some are looking for items in category C1 while others want an item in a different category C2, and that this diversity is captured in the category demand table. The paper gives an algorithm that maximizes the chance that a user will find an item in his or her desired category. Let’s call this the Agrawal algorithm.

The Agrawal algorithm is a step above MMR in elegance because it doesn’t require fiddling with an arbitrary constant like λ. It works well for categories, but what about other forms of diversity like brand or item condition (new vs. used)? Just as we record category demand, we could record demand for brands, item condition, etc. But then we would need a way to soup-up the Agrawal algorithm to combine these different demands, and presumably would need to prioritize them. And like MMR, the Agrawal algorithm heavily penalizes an item if a similar one already exists, which is not appropriate for e-commerce.

A paper by Michael Welch et. al. Search result diversity for informational queries, WWW ’11, addresses one of the objections to Agrawal. They assume that users want to see more than one document per category. Specifically, they introduce a random variable J representing the number of items a user wants to see. You have to provide an estimate for the probability distribution of J (i.e., find numbers pj = Pr(J = j)) and then the algorithm uses that in its ranking. But this approach still doesn’t address our requirement to gracefully handle multiple dimensions of diversity.

There’s another factor to take into account. With multiple dimensions, we are unlikely to have a good estimate of the demand. For example, if the dimensions are category, brand and condition, we would ideally want the demands for each tuple: for example, (category=TV, Brand=Sony, condition=Used). For all these reasons, we abandoned the Carbonell → Agrawal → Welch train of thought, and took a different approach.

Our solution: agents

For each query, we specify which dimensions are important, together with constraints in the form of a max and/or min. For example, for the query “flat screen TV”, we might want at least 10% new items, and at most 50% with brand Samsung. This fits in nicely with our search architecture, which lets us customize the search behavior for a specific query by putting info into DSBE, a database indexed by query. Also we expect that in the common case the min/max constraints aren’t violated, and so the exact value of the constraints aren’t critical.

You can think of this as a very simple agent system. Each dimension has an agent with constraints (max/min). Each agent monitors the construction of the results page, and tries to keep its constraint satisfied. The agents won’t always succeed, so think of the requirements as being soft constraints.

Whenever you have multiple constraints, your first question should be how to prioritize them. The answer falls out almost automatically from the agent algorithm, which I now describe.

Each agent monitors the construction of the result set. At each stage, the agent checks on its constraints. If any are violated, it scans through the unplaced items looking for an item that will get the result set closer to satisfying its constraints. This item is the candidate.

If the agent’s constraint is violated, it is unhappy and it quantifies its unhappiness by summing two terms. The first term measures how much the placed items deviate from the agent’s constraints. The more deviation, the more unhappiness. The second term involves the candidate. There will be a penalty for placing the candidate on the search results page (SRP) instead of placing the default item, because the candidate has a lower score (P) than the default has. Summarizing the terminology:

  • Candidate: An item proposed by the agent for the next spot on the results page. If no agent proposes a candidate, the default item is the unplaced item of highest score.
  • Deviance: How far the placed items deviate from the agent’s constraints. Larger means more deviation.
  • Penalty: The penalty (in reduced score) for placing the candidate instead of the default. Larger means more of a penalty, i.e. more gap between the scores.
  • Unhappiness: The agent’s unhappiness. If unhappiness>0 then the agent’s candidate will be a contender for the next placed item.

Now back to the question of prioritizing. The priority falls out automatically. If multiple agents are unhappy, we simply pick the one with the largest unhappiness score, and use its candidate as the next item to be placed.

This approach isn’t hassle-free. You need to pick constraints (max/min) for each query. And you need to quantify deviance and penalty. Or in other words, find how to weight them, which is reminiscent of the parameter λ in MMR. But we prefer this approach because it is efficient, handles multiple constraints, and is not too sensitive to the exact values of max and min. For most queries, we expect the constraints to hold with the default ranking, and so placing an agent’s candidate is the exception rather than the rule.

A sample formula for unhappiness

The description in the previous section was abstract in that it talked about deviance and penalty without offering a way to compute them. Here is one way.

A typical constraint requires that the fraction items with property P be at least (this is a min constraint). might be that an item is used, or has brand Sony, or is newly listed. The straightforward way to compute deviation from a constraint is to look at the number of items with P placed so far and compare it to the number needed to meet the constraint. Suppose that there are n items placed so far, and that k of them are in P.  Since the constraint is min = f we’d like at least n items. If k < nf the constraint isn’t met, so set the deviance to nf − k . Or more precisely, deviance is (nf − k)+ where xx when > 0 and 0 otherwise.

A very simple way to compute the penalty is to compare the score of the candidate item IC with the item that would otherwise be placed, Idefault.  The penalty is S(IC ) − S(Idefault). Since items are sorted in score order, the penalty is always nonnegative.

Unhappiness is computed using the formula unhappiness = deviance − λ × penalty.  Imagine  deviance > 0 but the only way to fix it is by placing an item with a huge penalty. The large − term makes unhappiness negative, so the agent isn’t unhappy after all. In other words, agents are altruistic and don’t insist on their constraint if it results in placing a poor item. The parameter \lambda controls the tradeoff between score and diversity.

As a preliminary formula, unhappiness is

    \[\text{unhappiness} = \text{deviance} - \lambda \times \text{penalty} = (nf - k)^+ - \lambda ( S(I_C) - S(I_{\scriptsize{\text{default}}}))\]

But there’s one issue remaining. Suppose the constraint asks for at least 10%, and suppose the first item placed does not have the desired property. Then n = 1, f = 0.1, and = 0 so (nf − k )+ =(0.1)+ =0.1 and the constraint is already unhappy and will be in the running to have its proposed items placed next. This seems like jumping the gun, since the goal is 10% but only one item has been placed. There are at least two ways to account for this situation.

The first would be rounding: replace (nf − k)+ with round(nf − k)+.   But I think of constraints as something only triggered in exceptional circumstances, and so prefer to have unhappiness go negative only at the last possible minute. In other words, if it is still possible to satisfy nf ≤ k on the next round, then don’t trigger the constraint.

Writing this as an equation, suppose I pass on this constraint and the next item turns out to not have P. Then k stays the same and n becomes + 1. But on the next round I pick an item with P. Then k becomes + 1 and n becomes + 2. So I can satisfy the constraint in the future if (+ 2)f ≤ + 1. So replace (nf−k)+ with ((n+2)f − (k+1))+. The formula becomes

    \[ \boxed{ \footnotesize{\mbox{unhappiness}} = \footnotesize{\mbox{deviance}} - \lambda \times \mbox{penalty} = ((n+2)f - k - 1)^+ - \lambda (S(I_C) - S(I_{\scriptsize{\text{default}}}))} \]


There are two types of constraints based on a property P. The first is a min constraint, which asks that the fraction of items with P be greater than fmin. The second is max constraint, asking for the fraction to be less than fmax. Most properties are something very specific, like item=used. But there is also an any property. This would be used (for example) to keep any one seller from dominating the results. A max constraint with property seller, f and using any asks that there be no seller id with more than f of the results. The any property doesn’t make sense with the min constraint.

The any property is also a nice way to do duplicate (or fuzzy) dup removal. We assign a hash to each item, so that duplicate (or similar) items have the same hash. Then using a max with f = 1/50 means (roughly) that each item will appear only once on each 50-item page. If λ = 0 the requirement is absolute (unless there are competing agents). But by setting λ to a small number, we can allow dups if the alternative is to show very low quality items.

The algorithm

I close with a more formal description of the agent-based diversity algorithm. It can be implemented quite efficiently. In MMR (as well as Welch and Agrawal), at each step you must scan all unplaced items (think O(n2)). In our algorithm, you scan each item once for the entire algorithm (although this is done separately for each constraint). Scanning is implemented by having a pointer per constraint. The pointer moves down the unplaced list as the algorithm proceeds.

Input: a list of unplaced items, sorted by score
Output: items in their final arrangement for display on the search results page (SRP)
Remove the top item from unplaced and append it to SRP
For every constraint C, set C.ptr := first unplaced item
While unplaced is not empty
 initialize the set candidates to be empty
 ForEach C in set of soft constraints
 If C.unhappiness() > 0
 add C to the candidates
 // this has the side effect of updating C.ptr
 If candidates = empty then
 set item to be the top (highest score) item on unplaced
 take the element C in candidates with the largest unhappiness score
 set item := C.proposed_item()
 if there are any C.ptr pointing at item, move them down one
 remove item from unplaced and append it to SRP
Method C.unhappiness() =
 n := number of items placed in SRP so far
 f := C.f // the bound for this constraint
 k := the number of items placed so far that satisfy the constraint C
 // For the "any" property, k is the max number of items with the
 // same property value that satisfy the constraint
 if C.op = min then
 deviance := ((n+2)f - k - 1)+
 deviance := (k + 1 - (n+2)f)+
 Starting at C.ptr, scan down unplaced until reaching an item I that will reduce deviance
 C.ptr := I
 Idefault := highest scoring unplaced item
 penalty := S(I) - S(Idefault)
 unhappiness := deviance - lambda * penalty
 // set proposed_item field and return
 C.proposed_item := I
 return unhappiness


We are very excited to announce that eBay has released to the open-source community our distributed analytics engine: Kylin (http://kylin.io). Designed to accelerate analytics on Hadoop and allow the use of SQL-compatible tools, Kylin provides a SQL interface and multi-dimensional analysis (OLAP) on Hadoop to support extremely large datasets.

Kylin is currently used in production by various business units at eBay. In addition to open-sourcing Kylin, we are proposing Kylin as an Apache Incubator project.


The challenge faced at eBay is that our data volume has become bigger while our user base has become more diverse. Our users – for example, in analytics and business units – consistently ask for minimal latency but want to continue using their favorite tools, such as Tableau and Excel.

So, we worked closely with our internal analytics community and outlined requirements for a successful product at eBay:

  1. Sub-second query latency on billions of rows
  2. ANSI-standard SQL availability for those using SQL-compatible tools
  3. Full OLAP capability to offer advanced functionality
  4. Support for high cardinality and very large dimensions
  5. High concurrency for thousands of users
  6. Distributed and scale-out architecture for analysis in the TB to PB size range

We quickly realized nothing met our exact requirements externally – especially in the open-source Hadoop community. To meet our emergent business needs, we decided to build a platform from scratch. With an excellent team and several pilot customers, we have been able to bring the Kylin platform into production as well as open-source it.

Feature highlights

Kylin is a platform offering the following features for big data analytics:

  • Extremely fast OLAP engine at scale: Kylin is designed to reduce query latency on Hadoop for 10+ billion rows of data.
  • ANSI SQL on Hadoop: Kylin supports most ANSI SQL query functions in its ANSI SQL on Hadoop interface.
  • Interactive query capability: Users can interact with Hadoop data via Kylin at sub-second latency – better than Hive queries for the same dataset.
  • MOLAP cube query serving on billions of rows: Users can define a data model and pre-build in Kylin with more than 10+ billions of raw data records.
  • Seamless integration with BI Tools: Kylin currently offers integration with business intelligence tools such as Tableau and third-party applications.
  • Open-source ODBC driver: Kylin’s ODBC driver is built from scratch and works very well with Tableau. We have open-sourced the driver to the community as well.
  • Other highlights: 
  • Job management and monitoring
  • Compression and encoding to reduce storage
  • Incremental refresh of cubes
  • Leveraging of the HBase coprocessor for query latency
  • Approximate query capability for distinct counts (HyperLogLog)
  • Easy-to-use Web interface to manage, build, monitor, and query cubes
  • Security capability to set ACL at the cube/project level
  • Support for LDAP integration

The fundamental idea

The idea of Kylin is not brand new. Many technologies over the past 30 years have used the same theory to accelerate analytics. These technologies include methods to store pre-calculated results to serve analysis queries, generate each level’s cuboids with all possible combinations of dimensions, and calculate all metrics at different levels.

For reference, here is the cuboid topology:


When data becomes bigger, the pre-calculation processing becomes impossible – even with powerful hardware. However, with the benefit of Hadoop’s distributed computing power, calculation jobs can leverage hundreds of thousands of nodes. This allows Kylin to perform these calculations in parallel and merge the final result, thereby significantly reducing the processing time.

From relational to key-value

As an example, suppose there are several records stored in Hive tables that represent a relational structure. When the data volume grows very large – 10+ or even 100+ billions of rows – a question like “how many units were sold in the technology category in 2010 on the US site?” will produce a query with a large table scan and a long delay to get the answer. Since the values are fixed every time the query is run, it makes sense to calculate and store those values for further usage. This technique is called Relational to Key-Value (K-V) processing. The process will generate all of the dimension combinations and measured values shown in the example below, at the right side of the diagram. The middle columns of the diagram, from left to right, show how data is calculated by leveraging MapReduce for the large-volume data processing.


Kylin is based on this theory and is leveraging the Hadoop ecosystem to do the job for huge volumes of data:

  1. Read data from Hive (which is stored on HDFS)
  2. Run MapReduce jobs to pre-calculate
  3. Store cube data in HBase
  4. Leverage Zookeeper for job coordination


The following diagram shows the high-level architecture of Kylin.


This diagram illustrates how relational data becomes key-value data through the Cube Build Engine offline process. The yellow lines also illustrate the online analysis data flow. The data requests can originate from SQL submitted using a SQL-based tool, or even using third-party applications via Kylin’s RESTful services. The RESTful services call the Query Engine, which determines if the target dataset exists. If so, the engine directly accesses the target data and returns the result with sub-second latency. Otherwise, the engine is designed to route non-matching dataset queries to SQL on Hadoop, enabled on a Hadoop cluster such as Hive.

Following are descriptions of all of the components the Kylin platform includes.

  • Metadata Manager: Kylin is a metadata-driven application. The Metadata Manager is the key component that manages all metadata stored in Kylin, including the most important cube metadata. All other components rely on the Metadata Manager.
  • Job Engine: This engine is designed to handle all of the offline jobs including shell script, Java API, and MapReduce jobs. The Job Engine manages and coordinates all of the jobs in Kylin to make sure each job executes and handles failures.
  • Storage Engine: This engine manages the underlying storage – specifically the cuboids, which are stored as key-value pairs. The Storage Engine uses HBase – the best solution from the Hadoop ecosystem for leveraging an existing K-V system. Kylin can also be extended to support other K-V systems, such as Redis.
  • REST Server: The REST Server is an entry point for applications to develop against Kylin. Applications can submit queries, get results, trigger cube build jobs, get metadata, get user privileges, and so on.
  • ODBC Driver: To support third-party tools and applications – such as Tableau – we have built and open-sourced an ODBC Driver. The goal is to make it easy for users to onboard.
  • Query Engine: Once the cube is ready, the Query Engine can receive and parse user queries. It then interacts with other components to return the results to the user.

In Kylin, we are leveraging an open-source dynamic data management framework called Apache Calcite to parse SQL and plug in our code. The Calcite architecture is illustrated below. (Calcite was previously called Optiq, which was written by Julian Hyde and is now an Apache Incubator project.)


Kylin usage at eBay

At the time of open-sourcing Kylin, we already had several eBay business units using it in production. Our largest use case is the analysis of 12+ billion source records generating 14+ TB cubes. Its 90% query latency is less than 5 seconds. Now, our use cases target analysts and business users, who can access analytics and get results through the Tableau dashboard very easily – no more Hive query, shell command, and so on.

What’s next

  • Support TopN on high-cardinality dimension: The current MOLAP technology is not perfect when it comes to querying on a high-cardinality dimension – such as TopN on millions of distinct values in one column. Similar to search engines (as many researchers have pointed out), the inverted index is the reasonable mechanism to use to pre-build such results.
  • Support Hybrid OLAP (HOLAP): MOLAP is great to serve queries on historical data, but as more and more data needs to be processed in real time, there is a growing requirement to combine real-time/near-real-time and historical results for business decisions. Many in-memory technologies already work on Relational OLAP (ROLAP) to offer such capability. Kylin’s next generation will be a Hybrid OLAP (HOLAP) to combine MOLAP and ROLAP together and offer a single entry point for front-end queries.

Open source

Kylin has already been open-sourced to the community. To develop and grow an even stronger ecosystem around Kylin, we are currently working on proposing Kylin as an Apache Incubator project. With distinguished sponsors from the Hadoop developer community supporting Kylin, such as Owen O’Malley (Hortonworks co-founder and Apache member) and Julian Hyde (original author of Apache Calcite, also with Hortonworks), we believe that the greater open-source community can take Kylin to the next level.

We welcome everyone to contribute to Kylin. Please visit the Kylin web site for more details: http://kylin.io.

To begin with, we are looking for open-source contributions not only in the core code base, but also in the following areas:

  1. Shell Client
  2. RPC Server
  3. Job Scheduler
  4. Tools

For more details and to discuss these topics further, please follow us on twitter @KylinOLAP and join our Google group: https://groups.google.com/forum/#!forum/kylin-olap


Kylin has been deployed in production at eBay and is processing extremely large datasets. The platform has demonstrated great performance benefits and has proved to be a better way for analysts to leverage data on Hadoop with a more convenient approach using their favorite tool. We are pleased to open-source Kylin. We welcome feedback and suggestions, and we look forward to the involvement of the open-source community.




NoSQL Data Modeling

by Donovan Hsieh on 10/10/2014

in Data Infrastructure and Services

Data modeling for RDBMS has been a well-defined discipline for many years. Techniques like logical to physical mapping and normalization / de-normalization have been widely practiced by professionals, including novice users. However, with the recent emergence of NoSQL databases, data modeling is facing new challenges to its relevance. Generally speaking, NoSQL practitioners focus on physical data model design rather than the traditional conceptual / logical data model process for the following reasons:

  • Developer-centric mindset – With flexible schema (or schema-free) support in NoSQL databases, application developers typically assume data model design responsibility. They have been ingrained with the notion that the database schema is an integral part of application logic.
  • High-performance queries running in massive scale-out distributed environments – Contrary to traditional, centralized scale-up systems (including the RDBMS tier), modern applications run in distributed, scale-out environments. To accomplish scale-out, application developers are driven to tackle scalability and performance first through focused physical data model design, thus abandoning the traditional conceptual, logical, and physical data model design process.
  • Big and unstructured data – With its rigidly fixed schema and limited scale-out capability, the traditional RDBMS has long been criticized for its lack of support for big and unstructured data. By comparison, NoSQL databases were conceived from the beginning with the capability to store big and unstructured data using flexible schemas running in distributed scale-out environments.

In this blog post, we explore other important mindset changes in NoSQL data modeling: development agility through flexible schemas vs. database manageability; the business and data model design process; the role of RDBMS in NoSQL data modeling; NoSQL variations that affect data modeling; and visualization approaches for NoSQL logical and physical data modeling. We end the post with a peak into the NoSQL data modeling future.

Development agility vs. database manageability

One highly touted feature in today’s NoSQL is application development agility. Part of this agility is accomplished through flexible schemas, where developers have full control over how data is stored and organized in their NoSQL databases. Developers can create or modify database objects in application code on the fly without relying on DBA execution. The result is, indeed, increased application development and deployment agility.

However, the flexible schema is not without its challenges. For example, dynamically created database objects can cause unforeseen database management issues due to the lack of DBA oversight. Furthermore, unsupervised schema changes increase DBA challenges in diagnosing associated issues. Frequently, such troubleshooting requires the DBA to review application code written in programming languages (e.g., Java) rather than in RDBMS DDL (Data Definition Language) – a skill that most DBAs do not possess.

NoSQL business and data model design process

In old-school software engineering practice, sound business and (relational) data model designs are key to successful medium- to large-scale software projects. As NoSQL developers assume business / data model design ownership, another dilemma arises: data modeling tools. For example, traditional RDBMS logical and physical data models are governed and published by dedicated professionals using commercial tools, such as PowerDesigner or ER/Studio.

Given the nascent state of NoSQL technology, there isn’t a professional-quality data modeling tool for such tasks. It is not uncommon for stakeholders to review application source code in order to uncover data model information. This is a tall order for non-technical users such as business owners or product managers. Other approaches, like sampling actual data from production databases, can be equally laborious and tedious.

It is obvious that extensive investment in automation and tooling is required. To help alleviate this challenge, we recommend that NoSQL projects use the business and data model design process shown in the following diagram (illustrated with MongoDB’s document-centric model):


Figure 1

  • Business Requirements & Domain Model: At the high level, one can continue using database-agnostic methodologies, such as domain-driven design, to capture and define business requirements
  • Query Patterns & Application Object Model: After preliminary business requirements and the domain model are produced, one can work iteratively and in parallel to analyze top user access patterns and the application model, using UML class or object diagrams. With RDMS, applications can implement database access using either a declarative query (i.e., using a single SQL table join) or a navigational approach (i.e., walking individual tables embedded in application logic). The latter approach typically requires an object-relational mapping (ORM) layer to facilitate tedious plumbing work. By nature, almost all NoSQL databases belong to the latter category. MongoDB can support both approaches through the JSON Document model, SQL-subset query, and comprehensive secondary indexing capabilities.
  • JSON Document Model & MongoDB Collection / Document: This part is where native physical data modeling takes place. One has to understand the specific NoSQL product’s strengths and weaknesses in order to produce efficient schema designs and serve effective, high-performance queries. For example, modeling social network entities like followed and followers is very different from modeling online blogging applications. As such, social networking applications are best implemented using Graph NoSQL databases like Neo4j, while online blogging applications can be implemented using other flavors of NoSQL like MongoDB.

RDBMS data modeling influence on NoSQL

Interestingly enough, old-school RDBMS data modeling techniques still play a meaningful role for those who are new to NoSQL technology. Using document-centric MongoDB as an example, the following diagram illustrates how one can map a relational data model to a comparable MongoDB document-centric data model:


Figure 2

NoSQL data model variation

In the relational world, logical data models are reasonably portable among different RDBMS products. In a physical data model, design specifications such as storage clauses or non-standard SQL extensions might vary from vendor to vendor. Various SQL standards, such as SQL-92 and the latest SQL:2008 as defined by industry bodies like ANSI/ISO, can help application portability across different database platforms.

However, in the NoSQL world, physical data models vary dramatically among different NoSQL databases; there is no industry standard comparable to SQL-92 for RDBMS. Therefore, it helps to understand key differences in the various NoSQL database models:

  • Key-value stores – Collections comprised of unique keys having 1-n valid values
  • Column families – Distributed data stores in which a column consists of a unique key, values for the key, and a timestamp differentiating current from stale values
  • Document databases – Systems that store and manage documents and their metadata (type, title, author, creation/modification/deletion date, etc.)
  • Graph databases – Systems that use graph theory to represent and store data as nodes (people, business, accounts, or other entities), node properties, and edges (lines connecting nodes/properties to each other)

The following diagram illustrates the comparison landscape based on model complexity and scalability:


Figure 3

It is worth mentioning that for NoSQL data models, a natural evolutionary path exists from simple key-value stores to the highly complicated graph databases, as shown in the following diagram:


Figure 4

NoSQL data model visualization

For conceptual data models, diagramming techniques such as the Entity Relationship Diagram can continue to be used to model NoSQL applications. However, logical and physical NoSQL data modeling requires new thinking, due to each NoSQL product assuming a different native structure. One can intuitively use any of the following three visualization approaches, using a document-centric data model like MongoDB as an example:

  • Native visual representation of MongoDB collections with support for nested sub-documents (see Figure 2 above)

Pros – It naturally conveys a complex document model through an intuitive visual representation.
Cons – Without specialized tools support, visualization results in ad-hoc drawing using non-uniform conventions or notations.

  • Reverse engineering selected sample documents using JSON Designer (see Figure 5 below)

Pros – It can easily reverse engineer a hierarchical model into a visual representation from existing JSON documents stored in NoSQL databases like MongoDB.
Cons – As of this writing, JSON Designer is available only on iPhone / iPad. Furthermore, it does not include native DB objects, such as MongoDB indexes.


Figure 5

  • Traditional RDBMS data modeling tools like PowerDesigner (see Figure 6 below)

Pros – Commercial tools support is available.
Cons – it requires tedious manual preparation and diagram arrangement to represent complex and deeply nested document structure.


Figure 6

In a future post, we’ll cover specific data model visualization techniques for other NoSQL products such as Cassandra, which is based on the Column Family structure.

New NoSQL data modeling opportunities

Like any emerging technology, NoSQL will mature as it becomes mainstream. We envision the following new data modeling opportunities for NoSQL:

  • Reusable data model design patterns (some product-specific and some agnostic) to help reduce application development effort and cost
  • Unified NoSQL model repository to support different NoSQL products
  • Bi-directional, round-trip engineering support for (data) model-driven design processes and tools
  • Automated data model extraction from application source code
  • Automated code-model-data consistency validation and consistency conformance metrics
  • Strong control for application / data model change management, with proactive tracking and reconciliation between application code, embedded data models, and the actual data in the NoSQL databases


Don’t Build Pages, Build Modules

by Senthil Padmanabhan on 10/02/2014

in Software Engineering

We are in an interesting phase of rethinking frontend engineering across eBay Marketplaces, and this blog summarizes where we are heading.

Modular programming is a fundamental design technique that has been practiced since the dawn of software engineering. It is still the most recommended pattern for building maintainable software, and the Node.js community fully embraces this design philosophy. Most Node.js modules are created using smaller modules as building blocks to achieve the final goal. Now with Web Components gaining momentum, we decided to change our approach toward building frontend web applications.

Modularity was already in our frontend codebase, but only in the scope of a particular language. Most of our shared JavaScript (overlays, tabs, carousel, etc.) and CSS (buttons, grid, forms, icons, etc.) were written in a modular fashion. This is great, but when it comes to a page or a view, the thinking was still about building pages, as opposed to building UI modules. With this mindset, we found that as the complexity of pages grew, it became exponentially more difficult to maintain them. What we wanted was a simple way to divide a page into small and manageable pieces, and to develop each piece independently. This is when we came up with the notion, “Don’t build pages, build modules.”

Modular thinking

In general, everyone understands and agrees with the concept of frontend modules. But to make the concept a reality, we needed to deviate from our current style of web development.

Decomposition: First, we wanted to move away from the idea of directly building a page. Instead, when a requirement comes in the form of a page, we decompose it into logical UI modules. We do so recursively until a module becomes FIRST. This means a page is made up of a few top-level modules, which in turn are built from sub-modules, very similar to how JavaScript modules are built in the Node.js world. There are common styles and JavaScript (e.g., jQuery) that all modules depend on. These files together become a module of their own (e.g., a base module) and are added as dependencies of other modules. Engineers start working independently on these modules, and a page is nothing more than a grid that finally assembles them.


DOM encapsulation: We wanted all of our frontend modules to be associated with a DOM node and for that node to be the module’s root element. So we place all client-side JavaScript behavior like event binding, querying the DOM, jQuery plugin triggers, etc. within the scope of the module’s root element. This gives perfect encapsulation to our modules by making them restrictive and truly independent. Obviously we needed some sort of JavaScript abstraction to achieve this encapsulation, and we decided to go with the lightweight widgets functionality (named Marko Widgets) offered by RaptorJS. Marko widgets are a small module (~4 KB) providing a simple mechanism for instantiating widgets and binding them to DOM elements. Instantiated widgets are made into observables, and can also be destroyed when they need to be removed from the DOM. To bind a JavaScript module to a DOM node, we simply used the w-bind directive as shown below:

<div class="my-module" w-bind="./my-module-widget.js"> ... </div>

When rendered, the Marko widgets module tracks which modules are associated with which DOM nodes, and automatically binds the behavior after adding the HTML to the DOM. Some of our modules, like our tracking module, had JavaScript functionality, but no DOM node association. In those scenarios, we use the <noscript> tag to achieve encapsulation. With respect to CSS, we name-space all class names within a module with the root element’s class name, separated with ‘-’ such as gallery-title, gallery-thumbnail, etc.

Packaging: The next big question was how do we package the modules? Frontend package management has always been challenging, and is a hotly debated topic. Packaging for the same asset type is quite straightforward. For instance, in JavaScript once we nail down a module pattern (CommonJS, AMD, etc.), then packaging becomes easy with tools like browserify. The problem is when we need to bundle other asset types like CSS and markup templates. Here, our in-house Raptor Optimizer came to the rescue. The optimizer is a JavaScript module bundler very similar to browserify or webpack, but with a few differences that make it ideal for our module ecosystem. All it needs is an optimizer.json file in the module directory, to list out the CSS and markup template (dust or marko) dependencies. For JavaScript dependencies, the optimizer scans the source code in the current directory, and resolves them recursively. Finally, an ordered, de-duped bundle is inserted in the CSS and JavaScript slot of the page – for example:


Note that the markup templates will be included only when rendering on the client side. Including them otherwise will unnecessarily increase JavaScript file size.

File organization

Going modular also meant changing the way files were structured. Before applying modularity to the frontend codebase, teams would typically create separate top-level directories for JavaScript, CSS, images, fonts, etc. But with the new approach it made sense to group all files associated with a module under the same directory, and to use the module name as the directory name. This practice raised some concerns initially, mainly around violating a proven file structuring scheme, and around tooling changes related to bundling and CDN pushing. But engineers quickly came to an agreement, as the advantages clearly outweighed the disadvantages. The biggest benefit is that the new structure truly promoted module-level encapsulation:  all module-associated files live together and can be packaged.  In addition, any action on a module (deleting, renaming, refactoring, etc., which happen frequently in large codebases) becomes super easy.


Module communication

We wanted all of our modules to follow the Law of Demeter – meaning two independent modules cannot directly talk to each other. The obvious solution was to use an event bus for communication between client-side modules. We evaluated various eventing mechanisms, with the goal of having it centralized and also not introducing a large library dependency. Surprisingly, we settled on the eventing model that comes with jQuery itself. jQuery’s trigger, on, and off APIs do a fantastic job of abstracting out all eventing complexities, for both DOM and custom events. We wrote a small dispatcher wrapper, which handles interactions between modules by triggering and listening to events on the document element:

(function($) {
    'use strict';
    var $document = $(document.documentElement);

    // Create the dispatcher
    $.dispatcher = $.dispatcher || {};

    var dispatcherMethods = {
        trigger: function(event, data, elem) {
            // If element is provided trigger from element
            if(elem) {
                // Wrap in jQuery and call trigger                
                return $(elem).trigger(event, data);
            } else {
                return $document.trigger(event, data);

        on: function(event, callback, scope) {
            return $document.on(event, $.proxy(callback, scope || $document));

        off: function(event) {
            return $document.off(event);
    }; // dispatcherMethods end

    // Attach the dispatcher methods to $.dispatcher
    $.extend(true, $.dispatcher, dispatcherMethods);

Modules can now use the $.dispatcher to trigger and listen to custom events without having any knowledge about other modules. Another advantage of using the jQuery DOM-based eventing model is that we get all event dynamics (propagation and name-spacing) for free.

// Module 1 firing a custom event 'sliderSwiped'
$.dispatcher.trigger('sliderSwiped', {
    activeItemId: 1234

// Module 2 listening on 'sliderSwiped' and performing an action
$.dispatcher.on('sliderSwiped', function(evt, data) {

Some teams prefer to create a centralized mediator module to handle the communication. We leave that to engineers’ discretion.

Multiscreen and view model standardization

One of the biggest advantages of frontend modules is they perfectly fit in the multiscreen world. Flows change based on device dimensions, and making a page work either responsively or adaptively on all devices is not practical. But with modules, things fall in place. When engineers finalize the modules in a view, they also evaluate how they look and behave across various screen sizes. Based on this evaluation, the module name and associated view model JSON schema are agreed upon. But the implementation of the module is based upon the device. For some modules, just a responsive implementation is sufficient to work across all screens. For others, the design and interactions (touch or no-touch) would be completely different, thus requiring different implementations. However different the implementations may be, the module name and the view model powering it would be the same.

We indeed extended this concept to the native world, where iOS and Android apps also needed the same modules and view models. But the implementation is still native (Objective-C or Java) to the platform. All clients talk to the frontend servers, which are cognizant of the modules that a particular user agent needs and respond with the appropriate view model chunks. This approach gave us a perfect balance in terms of consistency and good user experience (by not compromising on the implementation). Companies like LinkedIn have already implemented a view-based JSON model that has proved successful. The granularity of the view model is decided by engineers and product managers together, depending on how much control they need over the module. The general guideline is to make the view model’s JSON as smart as possible and the modules dumb (or thin), thus providing a central place to control all clients.

Associated benefits

All of the other benefits of modular programming come for free:

  • Developer productivity – engineers can work in parallel on small contained pieces, resulting in a faster pace.
  • Unit testing – it has never been easier.
  • Debugging – it’s easy to nail down the problem, and even if one module is faulty others are still intact.

Finally, this whole approach takes us closer to the way the web is evolving. Our idea of bundling HTML, CSS, and JS to create an encapsulated UI module puts us on a fast track to adoption. We envision an ideal future where all of our views, across devices, are a bunch of web components.


As mentioned earlier, we are indeed in the process of rethinking frontend engineering at eBay, and modularization is one of the first steps resulting from that rethinking. Thanks to my colleagues Mahdi Pedramrazi and Patrick Steele-Idem for teaming up and pioneering this effort across the organization.

Frontend Engineer


Last year, our Gumtree Australia team started to use behavior-driven development in our agile development process. Initially, we searched for an open-source BDD framework to implement for BDD testing. Since we work on a web application, we wanted our automated BDD tests to exercise the application as a customer view, via a web interface. Of course, many web testing libraries do just that, including open-source tools such as Selenium and WebDriver (Selenium 2). However, we discovered that no existing BDD framework supports web automation testing directly.

We decided to simply integrate the open-source BDD tool JBehave with Selenium, but soon learned that this solution doesn’t completely fulfill the potential of automated BDD tests. It can’t drive the project development and documentation processes as much as we expected.

Different stakeholders need to view BDD tests at different levels. For instance, QA needs results to show how the application behaved under test. Upper managers are not so interested in the finer details, but rather want to see the number and complexity of the features defined and implemented so far, and if the project is still on track.

We needed a testing framework that could let us express and report on BDD tests at different levels, manage the stories and their scenarios effectively, and then drill down into the details as required. And so we started to develop a product that matches our needs. We named this product the Behavior Automation Framework (Beaf) and designed it to make the practice of behavior-driven development easier. Based on JBehave as well as more traditional tools like TestNg, Beaf includes a host of features to simplify writing automated BDD tests and interpreting the results.


Beaf’s extensions and utilities improve web testing on WebDriver/Selenium2 in four ways:

  • Supplying a Story web console, where PMs/QAs can manage all the existing stories, divide stories into different categories or groups, and create/edit stories using the correct BDD format.


  • Organizing web tests into reusable steps, and mapping them back to their original requirements and user stories.


  • Generating reports and documentation about BDD tests. Whenever a test case is executed, Beaf generates a report. Each report contains a narrative description of the test, including a short comment and screenshot for each step. The report provides not only information about the test results, but also documentation about how the scenarios under test have been implemented. Reports serve as living documentation, illustrating how an application implements the specified requirements.


  • Incorporating high-level summaries and aggregations of test results into reports. These overviews include how many stories and their scenarios have been tested by automated BDD tests. Taken as a set of objective metrics, test results show the relative size and progress of each feature being implemented.


When executing a test, whether it be with JUnit, jMock, or another framework, Beaf handles many of the Selenium 2/WebDriver infrastructure details. For example, Beaf can run cases cross-platform, including desktop (Firefox, Chrome), mobile (iPhone), and tablet (iPad). Testers can opt to open a new browser for each test, or use the same browser session for all of the tests in a class. The browser/device to be used can also be set in the Beaf configuration.


Using common steps provides a layer of abstraction between the behavior being tested and the way the web application implements that behavior. This level of abstraction makes it easier to manage changes in implementation details, because a desired behavior will generally change less frequently than the details of how it is to be implemented. Abstraction also allows implementation details to be centralized in one place.

@When("posting an Ad in the \"$category\" category")
public void postingAd(@Named("category") String category) throws Throwable
// 1. PageFactory will generate ad post pages under specified category.  
// 2. Page status will be sent to next step by ThreadLocal.
postAdPage.set(PostAdPageFactory.createPostAdPage(category, AdType.OFFER.name()));

In addition to test reports, Beaf supplies a very useful web module called the Beaf Dashboard, which provides a higher-level view of current status. The dashboard shows the state of all of the stories, both in terms of their relative priorities and in terms of how many P1/P2/P3 stories and scenarios are fully, partially, or not automated. This information gives a good idea of the amount of work involved in implementing different parts of the project. The dashboard also keeps track of test results over time, so that users can visualize in concrete terms the amount of work done so far versus the estimated amount of work remaining to be done.


Beaf facilitates QA people joining projects in stages. It provides PD/QA with the detailed information required to test and update code, while giving business managers and PMs the more high-level views and reports that suit their needs. However, the greater potential of Beaf is the ability to turn automated web tests into automated web acceptance testing, in the true spirit of BDD.


At eBay we run Hadoop clusters comprising thousands of nodes that are shared by thousands of users. We analyze data on these clusters to gain insights for improved customer experience. In this post, we look at distributing RPC resources fairly between heavy and light users, as well as mitigating denial of service attacks within Hadoop. By providing appropriate response times and increasing system availability, we offer a better Hadoop experience.

Problem: namenode slowdown

In our clusters, we frequently deal with slowness caused by heavy users, to the point of namenode latency increasing from less than a millisecond to more than half a second. In the past, we fixed this latency by finding and terminating the offending job. However, this reactive approach meant that damage had already been done—in extreme cases, we lost cluster operation for hours.

This slowness is a consequence of the original design of Hadoop. In Hadoop, the namenode is a single machine that coordinates HDFS operations in its namespace. These operations include getting block locations, listing directories, and creating files. The namenode receives HDFS operations as RPC calls and puts them in a FIFO call queue for execution by reader threads. The dataflow looks like this:

FIFO call queue

Though FIFO is fair in the sense of first-come-first-serve, it is unfair in the sense that users who perform more I/O operations on the namenode will be served more than users who perform less I/O. The result is the aforementioned latency increase.

We can see the effect of heavy users in the namenode auditlogs on days where we get support emails complaining about HDFS slowness:


Each color is a different user, and the area indicates call volume. Single users monopolizing cluster resources are a frequent cause of slowdown. With only one namenode and thousands of datanodes, any poorly written MapReduce job is a potential distributed denial-of-service attack.

Solution: quality of service

Taking inspiration from routers—some of which include QoS (quality of service) capabilities—we replaced the FIFO queue with a new type of queue, which we call the FairCallQueue.


The scheduler places incoming RPC calls into a number of queues based on the call volume of the user who made the call. The scheduler keeps track of recent calls, and prioritizes calls from lighter users over calls from heavy users.

The multiplexer controls the penalty of being in a low-priority queue versus a high-priority queue. It reads calls in a weighted round-robin fashion, preferring to read from high-priority queues and infrequently reading from the lowest-priority queues. This ensures that high-priority requests are served first, and prevents starvation of low-priority RPCs.

The multiplexer and scheduler are connected by a multi-level queue; together, these three form the FairCallQueue. In our tests at scale, we’ve found the queue is effective at preserving low latencies even in the face of overwhelming denial-of-service attacks on the namenode.

This plot shows the latency of a minority user during three runs of a FIFO queue (QoS disabled) and the FairCallQueue (QoS enabled). As expected, the latency is much lower when the FairCallQueue is active. (Note: spikes are caused by garbage collection pauses, which are a separate issue).


Open source and beyond

The 2.4 release of Apache Hadoop includes the prerequisites to namenode QoS. With this release, cluster owners can modify the implementation of the RPC call queue at runtime and choose to leverage the new FairCallQueue. You can try the patches on Apache’s JIRA: HADOOP-9640.

The FairCallQueue can be customized with other schedulers and multiplexers to enable new features. We are already investigating future improvements, such as weighting different RPC types for more intelligent scheduling and allowing users to manually control which queues certain users are scheduled into. In addition, there are features submitted from the open source community that build upon QoS, such as RPC client backoff and Fair Share queuing.

With namenode QoS in place, we have improved our users’ experience of our Hadoop clusters by providing faster and more uniform response times to well-behaved users while minimizing the impact of poorly written or badly behaved jobs. This in turn allows our analysts to be more productive and focus on the things that matter, like making your eBay experience a delightful one.

- Chris Li

eBay Global Data Infrastructure Analytics Team


The Platform and Infrastructure team at eBay Inc. is happy to announce the open-sourcing of Oink – a self-service solution to Apache Pig.

Pig and Hadoop overview

Apache Pig is a platform for analyzing large data sets. It uses a high-level language for expressing data analysis programs, coupled with the infrastructure for evaluating these programs. Pig abstracts the Map/Reduce paradigm, making it very easy for users to write complex tasks using Pig’s language, called Pig Latin. Because execution of tasks can be optimized automatically, Pig Latin allows users to focus on semantics rather than efficiency. Another key benefit of Pig Latin is extensibility:  users can do special-purpose processing by creating their own functions.

Apache Hadoop and Pig provide an excellent platform for extracting and analyzing data from very large application logs. At eBay, we on the Platform and Infrastructure team are responsible for storing TBs of logs that are generated every day from thousands of eBay application servers. Hadoop and Pig offer us an array of tools to search and view logs and to generate reports on application behavior. As the logs are available in Hadoop, engineers (users of applications) also have the ability to use Hadoop and Pig to do custom processing, such as Pig scripting to extract useful information.

The problem

Today, Pig is primarily used through the command line to spawn jobs. This model wasn’t well suited to the Platform team at eBay, as the cluster that housed the application logs was shared with other teams. This situation created a number of issues:

  • Governance – In a shared-cluster scenario, governance is critically important to attain. Pig scripts and requests of one customer should not impact those of other customers and stakeholders of the cluster. In addition, providing CLI access would make governance difficult in terms of controlling the number of job submissions.
  • Scalability – CLI access to all Pig customers created another challenge:  scalability. Onboarding customers takes time and is a cumbersome process.
  • Change management – No easy means existed to upgrade or modify common libraries.

Hence, we needed a solution that acted as a gateway to Pig job submission, provided QoS, and abstracted the user from cluster configuration.

The solution:  Oink

Oink solves the above challenges not only by allowing execution of Pig requests through a REST interface, but also by enabling users to register jars, view the status of Pig requests, view Pig request output, and even cancel a running Pig request. With the REST interface, the user has a cleaner way to submit Pig requests compared to CLI access. Oink serves as a single point of entry for Pig requests, thereby facilitating rate limiting and QoS enforcement for different customers.

oinkOink runs as a servlet inside a web container and allows users to run multiple requests in parallel within a single JVM instance. This capability was not supported initially, but rather required the help of the patch found in PIG-3866. This patch provides multi-tenant environment support so that different users can share the same instance.

With Oink, eBay’s Platform and Infrastructure team has been able to onboard 100-plus different use cases onto its cluster. Currently, more than 6000 Pig jobs run every day without any manual intervention from the team.

Special thanks to Vijay Samuel, Ruchir Shah, Mahesh Somani, and Raju Kolluru for open-sourcing Oink. If you have any queries related to Oink, please submit an issue through GitHub.

{ 1 comment }

Using Spark to Ignite Data Analytics

by eBay Global Data Infrastructure Analytics Team on 05/28/2014

in Data Infrastructure and Services,Machine Learning

At eBay we want our customers to have the best experience possible. We use data analytics to improve user experiences, provide relevant offers, optimize performance, and create many, many other kinds of value. One way eBay supports this value creation is by utilizing data processing frameworks that enable, accelerate, or simplify data analytics. One such framework is Apache Spark. This post describes how Apache Spark fits into eBay’s Analytic Data Infrastructure.


What is Apache Spark?

The Apache Spark web site describes Spark as “a fast and general engine for large-scale data processing.” Spark is a framework that enables parallel, distributed data processing. It offers a simple programming abstraction that provides powerful cache and persistence capabilities. The Spark framework can be deployed through Apache Mesos, Apache Hadoop via Yarn, or Spark’s own cluster manager. Developers can use the Spark framework via several programming languages including Java, Scala, and Python. Spark also serves as a foundation for additional data processing frameworks such as Shark, which provides SQL functionality for Hadoop.

Spark is an excellent tool for iterative processing of large datasets. One way Spark is suited for this type of processing is through its Resilient Distributed Dataset (RDD). In the paper titled Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing, RDDs are described as “…fault-tolerant, parallel data structures that let users explicitly persist intermediate results in memory, control their partitioning to optimize data placement, and manipulate them using a rich set of operators.” By using RDDs,  programmers can pin their large data sets to memory, thereby supporting high-performance, iterative processing. Compared to reading a large data set from disk for every processing iteration, the in-memory solution is obviously much faster.

The diagram below shows a simple example of using Spark to read input data from HDFS, perform a series of iterative operations against that data using RDDs, and write the subsequent output back to HDFS.


In the case of the first map operation into RDD(1), not all of the data could fit within the memory space allowed for RDDs. In such a case, the programmer is able to specify what should happen to the data that doesn’t fit. The options include spilling the computed data to disk and recreating it upon read. We can see in this example how each processing iteration is able to leverage memory for the reading and writing of its data. This method of leveraging memory is likely to be 100X faster than other methods that rely purely on disk storage for intermittent results.

Apache Spark at eBay

Today Spark is most commonly leveraged at eBay through Hadoop via Yarn. Yarn manages the Hadoop cluster’s resources and allows Hadoop to extend beyond traditional map and reduce jobs by employing Yarn containers to run generic tasks. Through the Hadoop Yarn framework, eBay’s Spark users are able to leverage clusters approaching the range of 2000 nodes, 100TB of RAM, and 20,000 cores.

The following example illustrates Spark on Hadoop via Yarn.


The user submits the Spark job to Hadoop. The Spark application master starts within a single Yarn container, then begins working with the Yarn resource manager to spawn Spark executors – as many as the user requested. These Spark executors will run the Spark application using the specified amount of memory and number of CPU cores. In this case, the Spark application is able to read and write to the cluster’s data residing in HDFS. This model of running Spark on Hadoop illustrates Hadoop’s growing ability to provide a singular, foundational platform for data processing over shared data.

The eBay analyst community includes a strong contingent of Scala users. Accordingly, many of eBay’s Spark users are writing their jobs in Scala. These jobs are supporting discovery through interrogation of complex data, data modelling, and data scoring, among other use cases. Below is a code snippet from a Spark Scala application. This application uses Spark’s machine learning library, MLlib, to cluster eBay’s sellers via KMeans. The seller attribute data is stored in HDFS.

 * read input files and turn into usable records
 var table = new SellerMetric()
 val model_data = sc.sequenceFile[Text,Text](
   v => parseRecord(v._2,table)
   v => v != null


 * build training data set from sample and summary data
 val train_data = sample_data.map( v =>
     i => zscore(v._2(i),sample_mean(i),sample_stddev(i))

 * train the model
 val model = KMeans.train(train_data,CLUSTERS,ITERATIONS)
 * score the data
 val results = grouped_model_data.map( 
   v => (
         i => zscore(v._2(i),sample_mean(i),sample_stddev(i))

In addition to  Spark Scala users, several folks at eBay have begun using Spark with Shark to accelerate their Hadoop SQL performance. Many of these Shark queries are easily running 5X faster than their Hive counterparts. While Spark at eBay is still in its early stages, usage is in the midst of expanding from experimental to everyday as the number of Spark users at eBay continues to accelerate.

The Future of Spark at eBay

Spark is helping eBay create value from its data, and so the future is bright for Spark at eBay. Our Hadoop platform team has started gearing up to formally support Spark on Hadoop. Additionally, we’re keeping our eyes on how Hadoop continues to evolve in its support for frameworks like Spark, how the community is able to use Spark to create value from data, and how companies like Hortonworks and Cloudera are incorporating Spark into their portfolios. Some groups within eBay are looking at spinning up their own Spark clusters outside of Hadoop. These clusters would either leverage more specialized hardware or be application-specific. Other folks are working on incorporating eBay’s already strong data platform language extensions into the Spark model to make it even easier to leverage eBay’s data within Spark. In the meantime, we will continue to see adoption of Spark increase at eBay. This adoption will be driven by chats in the hall, newsletter blurbs, product announcements, industry chatter, and Spark’s own strengths and capabilities.


In part I of this post we laid out in detail how to run a large Jenkins CI farm in Mesos. In this post we explore running the builds inside Docker containers and more:

  • Explain the motivation for using Docker containers for builds.
  • Show how to handle the case where the build itself is a Docker build.
  • Peek into how the Mesos 0.19 release is going to change Docker integration.
  • Walk through a Vagrant all-in-one-box setup so you can try things out.


Jenkins follows the master-slave model and is capable of launching tasks as remote Java processes on Mesos slave machines. Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications or frameworks. We can leverage the capabilities of Jenkins and Mesos to run a Jenkins slave process within a Docker container using Mesos as the resource manager.

Why use Docker containers?

This page gives a good picture of what Docker is all about.

At eBay Inc., we have several different build clusters. They are primarily partitioned due to a number of factors:  requirements to run different OS flavors (mostly RHEL and Ubuntu), software version conflicts, associated application dependencies, and special hardware. When using Mesos, we try to operate on a single cluster with heteregeneous workloads instead of having specialized clusters. Docker provides a good solution to isolate the different dependencies inside the container irrespective of the host setup where the Mesos slave is running, thereby helping us operate on a single cluster. Special hardware requirements can always be handled though slave attributes that the Jenkins plugin already supports. Overall, then, this setup scheme helps maintain consistent host images in the cluster, avoids having to introduce a wide combination of different flavors of Mesos slave hosts running, yet handles all the varied build dependencies within a container.

Now why support Docker-in-Docker setup?

When we started experimenting with running the builds in Docker containers, some of our teammates were working on enabling Docker images for applications. They posed the question, How do we support Docker build and push/pull operations within the Docker container used for the build? Valid point! So, we will explore two ways of handling this challenge. Many thanks to Jérôme Petazzoni from the Docker team for his guidance.

Environment setup

A Vagrant development VM setup demonstrates CI using Docker containers. This VM can be used for testing other frameworks like Chronos and Aurora; however, we will focus on the CI use of it with Marathon. The screenshots shown below have been taken from the Vagrant development environment setup, which runs a cluster of three Mesos masters, three Mesos slave instances, and one Marathon instance. (Marathon is a Mesos framework for long-running services. It provides a REST API for starting, stopping, and scaling services.) mesos1 marathon1 mesos2 mesos3

Running Jenkins slaves inside Mesos Docker containers requires the following ecosystem:

  1. Jenkins master server with the Mesos scheduler plugin installed (used for building Docker containers via CI jobs).
  2. Apache Mesos master server with at least one slave server .
  3. Mesos Docker Executor installed on all Mesos slave servers. Mesos slaves delegate execution of tasks within Docker containers to the Docker executor. (Note that integration with Docker changes with the Mesos 0.19 release, as explained in the miscellaneous section at the end of this post.)
  4. Docker installed on all slave servers (to automate the deployment of any application as a lightweight, portable, self-sufficient container that will run virtually anywhere).
  5. Docker build container image in the Docker registry.
  6. Marathon framework.

1. Creating the Jenkins master instance

We needed to first launch a standalone Jenkins master instance in Mesos via the Marathon framework.  We placed Jenkins plugins in the plugins directory, and included a default config.xml file with pre-configured settings. Jenkins was then launched by executing the jenkins.war file. Here is the directory structure that we used for launching the Jenkins master:

├── README.md
├── config.xml
├── hudson.model.UpdateCenter.xml
├── jenkins.war
├── jobs
├── nodeMonitors.xml
├── plugins
│   ├── mesos.hpi
│   └── saferestart.jpi
└── userContent
└── readme.txt
3 directories, 8 files

2. Launching the Jenkins master instance

Marathon launched the Jenkins master instance using the following command, also shown in the Marathon UI screenshots below. We zipped our Jenkins files and downloaded them for the job by using the URIs field in the UI; however, for demonstration purposes, below we show using a Git repository to achieve the same goal.

git clone https://github.com/ahunnargikar/jenkins-standalone && cd jenkins-standalone;
export JENKINS_HOME=$(pwd);
java -jar jenkins.war






3. Launching Jenkins slaves using the Mesos Docker executor


Here’s a sample supervisord startup configuration for a Docker image capable of executing Jenkins slave jobs:


command=/bin/bash -c "eval $JENKINS_COMMAND"

As you can see, Jenkins passed its slave launch command as an environment variable to the Docker container. The container then initialized the Jenkins slave process, which fulfilled the basic requirement for kicking off the Jenkins slave job.

This configuration was sufficient to launch regular builds within the Docker container of choice. Now let’s walk through the two options that we explored to run Docker operations for a CI build inside a Docker container. Strategy #1 required use of supervisord to control the Docker daemon process. For the default case (regular non-Docker builds) and strategy #2, supervisord was not required; one could simply pass the command directly to the Docker container.

3.1 Strategy #1 – Using an individual Docker-in-Docker (dind) setup on each Mesos slave

This strategy, inspired by this blog,  involved a dedicated Docker daemon inside the Docker container. The advantage of this approach was that we didn’t have a single Docker daemon handling a large number of container builds. On the flip side, each container was now absorbing the I/O overhead of downloading and duplicating all the AUFS file system layers.


The Docker-in-Docker container had to be launched in privileged mode (by including the “-privileged” option in the Mesos Docker executor code); otherwise, nested Docker containers wouldn’t work. Using this strategy, we ended up having two Docker executors:  one for launching Docker containers in non-privileged mode (/var/lib/mesos/executors/docker) and the other for launching Docker-in-Docker containers in privileged mode (/var/lib/mesos/executors/docker2). The supervisord process manager configuration was updated to run the Docker daemon process in addition to the Jenkins slave job process.


The following Docker-in-Docker image has been provided for demonstration purposes for testing out the multi-Docker setup:


In real life, the actual build container image would capture the build dependencies and base image flavor, in addition to the contents of the above dind image. The actual command that the Docker executor ran looked similar to this one:

docker run 
-cidfile /tmp/docker_cid.6c6bba3db72b7483 
-c 51 -m 302365697638 
-e JENKINS_COMMAND=wget -O slave.jar && java -DHUDSON_HOME=jenkins -server -Xmx256m -Xms16m -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -jar slave.jar  -jnlpUrl hashish/jenkins-dind

3.2 Strategy #2 – Using a shared Docker Setup on each Mesos slave

All of the Jenkins slaves running on a Mesos slave host could simply use a single Docker daemon for running their Docker containers, which was the default standard setup. This approach eliminated redundant network and disk I/O involved with downloading the AUFS file system layers. For example, all Java application projects could now reuse the same AUFS file system layers that contained the JDK, Tomcat, and other static Linux package dependencies. We lost isolation as far as the Docker daemon was concerned, but we gained a massive reduction in I/O and were able to leverage caching of build layers. This was the optimal strategy for our use case.


The Docker container mounted the host’s /var/run/docker.sock file descriptor as a shared volume so that its native Docker binary, located at /usr/local/bin/docker, could now communicate with the host server’s Docker daemon. So all Docker commands were now directly being executed by the host server’s Docker daemon. This eliminated the need for running individual Docker daemon processes on the Docker containers that were running on a Mesos slave server.

The following Docker image has been provided for demonstration purposes for a shared Docker setup. The actual build Docker container image of choice essentially just needed to execute the Docker binary via its CLI. We could even have mounted the Docker binary from the host server itself to the same end.


The actual command that the Docker executor ran looked similar to this:

docker run 
-cidfile /tmp/docker_cid.6c6bba3db72b7483 
-v /var/run/docker.sock:/var/run/docker.sock 
-c 51 -m 302365697638 
-e JENKINS_COMMAND=wget -O slave.jar && java -DHUDSON_HOME=jenkins -server -Xmx256m -Xms16m -XX:+UseConcMarkSweepGC -Djava.net.preferIPv4Stack=true -jar slave.jar  -jnlpUrl hashish/jenkins-dind-single

4. Specifying the cloud configuration for the Jenkins master

We then needed to configure the Jenkins master so that it would connect to the Mesos master server and start receiving resource offers, after which it could begin launching tasks on Mesos. The following screenshots illustrate how we configured the Jenkins master via its web administration UI.






Note: The Docker-specific configuration options above are not available in the stable release of the Mesos plugin. Major changes are underway in the upcoming Mesos 0.19.0 release, which will introduce the pluggable containerizer functionality. We decided to wait for 0.19.0 to be released before making a pull request for this feature. Instead, a modified .hpi plugin file was created from this Jenkins Mesos plugin branch and has been included in the Vagrant dev setup.



5. Creating the Jenkins Mesos Docker job

Now that the Jenkins scheduler had registered as a framework in Mesos, it started receiving resource offers from the Mesos master. The next step was to create a Jenkins job that would be launched on a Mesos slave whose resource offer satisfied the cloud configuration requirements.

5.1 Creating a Docker Tomcat 7 application container image

Jenkins first needed a Docker container base image that packaged the application code and dependencies as well as a web server. For demonstration purposes, here’s a sample Docker Tomcat 7 image created from this Github repository:


Every application’s Git repository would be expected to have its unique Dockerfile with whatever combination of Java/PHP/Node.js pre-installed in a base container. In the case of our Java apps, we simply built the .war file using Maven and then inserted it into the Docker image during build time. The Docker image was then tagged with the application name, version, and timestamp, and then uploaded into our private Docker registry.

5.2 Running a Jenkins Docker job

For demonstration purposes, the following example assumes that we are building a basic Java web application.







Once Jenkins built and uploaded the new application’s Docker image containing the war, dependencies, and other packages, this Docker image was launched in Mesos and scaled up or down to as many instances as required via the Marathon APIs.

Miscellaneous points

Our Docker integration with Mesos is going to be outdated soon with the 0.19 release. Our setup was against Mesos 0.17 and Docker 0.9.  You can read about the Mesos pluggable containerizer feature in this blog and in this ticket. The Mesosphere team is also working on the deimos project to integrate Docker with the external containerization approach. There is an old pull request against the Mesos Jenkins plugin to integrate containerization once it’s released. We will update our setup accordingly when this feature is rolled out. We’d like to add a disclaimer that the Docker integration in the above post hasn’t been tested at scale yet; we will do our due diligence once Mesos 0.19 and deimos are out.

For different build dependencies, you can define a build label for each. A merged PR already specifies the attributes per label. Hence, a Docker container image of choice can be added per build label.


This concludes the description of our journey, giving a good overview of how we ran a distributed CI solution on top of Mesos, utilizing resources in the most efficient manner and isolating build dependencies through Docker.


Copyright © 2011 eBay Inc. All Rights Reserved - User Agreement - Privacy Policy - Comment Policy