Author Archives: Senthil Padmanabhan

Browse eBay with Style and Speed

One of the top initiatives for eBay this year is to provide a compelling browse experience to our users. In a recent interview, Devin Wenig has given a good overview of why this matters to eBay. The idea is to leverage structured data and machine learning to allow users to shop across a whole spectrum of value, where some users might desire great savings, while others may want to focus on, say, best selling products.

When we started to design the experience, our first area of focus was mobile web. Similar to many other organizations, mobile web has been our highest growing sector. We wanted to launch the new browse experience on mobile web first, followed by desktop and native.

The core design principles of the new mobile web browse experience were to keep it simple, accessible, and fast, really fast. On the front-end side of things, we made a couple of choices to achieve this.

  • Lean and accessible — From the beginning we wanted the page to be as lean as possible. This meant keeping the HTML, CSS, and JS to a minimum. To achieve this goal, we followed a modular architecture and started building atomic components. Basically a page is a bunch of modules, and a module is built from other sub-modules, and so on. This practice enabled maximum code reuse, which in turn reduced the size of resources (CSS and JS) drastically. In addition, our style library enforced accessibility through CSS — by using ARIA attributes to define styles rather than just class names. This forces developers to write a11y-friendly markup from the beginning, instead of it being an afterthought. You can read more about it here.
  • Code with the platform — The web platform has evolved into a more developer friendly stack, and we wanted to leverage this aspect — code with the platform vs. coding against it. What this meant was that we could reduce the dependency on big libraries and frameworks and start using the native APIs to achieve the same. For instance, we tried to avoid jQuery for DOM manipulations and instead use the native DOM APIs. Similarly, we could use the fetch polyfill instead of $.ajax etc. The end result was a faster loading page that was also very responsive to user interactions. BTW, jQuery is still loaded in the page, because some of eBay platform specific code is dependent on it, and we are working towards removing the dependency altogether.

But our efforts did not stop there. The speed aspect was very critical for us, and we wanted to do more for speed. That is when we ran into AMP.

Experimenting with AMP

The AMP project was announced around the same time we started the initial brainstorming for browse. It seemed to resonate a lot with our own thinking on how we wanted to render the new experience. Although AMP was more tuned towards publisher-based content, it was still an open source project built using the open web. Also, a portion of the traffic to the new browse experience is going to be from search engines, which made it more promising to look into AMP. So we quickly pinged the AMP folks at Google and discussed the idea of building an AMP version for the browse experience, in addition to the normal mobile web pages. They were very supportive of it. This positive reaction encouraged us to start looking into AMP technology for the eCommerce world and in parallel develop an AMP version of browse.

Today we are proud to announce that the AMP version of the new browse experience is live, and about 8 million AMP-based browse nodes are available in production. Check out some of the popular queries in a mobile browser — Camera Drones and Sony PlayStation, for example. Basically adding amp/ to the path of any browse URL will render an AMP version (for example, non-AMP, AMP). We have not linked all of them from our regular (non-AMP) pages yet. This step is waiting on few pending tasks to be completed. For now, we have enabled this new browse experience only in mobile web. In the next couple of weeks, the desktop web experience will also be launched.

So how was the experience in implementing AMP for the eCommerce world? We have highlighted some of our learnings below.

What worked well?

  • Best practices — One of the good things about AMP is that at the end of the day it is a bunch of best practices for building mobile web pages. We were already following some of them, but adoption was scattered across various teams, each having its own preference. This initiative helped us consolidate the list and incorporate these best practices as a part of our regular development life cycle itself. This made our approach towards AMP more organic, rather than a forced function. The other good side effect of this is even our non-AMP pages become faster.
  • Less forking in code — This follows the previous point. Since we started following some of the AMP best practices for building regular pages, we were able to reuse most of the UI components between our non-AMP and AMP browse page. This resulted in less forking in code, which otherwise would have become a maintenance nightmare. Having said that, there is still some forking when it comes to JavaScript-based components, and we are still figuring out the best solution.
  • AMP Component list — Although the AMP project’s initial focus was more towards publisher-based content and news feeds, the AMP component list was still sufficient to build a basic product for viewing eCommerce pages. Users will not be able to do actions on items (such as “Add To Cart”), but they still get a solid browsing experience. The good news is that the list is getting better and growing day by day. Components like sidebar, carousel, and lightbox are critical in providing a compelling eCommerce experience.
  • Internal AMP platform — We have been thinking about leveraging the AMP ecosystem for our own search, similar to how Google handles AMP results. This plan is in very early stages of discussion, but the possibility of our search using AMP technology is very interesting.

The complex parts

  • Infrastructure components — To launch an eBay page to production, a lot of infrastructure components automatically come into play. These are things like Global header/footer, site speed beacon kit, experimentation library, and the analytics module. All of them have some amount of JavaScript, which immediately disqualifies them from being used in the AMP version. This adds complexity in development. We had to fork few infrastructure components to support the AMP guidelines. They had to go through a strict regression cycle before being published, which added delays. Also, our default front-end server pipeline had to be conditionally tweaked to exclude or swap certain modules. It was a good learning curve, and over time we have also replaced our early quick hacks with more robust and sustainable solutions.
  • Tracking — AMP provides user activity tracking through its amp-analytics component. amp-analytics can be configured in various ways, but it still was not sufficient for the granular tracking needs that eBay has. We also do stuff like session stitching, which needs cookie access. Creating an amp-analytics configuration to suit our needs was slowly becoming unmanageable. We need some enhancements in the component, which we are hoping to develop and commit to the project soon.

What’s next?

We are excited to partner with Google and everyone else participating on the AMP Project to close the gap in launching a full-fledged eCommerce experience in AMP. We have created a combined working group to tackle the gap, and we will be looking into these items and more.

  • Smart buttons — These enable us to do actions like “Add To Cart” and “Buy It Now” with authentication support.
  • Input elements — User interactive elements are critical to eCommerce experiences, be they simple search text boxes or checkboxes.
  • Advanced tracking — As mentioned earlier, we need more granular tracking for eBay, and so we have to figure out a way to achieve it.
  • A/B Testing — This will enable experimentation on AMP.

With items like these in place, AMP for eCommerce will soon start surfacing.

We will also be looking into creating a seamless transition from the AMP view to a regular page view, similar to what the Washington Post did using Service Workers. This will enable users to have a complete and delightful eBay experience without switching contexts.

We are also asked the question of if there is more focus towards web over native. The answer is NO. At eBay, we strongly believe that web and native do not compete each other. They indeed complement each other, and the combined ecosystem works very well for us. We will soon be launching these browse experiences in our native platforms.

We are on our path to making eBay the world’s first place to shop and this is a step towards it. Thanks to my colleague Suresh Ayyasamy, who partnered in implementing the AMP version of browse nodes and successfully rolling it to production.

Senthil

The Path to JavaScript Next

Like all other JavaScript enthusiasts around the globe, engineers at eBay were eagerly watching the space of ECMAScript 6. We have been playing around with the new syntax and trying it in a few internal projects, but nothing concrete happened at an organizational level. Once the spec was standardized in June 2015, however, there was a sudden excitement around it, and it was at both ends of the spectrum. Some engineers wanted to just dive in and use all ES6 features in production applications, whereas other engineers were very skeptical about the new shiny stuff and would rather spend time improving the product or fixing other low-hanging fruits. Obviously we needed a balance, and this post summarizes how we started adopting ES6 into eBay’s ecosystem. Our approach should resonate with engineers in big organizations with huge codebases that span across multiple domains.

Why ES6?

Before getting started, we needed to convince ourselves of the need for ES6. Yes, ES6 is cool, our industry peers are using it, and engineers can learn new stuff (good for their careers), but how effective is it as a technology, for our product as well as the overall organization? Also, engineers are comfortable with ES5 and things are going well. So why ES6?

  • Firstly, ES6 is official. It is no longer a group of proposals that browsers have eagerly implemented. The TC39 committee has approved the spec and the next version of JavaScript is finally a reality. As with any other programming language, we eventually need to move to the newer improved versions, so why not start now?
  • ES6 represents the biggest change to the language since its inception. Although there have been five earlier editions to the language, all of them were incremental revisions, none at this scale. Also, going forward there will be ongoing new additions to the language, some of which built on top of ES6 concepts. So the sooner we start adopting ES6, the more likely we can leverage the new powerful features.
  • One of the things that ES6 clearly succeeded in is making JavaScript very expressive. A good programming language should provide the syntax to express something from our mind into code as easily and quickly as possible. JavaScript is great in many aspects, but even with ES5, expressing a simple logic (such as default parameters) or a common pattern (such as Object.assign) required many lines of code or a custom abstraction or an external library. ES6 has now made it easy and straightforward, thus increasing productivity.
  • Finally, ES6 fixes some of the confusing aspects of the language. Even experienced JavaScript developers get confused by things like variable hoisting, the this keyword in callback functions, and prototypical inheritance. With ES6 offering much better ways of programming, these JavaScript intricacies will be gone. The end result is more elegant and bug-free code and easier training of new developers into JavaScript.

Now that we had some answers to the question “Why pursue ES6?” the next step was to deep dive into ES6 features and figure out which of them are most useful to our world. Which should we start with?

ES6 features

ES6 has introduced a large set of features to the current JavaScript world. Should we go ahead and embrace all of them in our application code or be selective in what we use? Here is where we needed to strike a balance and be cognizant of various aspects before adopting a feature: performance, debugging, maturity, day-to-day use cases, and roadmap (ES7 and beyond), for example. This analysis also boosted the confidence of engineers who were skeptical about the whole ES6 momentum.

At a high level, ES6 features can be grouped under two buckets: syntactic sugars and new features. Syntactic sugars are those which are technically possible now but require many lines of code or a custom abstraction or a library to achieve (such as classes and default parameters). New features, as the name suggests, are something newly introduced in ES6. The inspiration for the new feature could be from either a popular library or another language, or it could be a completely new feature (such as the Promise object, generators, and the Proxy object). With this background, we did some deep dives, followed by discussions with a few folks, and finally shortlisted the below set of ES6 features to get started on.

  • Block Scoping
  • Extended Parameter Handling: Default Parameters, Rest Parameters, and the Spread Operator
  • Arrow Functions
  • Template Literals
  • Destructuring
  • Promises

Reasoning

Below are some of the reasons why only the above features were shortlisted:

  • Common use case: The above list covers most of our day-to-day coding use cases. We went through a major portion of our critical code and observed that these were the missing patterns and features for which a library was used or sometimes handwritten. With the code in front of us, we were looking at the features that can be immediately applied, most of which made it to the final list. ES6 offers some powerful features like Proxies, Reflection, and Symbols. But as the saying goes, “With great power comes great responsibility”. These power features need to be used wisely to get the most benefit out of them. Also, use of these features in application code is very rare, and they are more suited for meta programming or building libraries.
  • Performance: The other important criterion for a feature to be shortlisted was performance, both native and transpiled (ES6 to ES5) performance. The Six Speed performance comparison was a good place to start with. In addition, we did a little bit of our own benchmarking. With all results in hand, the above features were selected. Most of the shortlisted features had identical performance when compared to the ES5 equivalent or just native ES6. Features like Map and Set are useful, but our tests indicated that they still need more maturity to be ready for prime time. Also, we did not feel the desperate need to use them right away.
  • Productivity and debugging: Before shortlisting the features, we tried using a lot of them in internal projects just to get a feel of the whole process. The features that we shortlisted were the ones that boosted our productivity tremendously after a short learning curve. Some of these features were so appealing that we immediately started missing them when writing in ES5. Another aspect to it is debugging. The shortlisted features were easy to debug even in the transpiled version, as most of them are simple wrappers that don’t affect the core logic.
  • Growing list: Lastly, this list is not final. It is an initial vetted set of ES6 features to get started and feel comfortable with (in terms of not bringing additional overhead to the site). As and when the need for other features arises, we will quickly do our checks and expand the list. To that point, the new ES6 module syntax is at the top of our next review. Other than just establishing an official syntax for JavaScript modules, the new module syntax also bring in tree-shaking capabilities, which is incredible. At eBay we are immersed in the CommonJS world of writing isomorphic JavaScript. Unfortunately, that means adopting a new module system will take time. Classes are another ES6 feature that we may start using soon, although there is not much excitement around it.

This exercise also helped us decide what ES6 features to avoid now. One of these is the new ES6 generators. Although they bring in great functionality, they are still not ready for prime time. In addition to performance problems, we feel that the upcoming Async/Await feature offers a better alternative to generators, providing a much cleaner syntax and readability. So for now we are a strict NO to generators.

Getting started

Now that we have a clear picture on ES6, the next thing was to put it in motion. The first obvious thing was to decide on a transpiler, and the unanimous choice was Babel. In the beginning, we will transpile all ES6 unconditionally to ES5. We know it is not ideal, as we are sending ES5 code even to browsers that support ES6. This is just a short-term solution until we figure out a smarter (browser-based or feature-based) transpiling mechanism, which is already in the works. The below steps highlight the things we have put in place to get started with ES6.

  • Our module bundler and asset pipeline tool, Lasso, was updated to support ES6 transpilation through Babel. For now, we will only transpile files with the .es6 extension. This was done as a preemptive action to avoid running the transpiler on the vast existing JavaScript code and introduce any production regression. We wanted to keep the production environment stable when working on ES6, and this was one way to achieve it. This also helps test the stability and speed of the transpilation process in various environments (dev, QA, and production) with a limited set of files. This restriction may be removed in the future as ES6 adoption grows. In the Node.js world, we do not transpile the code, as version 4.x supports almost all shortlisted features natively.
  • We created an ESLint config similar to eslint-config-airbnb, which will enforce ES6 coding guidelines and code styles for the shortlisted features. It is an integrated part of developer IDEs and CI jobs, and violations will be reported for each build. Using non-recommended features (for example, generators) will also be reported as errors. This is a critical step to guide engineers in the right direction when starting with ES6, as they get trained with the best practices from day one.
  • Our recommendation to developers newly starting with ES6 is to start writing unit tests with ES6. Unit testing seems to be a good place to try out and get comfortable with the various features and syntax of ES6. It can be used as a playground to ramp up in. Since unit tests are not shipped to production, developers can be more proactive and confident to try things that otherwise require caution. Once comfortable, you can get going with new application files created in ES6.
  • We did a couple of internal Brown Bags to spread ES6 awareness and ramp up developers on new features. This will also be added as a part of our new hire training. It will be an ongoing activity, with the training content updated with our latest recommendations.

What’s next?

  • As mentioned earlier, smart transpiling is something we want to set up in our build and release process. We do not have an exact answer yet, but it is at the top of our queue.
  • We will onboard more ES6 features (especially the new module syntax) as and when our criteria are met.
  • We will keep this exercise going to bring in newer JavaScript features as and when they reach stage 4.

This summarizes the path we took to bring ES6 into eBay. Our main goal was to organically adopt ES6, instead of it being a forcing function. Teams have slowly started using the new features and the feedback is very positive.

Lastly, two online resources “Understanding ECMAScript 6” and “Exploring ES6” have been incredibly helpful for our research and analysis. Huge thanks to Nicholas Zakas and Axel Rauschmayer for making it available.

Senthil

Packaging for Performance

An interesting topic of discussion in recent times has been around static resource (JS/CSS) packaging for web applications. Craig Silverstein’s and Rebecca Murphey’s write-ups on this topic provide great insights into the reality of packaging in today’s frontend engineering world. The main question that comes up is, should the strategy for JavaScript and CSS bundling (which is a current performance best practice) change when migrating to HTTP/2? Although in theory the benefit of bundling—which is to reduce the number of HTTP requests—becomes void with HTTP/2, the reality is that we are not there yet. The above articles are proof of that reality. At eBay we did similar research a few months back when prepping the site for HTTPS, and our findings were the same. In this post, I will outline the approach that we have taken towards packaging and the performance benefits of our approach.

The majority of eBay pages followed a naïve pattern for bundling resources. All the CSS and JavaScript for a page were bundled into one resource each, with the CSS included in the head tag and JS at the bottom. Though it was good in terms of reducing the number of HTTP requests, there were still opportunities to improve — the main one being a better use of browser caching. As users navigate through eBay pages, every unvisited page needs to download the whole JS and CSS, which includes the same core libraries (jQuery etc.) used in previous pages. With our plan towards moving to HTTPS (and HTTP/2), we knew this coarse-grained packaging would not be effective. Our study, and others’ studies, also indicated that avoiding bundling altogether and loading resources individually would still not be effective in terms of performance. We needed a balance, and that’s when we came up with our own packaging solution.

Inception

Our first step was to identify the core JS and CSS libraries used across all eBay pages and to aggregate them as one resource. To do this, we created an internal Node.js module called Inception. This module includes all the common JS and CSS modules and will be added as a dependent by each domain team (owners of the various eBay pages). The identified core JS libraries were jQuery, marko (templating engine), marko-widgets (UI component abstraction), and in-house analytics and tracking libraries. For CSS, we have our own library called Skin, from which we picked the core, button, icons, dialog, and form modules. The package bundler we use at eBay is Lasso. The Inception module, which plugs in along with Lasso, provides the following functionalities:

  • Enforces all domains (buying, selling, browse, checkout, etc.) to follow the exact version of the core JS and CSS libraries. Non-compliance will result in build failures.
  • Bundles the Inception resources as one URL with the same URL address across all domains. Examples are inception-hashcode.js and inception-hashcode.css.
  • Enables domain teams to still include any of the Inception JS/CSS libraries as a part of their own module dependencies. The Lasso optimizer will de-dupe libraries and ensure only one copy is sent to the browser. This functionality is critical for two reasons . First, we want to promote module-level encapsulation, so that when domain teams are building modules, they are free to add a dependency on a core library (say skin-button) without worrying about duplication. This also makes the module work standalone. Secondly, domain teams should not bear the overhead of tracking what is or isn’t present in Inception. They should be able to include any dependency they want, and the tooling can take care of the optimization.

Now with Inception in place, we started seeing these benefits:

  • Browser caching: One of the drawbacks mentioned earlier—bundling all resources as one URL—is poor leverage of browser caching. Inception fixes this drawback. Since the same URL is used across all domains for the core JS and CSS libraries (which BTW is the majority of the payload), browser caching is heavily utilized as users navigate through various eBay experiences. This caching is a massive improvement in terms of performance, especially for slow connections. In addition, with newer browser versions supporting code caching, we might also avoid the parse and compile times for the big Inception JS bundle.
  • Library consistency: Another problem that we saw in our previous bundling system was lack of consistency in core library versions used across domains. Since domains were maintaining the core libraries, users navigating from one domain to another might, for instance, get different versions of jQuery or button styles. The result is not only UI inconsistency but also implementation inconsistency across domains. This issue is also fixed with Inception, as it is a central place to manage core libraries.
  • Path to Progressive Web Apps: With all domain pages having the same core library dependencies, transition between them becomes easy, since only the application-specific JS and CSS has to be downloaded on each navigation. This approach will enable us to build our web apps using an Application Shell Architecture, thus paving the way to making eBay a Progressive Web App. We have already explored a similar route in the past (within a domain) using the Structured Page Fragments approach, and we have seen our perceived performance increase immensely.
  • Easy upgrade path: Finally, Inception also enables us to upgrade to newer versions of core libraries from a central place. Inception itself follows semantic versioning, so all domain teams depending on Inception will get the updates uniformly in a semantic manner. Upgrades were problematic previously, as we had to chase individual teams to do the upgrades manually.

Domains

Now with the core libraries being taken care of through Inception, what about the remaining resources in a page—i.e., application/domain-specific CSS and JS? For each domain we again came up with a packaging approach where we split the resources into two buckets: constants and variables.

  • Constants: The CSS and JS resources that remain the same on every request are bucketed as constants. These mainly pertain to the key UI components in each domain, which remain unaltered to various request parameters. The constant modules are bundled as one resource, and they again benefit from browser caching. When a user revisits a page, this bundle usually hits the browser cache and gets a performance advantage.
  • Variables: A small portion of the resources in each page vary based on request attributes. These variations might be due to experimentation, user sign-in status, business logic, etc. Such resources are bucketed as variables and have their own bundling, which happens at runtime. They will have the least cache-hit ratio and probably will need to be downloaded over the network for a new session.

Summary

To summarize, every page will have six resource bundles (3 for CSS and 3 for JS), and each bundle will have its own purpose. All URLs are hashed based on content; thus cache busting is automatically taken care of.

  1. Inception — bundles the core CSS and JS libraries. Highest in payload.
  2. Constants — bundles the unchanging application CSS and JS. Middle-range in payload.
  3. Variables — bundles the varying CSS and JS in an application. Least in payload.

In the current state, this packaging strategy seems to be the best fit for us in terms of performance. It has created the right balance between number of HTTP requests and browser caching. As we start migrating towards HTTP/2 next year, we will further evaluate this approach and try to come up with more fine-grained bundling solutions, and of course with performance being the key.