Author Archives: Senthil Padmanabhan

Experience the Lightning Bolt

 

Three months back we announced how we are transforming the shopping experience at eBay, enabling our users to browse with style and speed. Our goal was to provide an engaging experience not only to users who are within the eBay site, but also to mobile users accessing eBay from external platforms like Google and Twitter. This is where AMP technology comes into play. We implemented an AMP version for our new product browse experience, along with the regular mobile web pages, and launched them in June. At that time we did not make our AMP content discoverable to Google, as we had a few pending tasks to be completed. Also, AMP links surfaced in Google search results only for publisher-based content and not for eCommerce.

Things have changed now. Google announced that they are opening AMP beyond the news industry to include eCommerce, travel and so on. From our end, we wrapped up the pending items and linked the AMP pages from non-AMP pages to make them discoverable. Today we are happy to announce that users around the globe will start seeing eBay AMP links in Google search results and experience the lightning bolt — instant loading. We have close to 15 million AMP-based product browse pages, but not all will appear as AMP right away. This feature is being ramped up and will eventually surface. Check out some of the popular queries in a mobile browser — “iPhone 6 no contract” and “canon digital cameras,” for example. The AMP lightning bolt appears next to links as an indication. AMP for eCommerce is now a reality.

eBay AMP Product Browse Page eBay AMP link in Google search results (left); eBay AMP product browse page (right)

Between now and then

Following our initial launch in June, we did a couple of things to make AMP ready for prime time. We outline a few these efforts here.

Robust analytics system

Understanding how users interact with our pages is critical for us to provide the most optimized experience. The back-end system that powers the new product browse experience is designed in such a way that it constantly collects users’ screen activity, learns from it, and optimizes the experience for subsequent visits. For example, if users interact more often with a module that appears below the fold in the screen, then in future visits to the same browse page, that module will start appearing above the fold. Our non-AMP page has a custom analytics library that does the reporting to the back end.

AMP has a component (amp-analytics) for doing this. In our initial AMP launch, we used this component just to track page impressions. It provides a fairly exhaustive tracking mechanism. But what we wanted was more granular control at an element level, where each element dictates what it wants to track. We started working with the AMP team on this and came up with a spec. We went ahead implemented the spec and contributed it back to the open-source project. With the implementation in place, we were able to achieve a robust and advanced analytic system that reports user interactions like click, scroll, and visibility to our backend, which in turn optimizes the subsequent visits.

Feature parity

We mentioned in our previous blog that most of the code is shared between the AMP and non-AMP pages. Even with this code sharing, there were still small feature inconsistencies between the two versions. We closed these gaps, fixed the inconsistencies, and put a process in place to make sure they do not creep in. Having said that, there were certain UI components and behaviors that we were not able to achieve in the AMP version due to restrictions. Some of these components are eCommerce-specific. We are working with the AMP team to add them to the component list so everyone can benefit. A good example would be tabbed UI component, and there is already a feature request to get this implemented.

Streamlined build process

During the initial launch, we put manual effort into managing assets (CSS and JavaScript) between the AMP and non-AMP versions. In the AMP version, there should be no JavaScript, and all CSS should be inline, whereas in the non-AMP version, both CSS and JavaScript should be bundled and externalized. Doing this manually was not ideal. Our asset pipeline tool, Lasso, had a solution for this —  conditional dependencies. We created an AMP flag that gets initialized to true if the request is AMP and then set as a Lasso flag. The pipeline gets access to it and automatically bundles, externalizes, or inlines resources based on the conditions. This was a big time saver and ended up being very efficient.

The road ahead

We are not done yet; in fact, we are just getting started. We have a bunch of tasks lined up.

  • Beyond AMP — We know AMP pages are fast. But what about the subsequent pages the user visits? Currently when users click on a link in the AMP page, a new tab opens, and the destination page is loaded there. In our case, the mobile web version of the destination page is loaded. We want that experience also to be as fast and consistent as the AMP experience. There is an AMP component (amp-install-serviceworker) to achieve this goal, and our top priority is to leverage this utility and create a seamless transition from the AMP to the target page. We are also discussing with the Google team about how to avoid the new tab and continue the experience in the same window.
  • Cache freshness — AMP content is served from Google AMP cache, and the cache update policy can be found here. What this means to eBay is, for popular product queries, users always see fresh content. But for certain extremely rare queries, a few users may end up seeing stale content. While this is not a common scenario, there is an AMP component (amp-fresh) in the works to fix this. We will be integrating this component as soon as it is ready. In the meanwhile, we have a script that we manually run for a few products to update the AMP content in cache.
  • Unified version — Currently we have two versions of the new browse pages — AMP and non-AMP. The AMP version shows up to users searching in Google, and the non-AMP version shows up to users searching within eBay. Although both of them are highly optimized, look the same, and share most of the code, updating both versions is still a maintenance overhead. In addition, we always need to watch out for feature parity. In the future, based on how AMP pages are performing, we may choose to have one mobile version (AMP) and serve it to all platforms.

We are very excited to provide the AMP experience to our mobile users coming from Google. We have been playing with it for a while, and it is indeed lightning fast. Mobile browsing can be slow and sometimes frustrating, and this is where AMP comes in and guarantees a consistent and fast experience. We hope our users benefit from this new technology.

Senthil Padmanabhan | Principal Engineer at eBay

Browse eBay with Style and Speed

One of the top initiatives for eBay this year is to provide a compelling browse experience to our users. In a recent interview, Devin Wenig has given a good overview of why this matters to eBay. The idea is to leverage structured data and machine learning to allow users to shop across a whole spectrum of value, where some users might desire great savings, while others may want to focus on, say, best selling products.

When we started to design the experience, our first area of focus was mobile web. Similar to many other organizations, mobile web has been our highest growing sector. We wanted to launch the new browse experience on mobile web first, followed by desktop and native.

The core design principles of the new mobile web browse experience were to keep it simple, accessible, and fast, really fast. On the front-end side of things, we made a couple of choices to achieve this.

  • Lean and accessible — From the beginning we wanted the page to be as lean as possible. This meant keeping the HTML, CSS, and JS to a minimum. To achieve this goal, we followed a modular architecture and started building atomic components. Basically a page is a bunch of modules, and a module is built from other sub-modules, and so on. This practice enabled maximum code reuse, which in turn reduced the size of resources (CSS and JS) drastically. In addition, our style library enforced accessibility through CSS — by using ARIA attributes to define styles rather than just class names. This forces developers to write a11y-friendly markup from the beginning, instead of it being an afterthought. You can read more about it here.
  • Code with the platform — The web platform has evolved into a more developer friendly stack, and we wanted to leverage this aspect — code with the platform vs. coding against it. What this meant was that we could reduce the dependency on big libraries and frameworks and start using the native APIs to achieve the same. For instance, we tried to avoid jQuery for DOM manipulations and instead use the native DOM APIs. Similarly, we could use the fetch polyfill instead of $.ajax etc. The end result was a faster loading page that was also very responsive to user interactions. BTW, jQuery is still loaded in the page, because some of eBay platform specific code is dependent on it, and we are working towards removing the dependency altogether.

But our efforts did not stop there. The speed aspect was very critical for us, and we wanted to do more for speed. That is when we ran into AMP.

Experimenting with AMP

The AMP project was announced around the same time we started the initial brainstorming for browse. It seemed to resonate a lot with our own thinking on how we wanted to render the new experience. Although AMP was more tuned towards publisher-based content, it was still an open source project built using the open web. Also, a portion of the traffic to the new browse experience is going to be from search engines, which made it more promising to look into AMP. So we quickly pinged the AMP folks at Google and discussed the idea of building an AMP version for the browse experience, in addition to the normal mobile web pages. They were very supportive of it. This positive reaction encouraged us to start looking into AMP technology for the eCommerce world and in parallel develop an AMP version of browse.

Today we are proud to announce that the AMP version of the new browse experience is live, and about 8 million AMP-based browse nodes are available in production. Check out some of the popular queries in a mobile browser — Camera Drones and Sony PlayStation, for example. Basically adding amp/ to the path of any browse URL will render an AMP version (for example, non-AMP, AMP). We have not linked all of them from our regular (non-AMP) pages yet. This step is waiting on few pending tasks to be completed. For now, we have enabled this new browse experience only in mobile web. In the next couple of weeks, the desktop web experience will also be launched.

So how was the experience in implementing AMP for the eCommerce world? We have highlighted some of our learnings below.

What worked well?

  • Best practices — One of the good things about AMP is that at the end of the day it is a bunch of best practices for building mobile web pages. We were already following some of them, but adoption was scattered across various teams, each having its own preference. This initiative helped us consolidate the list and incorporate these best practices as a part of our regular development life cycle itself. This made our approach towards AMP more organic, rather than a forced function. The other good side effect of this is even our non-AMP pages become faster.
  • Less forking in code — This follows the previous point. Since we started following some of the AMP best practices for building regular pages, we were able to reuse most of the UI components between our non-AMP and AMP browse page. This resulted in less forking in code, which otherwise would have become a maintenance nightmare. Having said that, there is still some forking when it comes to JavaScript-based components, and we are still figuring out the best solution.
  • AMP Component list — Although the AMP project’s initial focus was more towards publisher-based content and news feeds, the AMP component list was still sufficient to build a basic product for viewing eCommerce pages. Users will not be able to do actions on items (such as “Add To Cart”), but they still get a solid browsing experience. The good news is that the list is getting better and growing day by day. Components like sidebar, carousel, and lightbox are critical in providing a compelling eCommerce experience.
  • Internal AMP platform — We have been thinking about leveraging the AMP ecosystem for our own search, similar to how Google handles AMP results. This plan is in very early stages of discussion, but the possibility of our search using AMP technology is very interesting.

The complex parts

  • Infrastructure components — To launch an eBay page to production, a lot of infrastructure components automatically come into play. These are things like Global header/footer, site speed beacon kit, experimentation library, and the analytics module. All of them have some amount of JavaScript, which immediately disqualifies them from being used in the AMP version. This adds complexity in development. We had to fork few infrastructure components to support the AMP guidelines. They had to go through a strict regression cycle before being published, which added delays. Also, our default front-end server pipeline had to be conditionally tweaked to exclude or swap certain modules. It was a good learning curve, and over time we have also replaced our early quick hacks with more robust and sustainable solutions.
  • Tracking — AMP provides user activity tracking through its amp-analytics component. amp-analytics can be configured in various ways, but it still was not sufficient for the granular tracking needs that eBay has. We also do stuff like session stitching, which needs cookie access. Creating an amp-analytics configuration to suit our needs was slowly becoming unmanageable. We need some enhancements in the component, which we are hoping to develop and commit to the project soon.

What’s next?

We are excited to partner with Google and everyone else participating on the AMP Project to close the gap in launching a full-fledged eCommerce experience in AMP. We have created a combined working group to tackle the gap, and we will be looking into these items and more.

  • Smart buttons — These enable us to do actions like “Add To Cart” and “Buy It Now” with authentication support.
  • Input elements — User interactive elements are critical to eCommerce experiences, be they simple search text boxes or checkboxes.
  • Advanced tracking — As mentioned earlier, we need more granular tracking for eBay, and so we have to figure out a way to achieve it.
  • A/B Testing — This will enable experimentation on AMP.

With items like these in place, AMP for eCommerce will soon start surfacing.

We will also be looking into creating a seamless transition from the AMP view to a regular page view, similar to what the Washington Post did using Service Workers. This will enable users to have a complete and delightful eBay experience without switching contexts.

We are also asked the question of if there is more focus towards web over native. The answer is NO. At eBay, we strongly believe that web and native do not compete each other. They indeed complement each other, and the combined ecosystem works very well for us. We will soon be launching these browse experiences in our native platforms.

We are on our path to making eBay the world’s first place to shop and this is a step towards it. Thanks to my colleague Suresh Ayyasamy, who partnered in implementing the AMP version of browse nodes and successfully rolling it to production.

Senthil

The Path to JavaScript Next

Like all other JavaScript enthusiasts around the globe, engineers at eBay were eagerly watching the space of ECMAScript 6. We have been playing around with the new syntax and trying it in a few internal projects, but nothing concrete happened at an organizational level. Once the spec was standardized in June 2015, however, there was a sudden excitement around it, and it was at both ends of the spectrum. Some engineers wanted to just dive in and use all ES6 features in production applications, whereas other engineers were very skeptical about the new shiny stuff and would rather spend time improving the product or fixing other low-hanging fruits. Obviously we needed a balance, and this post summarizes how we started adopting ES6 into eBay’s ecosystem. Our approach should resonate with engineers in big organizations with huge codebases that span across multiple domains.

Why ES6?

Before getting started, we needed to convince ourselves of the need for ES6. Yes, ES6 is cool, our industry peers are using it, and engineers can learn new stuff (good for their careers), but how effective is it as a technology, for our product as well as the overall organization? Also, engineers are comfortable with ES5 and things are going well. So why ES6?

  • Firstly, ES6 is official. It is no longer a group of proposals that browsers have eagerly implemented. The TC39 committee has approved the spec and the next version of JavaScript is finally a reality. As with any other programming language, we eventually need to move to the newer improved versions, so why not start now?
  • ES6 represents the biggest change to the language since its inception. Although there have been five earlier editions to the language, all of them were incremental revisions, none at this scale. Also, going forward there will be ongoing new additions to the language, some of which built on top of ES6 concepts. So the sooner we start adopting ES6, the more likely we can leverage the new powerful features.
  • One of the things that ES6 clearly succeeded in is making JavaScript very expressive. A good programming language should provide the syntax to express something from our mind into code as easily and quickly as possible. JavaScript is great in many aspects, but even with ES5, expressing a simple logic (such as default parameters) or a common pattern (such as Object.assign) required many lines of code or a custom abstraction or an external library. ES6 has now made it easy and straightforward, thus increasing productivity.
  • Finally, ES6 fixes some of the confusing aspects of the language. Even experienced JavaScript developers get confused by things like variable hoisting, the this keyword in callback functions, and prototypical inheritance. With ES6 offering much better ways of programming, these JavaScript intricacies will be gone. The end result is more elegant and bug-free code and easier training of new developers into JavaScript.

Now that we had some answers to the question “Why pursue ES6?” the next step was to deep dive into ES6 features and figure out which of them are most useful to our world. Which should we start with?

ES6 features

ES6 has introduced a large set of features to the current JavaScript world. Should we go ahead and embrace all of them in our application code or be selective in what we use? Here is where we needed to strike a balance and be cognizant of various aspects before adopting a feature: performance, debugging, maturity, day-to-day use cases, and roadmap (ES7 and beyond), for example. This analysis also boosted the confidence of engineers who were skeptical about the whole ES6 momentum.

At a high level, ES6 features can be grouped under two buckets: syntactic sugars and new features. Syntactic sugars are those which are technically possible now but require many lines of code or a custom abstraction or a library to achieve (such as classes and default parameters). New features, as the name suggests, are something newly introduced in ES6. The inspiration for the new feature could be from either a popular library or another language, or it could be a completely new feature (such as the Promise object, generators, and the Proxy object). With this background, we did some deep dives, followed by discussions with a few folks, and finally shortlisted the below set of ES6 features to get started on.

  • Block Scoping
  • Extended Parameter Handling: Default Parameters, Rest Parameters, and the Spread Operator
  • Arrow Functions
  • Template Literals
  • Destructuring
  • Promises

Reasoning

Below are some of the reasons why only the above features were shortlisted:

  • Common use case: The above list covers most of our day-to-day coding use cases. We went through a major portion of our critical code and observed that these were the missing patterns and features for which a library was used or sometimes handwritten. With the code in front of us, we were looking at the features that can be immediately applied, most of which made it to the final list. ES6 offers some powerful features like Proxies, Reflection, and Symbols. But as the saying goes, “With great power comes great responsibility”. These power features need to be used wisely to get the most benefit out of them. Also, use of these features in application code is very rare, and they are more suited for meta programming or building libraries.
  • Performance: The other important criterion for a feature to be shortlisted was performance, both native and transpiled (ES6 to ES5) performance. The Six Speed performance comparison was a good place to start with. In addition, we did a little bit of our own benchmarking. With all results in hand, the above features were selected. Most of the shortlisted features had identical performance when compared to the ES5 equivalent or just native ES6. Features like Map and Set are useful, but our tests indicated that they still need more maturity to be ready for prime time. Also, we did not feel the desperate need to use them right away.
  • Productivity and debugging: Before shortlisting the features, we tried using a lot of them in internal projects just to get a feel of the whole process. The features that we shortlisted were the ones that boosted our productivity tremendously after a short learning curve. Some of these features were so appealing that we immediately started missing them when writing in ES5. Another aspect to it is debugging. The shortlisted features were easy to debug even in the transpiled version, as most of them are simple wrappers that don’t affect the core logic.
  • Growing list: Lastly, this list is not final. It is an initial vetted set of ES6 features to get started and feel comfortable with (in terms of not bringing additional overhead to the site). As and when the need for other features arises, we will quickly do our checks and expand the list. To that point, the new ES6 module syntax is at the top of our next review. Other than just establishing an official syntax for JavaScript modules, the new module syntax also bring in tree-shaking capabilities, which is incredible. At eBay we are immersed in the CommonJS world of writing isomorphic JavaScript. Unfortunately, that means adopting a new module system will take time. Classes are another ES6 feature that we may start using soon, although there is not much excitement around it.

This exercise also helped us decide what ES6 features to avoid now. One of these is the new ES6 generators. Although they bring in great functionality, they are still not ready for prime time. In addition to performance problems, we feel that the upcoming Async/Await feature offers a better alternative to generators, providing a much cleaner syntax and readability. So for now we are a strict NO to generators.

Getting started

Now that we have a clear picture on ES6, the next thing was to put it in motion. The first obvious thing was to decide on a transpiler, and the unanimous choice was Babel. In the beginning, we will transpile all ES6 unconditionally to ES5. We know it is not ideal, as we are sending ES5 code even to browsers that support ES6. This is just a short-term solution until we figure out a smarter (browser-based or feature-based) transpiling mechanism, which is already in the works. The below steps highlight the things we have put in place to get started with ES6.

  • Our module bundler and asset pipeline tool, Lasso, was updated to support ES6 transpilation through Babel. For now, we will only transpile files with the .es6 extension. This was done as a preemptive action to avoid running the transpiler on the vast existing JavaScript code and introduce any production regression. We wanted to keep the production environment stable when working on ES6, and this was one way to achieve it. This also helps test the stability and speed of the transpilation process in various environments (dev, QA, and production) with a limited set of files. This restriction may be removed in the future as ES6 adoption grows. In the Node.js world, we do not transpile the code, as version 4.x supports almost all shortlisted features natively.
  • We created an ESLint config similar to eslint-config-airbnb, which will enforce ES6 coding guidelines and code styles for the shortlisted features. It is an integrated part of developer IDEs and CI jobs, and violations will be reported for each build. Using non-recommended features (for example, generators) will also be reported as errors. This is a critical step to guide engineers in the right direction when starting with ES6, as they get trained with the best practices from day one.
  • Our recommendation to developers newly starting with ES6 is to start writing unit tests with ES6. Unit testing seems to be a good place to try out and get comfortable with the various features and syntax of ES6. It can be used as a playground to ramp up in. Since unit tests are not shipped to production, developers can be more proactive and confident to try things that otherwise require caution. Once comfortable, you can get going with new application files created in ES6.
  • We did a couple of internal Brown Bags to spread ES6 awareness and ramp up developers on new features. This will also be added as a part of our new hire training. It will be an ongoing activity, with the training content updated with our latest recommendations.

What’s next?

  • As mentioned earlier, smart transpiling is something we want to set up in our build and release process. We do not have an exact answer yet, but it is at the top of our queue.
  • We will onboard more ES6 features (especially the new module syntax) as and when our criteria are met.
  • We will keep this exercise going to bring in newer JavaScript features as and when they reach stage 4.

This summarizes the path we took to bring ES6 into eBay. Our main goal was to organically adopt ES6, instead of it being a forcing function. Teams have slowly started using the new features and the feedback is very positive.

Lastly, two online resources “Understanding ECMAScript 6” and “Exploring ES6” have been incredibly helpful for our research and analysis. Huge thanks to Nicholas Zakas and Axel Rauschmayer for making it available.

Senthil