Machine Translation: The Basics of Quality Estimation

 A good point to make before we go into more detail on QE is the difference between evaluation and estimation. You can evaluate the quality of MT output in two main ways: human evaluation (a person will check the translation and provide feedback) and automatic evaluation (there are different methods that can provide a score on the translation quality without human intervention).

Traditionally, automatically evaluating the quality of any given MT output has required a reference translation created by a human translator. The differences and similarities between the MT output and the reference translation can then be turned into a score to determine the quality of said output. This is the approach followed by certain methods, such as BLEU and NIST.

 The main differentiator of quality estimation is that it does not require a human reference translation.

QE is a prediction of the quality based on certain features. These features can be, for example, the number of noun or prepositional phrases in the source and target (and their difference), the number of named entities (names of places, people, companies, etc.), and many more. With these features, using techniques like machine learning, a QE model can be created to obtain a score that represents the estimation of the translation quality.

At eBay, we use MT to translate search queries, item titles and item descriptions; an earlier post discussed our work with search queries. To train our MT systems, we work with vendors that help us post-edit content. Due to the challenging nature of our content (user-generated, diversity of categories, millions of listings, etc.), a method to estimate the level of effort required for post-editing definitely adds value. QE can help you obtain important information on this effort in an automated manner. For example, one can estimate how many segments have a very low-quality translation and could be just discarded instead of post-edited.

So, what can you do with the help of QE? First and foremost, you can estimate the quality of translations at the segment and file level. Segment-level scores can help you target post-editing, focusing only on content that makes sense to post-edit. You can also estimate post-editing effort/time – it would be rather safe to assume that segments with a low-quality score take more time to post-edit. It is also possible to compare MT systems based on QE scores and see which one performs better. This last application example is especially helpful if you are trying to decide which engine you should use, or whether a new version of an engine is working better than its predecessor.

Announcing Marko v3: From HTML to HTML-JS

MarkoJS LogoMarko is one of the fastest, lightest, and most powerful HTML templating engines for Node.js and the browser, and we are very pleased to see a healthy and growing community. Marko has been downloaded over 100k times in the first few months of 2016, and the project has a very active Gitter chat room. We are excited to announce some huge improvements to the Marko templating engine as part of the v3 release.

Marko v3 introduces a new HTML-JS syntax and a new parser that makes Marko more intuitive. Marko’s clean, HTML-based syntax has been a strength, but over time it became clear that using a strict HTML parser was actually putting unnecessary constraints on the Marko language that negatively impacted code readability and usability. The new HTML-JS syntax that ships with Marko v3 breaks away from the limitations associated with the standard HTML syntax while still maintaining the look and feel of HTML.

Read more on the MarkoJS blog.

The Path to JavaScript Next

Like all other JavaScript enthusiasts around the globe, engineers at eBay were eagerly watching the space of ECMAScript 6. We have been playing around with the new syntax and trying it in a few internal projects, but nothing concrete happened at an organizational level. Once the spec was standardized in June 2015, however, there was a sudden excitement around it, and it was at both ends of the spectrum. Some engineers wanted to just dive in and use all ES6 features in production applications, whereas other engineers were very skeptical about the new shiny stuff and would rather spend time improving the product or fixing other low-hanging fruits. Obviously we needed a balance, and this post summarizes how we started adopting ES6 into eBay’s ecosystem. Our approach should resonate with engineers in big organizations with huge codebases that span across multiple domains.

Why ES6?

Before getting started, we needed to convince ourselves of the need for ES6. Yes, ES6 is cool, our industry peers are using it, and engineers can learn new stuff (good for their careers), but how effective is it as a technology, for our product as well as the overall organization? Also, engineers are comfortable with ES5 and things are going well. So why ES6?

  • Firstly, ES6 is official. It is no longer a group of proposals that browsers have eagerly implemented. The TC39 committee has approved the spec and the next version of JavaScript is finally a reality. As with any other programming language, we eventually need to move to the newer improved versions, so why not start now?
  • ES6 represents the biggest change to the language since its inception. Although there have been five earlier editions to the language, all of them were incremental revisions, none at this scale. Also, going forward there will be ongoing new additions to the language, some of which built on top of ES6 concepts. So the sooner we start adopting ES6, the more likely we can leverage the new powerful features.
  • One of the things that ES6 clearly succeeded in is making JavaScript very expressive. A good programming language should provide the syntax to express something from our mind into code as easily and quickly as possible. JavaScript is great in many aspects, but even with ES5, expressing a simple logic (such as default parameters) or a common pattern (such as Object.assign) required many lines of code or a custom abstraction or an external library. ES6 has now made it easy and straightforward, thus increasing productivity.
  • Finally, ES6 fixes some of the confusing aspects of the language. Even experienced JavaScript developers get confused by things like variable hoisting, the this keyword in callback functions, and prototypical inheritance. With ES6 offering much better ways of programming, these JavaScript intricacies will be gone. The end result is more elegant and bug-free code and easier training of new developers into JavaScript.

Now that we had some answers to the question “Why pursue ES6?” the next step was to deep dive into ES6 features and figure out which of them are most useful to our world. Which should we start with?

ES6 features

ES6 has introduced a large set of features to the current JavaScript world. Should we go ahead and embrace all of them in our application code or be selective in what we use? Here is where we needed to strike a balance and be cognizant of various aspects before adopting a feature: performance, debugging, maturity, day-to-day use cases, and roadmap (ES7 and beyond), for example. This analysis also boosted the confidence of engineers who were skeptical about the whole ES6 momentum.

At a high level, ES6 features can be grouped under two buckets: syntactic sugars and new features. Syntactic sugars are those which are technically possible now but require many lines of code or a custom abstraction or a library to achieve (such as classes and default parameters). New features, as the name suggests, are something newly introduced in ES6. The inspiration for the new feature could be from either a popular library or another language, or it could be a completely new feature (such as the Promise object, generators, and the Proxy object). With this background, we did some deep dives, followed by discussions with a few folks, and finally shortlisted the below set of ES6 features to get started on.

  • Block Scoping
  • Extended Parameter Handling: Default Parameters, Rest Parameters, and the Spread Operator
  • Arrow Functions
  • Template Literals
  • Destructuring
  • Promises

Reasoning

Below are some of the reasons why only the above features were shortlisted:

  • Common use case: The above list covers most of our day-to-day coding use cases. We went through a major portion of our critical code and observed that these were the missing patterns and features for which a library was used or sometimes handwritten. With the code in front of us, we were looking at the features that can be immediately applied, most of which made it to the final list. ES6 offers some powerful features like Proxies, Reflection, and Symbols. But as the saying goes, “With great power comes great responsibility”. These power features need to be used wisely to get the most benefit out of them. Also, use of these features in application code is very rare, and they are more suited for meta programming or building libraries.
  • Performance: The other important criterion for a feature to be shortlisted was performance, both native and transpiled (ES6 to ES5) performance. The Six Speed performance comparison was a good place to start with. In addition, we did a little bit of our own benchmarking. With all results in hand, the above features were selected. Most of the shortlisted features had identical performance when compared to the ES5 equivalent or just native ES6. Features like Map and Set are useful, but our tests indicated that they still need more maturity to be ready for prime time. Also, we did not feel the desperate need to use them right away.
  • Productivity and debugging: Before shortlisting the features, we tried using a lot of them in internal projects just to get a feel of the whole process. The features that we shortlisted were the ones that boosted our productivity tremendously after a short learning curve. Some of these features were so appealing that we immediately started missing them when writing in ES5. Another aspect to it is debugging. The shortlisted features were easy to debug even in the transpiled version, as most of them are simple wrappers that don’t affect the core logic.
  • Growing list: Lastly, this list is not final. It is an initial vetted set of ES6 features to get started and feel comfortable with (in terms of not bringing additional overhead to the site). As and when the need for other features arises, we will quickly do our checks and expand the list. To that point, the new ES6 module syntax is at the top of our next review. Other than just establishing an official syntax for JavaScript modules, the new module syntax also bring in tree-shaking capabilities, which is incredible. At eBay we are immersed in the CommonJS world of writing isomorphic JavaScript. Unfortunately, that means adopting a new module system will take time. Classes are another ES6 feature that we may start using soon, although there is not much excitement around it.

This exercise also helped us decide what ES6 features to avoid now. One of these is the new ES6 generators. Although they bring in great functionality, they are still not ready for prime time. In addition to performance problems, we feel that the upcoming Async/Await feature offers a better alternative to generators, providing a much cleaner syntax and readability. So for now we are a strict NO to generators.

Getting started

Now that we have a clear picture on ES6, the next thing was to put it in motion. The first obvious thing was to decide on a transpiler, and the unanimous choice was Babel. In the beginning, we will transpile all ES6 unconditionally to ES5. We know it is not ideal, as we are sending ES5 code even to browsers that support ES6. This is just a short-term solution until we figure out a smarter (browser-based or feature-based) transpiling mechanism, which is already in the works. The below steps highlight the things we have put in place to get started with ES6.

  • Our module bundler and asset pipeline tool, Lasso, was updated to support ES6 transpilation through Babel. For now, we will only transpile files with the .es6 extension. This was done as a preemptive action to avoid running the transpiler on the vast existing JavaScript code and introduce any production regression. We wanted to keep the production environment stable when working on ES6, and this was one way to achieve it. This also helps test the stability and speed of the transpilation process in various environments (dev, QA, and production) with a limited set of files. This restriction may be removed in the future as ES6 adoption grows. In the Node.js world, we do not transpile the code, as version 4.x supports almost all shortlisted features natively.
  • We created an ESLint config similar to eslint-config-airbnb, which will enforce ES6 coding guidelines and code styles for the shortlisted features. It is an integrated part of developer IDEs and CI jobs, and violations will be reported for each build. Using non-recommended features (for example, generators) will also be reported as errors. This is a critical step to guide engineers in the right direction when starting with ES6, as they get trained with the best practices from day one.
  • Our recommendation to developers newly starting with ES6 is to start writing unit tests with ES6. Unit testing seems to be a good place to try out and get comfortable with the various features and syntax of ES6. It can be used as a playground to ramp up in. Since unit tests are not shipped to production, developers can be more proactive and confident to try things that otherwise require caution. Once comfortable, you can get going with new application files created in ES6.
  • We did a couple of internal Brown Bags to spread ES6 awareness and ramp up developers on new features. This will also be added as a part of our new hire training. It will be an ongoing activity, with the training content updated with our latest recommendations.

What’s next?

  • As mentioned earlier, smart transpiling is something we want to set up in our build and release process. We do not have an exact answer yet, but it is at the top of our queue.
  • We will onboard more ES6 features (especially the new module syntax) as and when our criteria are met.
  • We will keep this exercise going to bring in newer JavaScript features as and when they reach stage 4.

This summarizes the path we took to bring ES6 into eBay. Our main goal was to organically adopt ES6, instead of it being a forcing function. Teams have slowly started using the new features and the feedback is very positive.

Lastly, two online resources “Understanding ECMAScript 6” and “Exploring ES6” have been incredibly helpful for our research and analysis. Huge thanks to Nicholas Zakas and Axel Rauschmayer for making it available.

Senthil