Monthly Archives: January 2013

Becoming a World-Class Tester

 A tester is somebody who knows that things can be different.

That is Jerry Weinberg's description of a world-class tester, which reminds us that testing is about critical thinking and imagination. It is about not believing anything at face value, but maintaining an investigative mindset.

Testing is questioning a product in order to evaluate it.

That definition of the activity of software testing, from James Bach, includes one of the most important aspects of testing: information discovery and delivery. Software testing is about revealing the unknown.

If quality means value to some person who matters (as Weinberg has defined it), then testing is in the service of discovering anything that threatens this value. To provide this service, a tester needs to be able to do a variety of things, such as asking good questions, commenting on requirements, and interacting with a product in such a way that it yields information that’s useful.

My definition of a world-class tester is a person who is able to rapidly discover highly relevant information about the product, who makes the most use of any resource that is available to him/her, and who has the respect of people involved in a project. It is a person who can be trusted.

So, given these definitions, what skill and mindset dimensions should you invest in to become a world-class tester? What areas do you need to be good at in order to avoid being discredited as a "mouse-click-monkey"?

Technical proficiency

At the higher levels of management, software testing is often seen as only a technical discipline. They think your effectiveness as a tester depends on your ability to write code that does the testing for you. But there are many aspects of software testing that are simply not automatable. Humans are incredibly skilled at thinking and adapting to new information that arises based on context, and since software is a system of ever-changing functions and states, sometimes it is more efficient to do exploration. Automation is the opposite of that, since it involves scripted procedures.

However, automation can free the time of testers to do more exploration, which allows discovery. Anything that is either highly repetitive/boring or that needs to be run over and over again, is a good candidate for automation. Automation is very valuable in these areas.

Developing powerful and maintainable automation is difficult. I am convinced that a mediocre software developer is not able to live up to the requirements for such a job. If you want to do useful test automation, you'd better be an excellent developer in addition to having other testing skills.

Here are some more skills and mindsets that I believe will help you:

Investigative curiosity

If you think that testing is a necessary evil – something to get out of the way until you can do something really cool – you are in danger of missing important problems. World-class testers are constantly exploring the capabilities and limitations of their products while maintaining a healthy skepticism about those products' capabilities.

If you want to give yourself a fancy title, think of yourself as a "compulsive empiricist," guided by irresistible impulses to find the truth about what's happening. World-class testing means following the urge to get to the bottom of things, not giving up until you have experienced enough value for yourself. When you see a tester shying away from unverified assumptions, that’s a good indication they take their job seriously.

Observational skills

If you want to observe something, you need to notice it first. Observation is the ability to notice important or significant details. While testing the product you need to be able to observe situations that are unusual.

Who would be a good teacher in the domain of observation? Go for a walk with a five-year-old and see what grabs his/her attention: a little crack in the pavement, a beautiful flower in a garden, a snail slowly making its way on a wall. Children are natural observers; they are interested in everything that happens around them. Their channels are wide open and they observe with joy.

Condition yourself to keep your channels open. Emulate child-like behavior. Train yourself in general awareness. Look closer. Slow down. Observe.

Communication skills

You will encounter circumstances where you have to defend your findings. You will have to convince people to take action, so you'd better know how to explain why your findings are relevant.

World-class testers actually practice describing their testing. They know their explanations need to go way beyond "Yeah, I tested it, trust me." If you can describe your testing, your credibility as someone who knows what he/she is doing will grow. One way to get better is to use the Heuristic Test Strategy Model when you explain what you are doing. Or, even better, you will come up with your own models.

As you tell your story, remember that communication is a two-way street. World-class testers follow up on answers they get from their questions and are alert to new information that comes up in conversation.

Writing skills and bug advocacy

The richest communication happens when you are able to talk with somebody face to face. You observe facial expressions, you can react to indicators of confusion, and misunderstandings can be clarified right away. Not so with writing. The channel is narrower and more prone to misunderstandings. Clarifications need more time and effort. 

Bug advocacy often goes along with writing skills. Your ability to describe the problem accurately and understandably influences the probability of the right bugs getting fixed. Not every reported bug should be fixed, but the ones that should had better not escape because of poor wording.

A bug report is a persuasive document that needs to influence the right people to act. Free your bug reports from irrelevant information and don’t make them come across like a tabloid headline screaming at the reader.

You might want to keep a record of your past bug reports. Which ones produced confusion? Could you have written them differently? Would a change of wording have resulted in better understanding?

Domain knowledge

As a tester you are not just comparing specifications with the implementation. There are many other references, such as user expectation, consistency with comparable products, or the history of your product, that you can use to identify problems. Domain knowledge is also one of the references you can develop. Talk to your customers. Have a chat with the marketing guys. Spend a day with Customer Support. The better you know what is important for your business, the better your judgment will be.

Willingness to learn

You are a knowledge worker. Knowledge is not static, especially not in the technical domain. Constant learning is essential in order to become better at what you do. A situation where humans are interacting with software is a situation where software is interacting with humans. It is helpful to try to understand how both work. I recommend exploring the social sciences (cognitive psychology, social interaction theories, etc.) as well as increasing your technical education.

In my opinion, every tester who wants to become really good will profit from participating in the excellent BBST (Black Box Software Testing) courses. Each course takes four weeks, and you will spend 15-25 hours each week on intensive hands-on exercises. You can find more information about these courses on the website of the Association for Software Testing.

Another course I highly recommend is RST (Rapid Software Testing), which focuses on context, exploration, and thinking to find better bugs. It is currently taught by James Bach, Michael Bolton, and Paul Holland. A skillful search on the engine of your choice will lead you to the right place.

Social skills

Testers who constantly whine about their job and how horrible everybody else is get my blood boiling. Self-victimization is poisonous. Sometimes you just have to let go. If you cannot do that, you will be perceived as a sourpuss. If the crowd goes for a Friday evening beer, you won’t be invited. Imagine what influence that has on people’s willingness to collaborate with you.


Humor helps. Among many other things, it helps to sustain your sanity. As a tester, you often live in a stressful environment. Smiling and laughter reduce the damaging effects of stress hormones. It helps you to stay focused on what you want to be doing: testing.


And above all, you need to practice your skills. If you don’t practice, you can’t reach the higher levels of proficiency. While you practice, observe yourself, find areas for improvement, and make all of your actions deliberate and self-reflective.


Aiming to become a world-class tester does not mean reaching a destination, but rather being on a constant journey into new territories. The exploration never ends--which is exhilarating, and makes software testing so rewarding.

Happy testing!

Early Detection of Frontend Single Points of Failure

Until fairly recently, a single point of failure (SPOF)—a condition that can bring down an entire system—has been predominantly associated with the backend world of server clusters, networking, and operations. However, with the increase in third-party widgets and the complexity of modern web applications, the web community is increasingly aware that a frontend SPOF can also bring down an entire site. As a result of a frontend SPOF, the user sees an entirely or partially blank page, or experiences delayed rendering of the page.

This post describes a tool developed at eBay, called SPOFCheck, which has been open-sourced and is being integrated into engineering life cycles not only at eBay but globally.


The awareness of frontend SPOF has increased tremendously among engineers, thanks to Steve Souders' initial research on this topic, as well as recent articles and blogs emphasizing its importance; see, for example, Bjorn Kaiser's post about third-party content causing SPOFs. Numerous utilities and plugins exist that can detect possible SPOF vulnerabilities in a web application, most notably, the Chrome plugin SPOF-O-Matic, and the YSlow 3PO extension.

At eBay, we wanted to detect SPOF at a very early stage, during the development lifecycle itself. Such early detection would require an additional hook in our automated testing pipeline. The solution we developed and open-sourced is SPOFCheck, a simple tool that works on our test URLs and produces SPOF alerts. The tool is integrated with our secondary jobs, which run daily automations on a testing server where a development branch is deployed. When SPOFCheck detects possible vulnerabilities, engineers receive SPOF alerts and act accordingly. As a result, SPOFs can be contained within the development cycle, rather than sneaking into staging or production.

Command-line interface

SPOFCheck is a command-line interface (CLI) that runs on Node.js. Source code, syntax, and usage documentation are available in GitHub. Running SPOFCheck involves simply specifying the output directory. Command-line options include output format (XML that most CI servers can parse, XML to be consumed by other utilities, or text), console-only output, and the rules (checks) to be run. By default, SPOFCheck runs the following rules:

  1. Always load third-party external scripts asynchronously in a non-blocking pattern.
  2. Load application JS in a non-blocking pattern or towards the end of the page.
  3. Attempt to inline the @font-face style; ensure that the font files are compressed and cacheable.
  4. Ensure that the font files are compressed, cached, and small in size.
  5. Ensure that inlined @font-face styles are not preceded by a SCRIPT tag (which causes SPOFs in Internet Explorer).


New rules can be easily added, either by pushing them to the rules array or by calling SPOFCheck's registerRules function.

At eBay, we've built a thin web client to provide SPOFCheck as a service. eBay engineers can create multiple SPOF projects, which can be triggered on demand or configured to run on a recurring schedule. Project owners designate the teams or individuals who receive SPOFCheck results.


Thanks to Steve Souders, whose work (cited above) is the source of SPOFCheck's default rules.

SPOFCheck reuses code logic from the GitHub projects spof-o-matic and 3po. The design and packaging of the tool is based on csslint, thanks to Nicholas Zakas and Nicole Sullivan.

Thanks also to perfplanet, who featured SPOFCheck in their 2012 performance calendar.

Tomcat/Jasper Performance Tuning

JavaServerPages (JSP) is productivity-enhancing technology for building dynamic web pages. With more than 10 years of history, it is popular and mature. JSP 2.0 includes new features that make the code cleaner, such as the Expression Language (EL), the tag file, and simple tags. However, these new features can cause some performance issues.

eBay uses JSP technology to build web pages. More specifically, we use Geronimo 3 as the web container and Geronimo's JSP engine, Tomcat/Jasper. To handle eBay's huge volume of requests, the JSP engine must be powerful and must provide exceptionally high performance. Because Tomcat/Jasper shows some weakness in performance when using hundreds of EL expressions and tags, we have focused much effort on profiling and optimizing Jasper to make JSP more efficient. Most of the performance solutions and patches that we have submitted to the Tomcat community have been accepted. As a result, Tomcat users will soon be able to benefit from our work.

This blog post discusses the main issues we have identified and addressed.

EL resolving

When a JSP page includes a large number of EL expressions, EL resolving becomes a bottleneck for page rendering. The following JProfiler screen shot shows a case where 40% of the Java processing time is for EL resolving; the whole JSP page costs 1,865 ms, while the EL resolving costs 722 ms.

This profiling enables us to identify the heavy methods from Jasper:


Looking into the code, we found that every expression node needs to call CompositeELResolver.getValue when trying to resolve the value. The bottleneck is that CompositeELResolver contains at least seven EL resolvers:  ImplicitObjectELResolver, MapELResolver, ResourceBundleELResolver, ListELResolver, ArrayELResolver, BeanELResolver, and ScopedAttributeELResolver. The method getValue tries to return the value from each of these resolvers one by one, until one of them returns a not-null value. This logic results in many unnecessary resolver calls.

We proposed an enhancement:  a customized CompositedELResolver in Jasper that avoids many unused resolver calls. This change, accepted in Tomcat 7.0.33, saves 10% of the EL resolving time. The Bugzilla URL for this issue is (The issue was filed by the Geronimo team, who tried to resolve the issue for us).

We've filed some related Tomcat bugs as well, such as Bug 53867 (also accepted in Tomcat 7.0.33) and Bug 53869.

JSP compilation

JSP compilation is a very heavy process that includes JSP reading, JSP parsing, Java generation and compilation, and class loading. We have achieved many optimizations in this logic as well.

JspReader was the focus of our initial performance-tuning work related to compilation. JspReader.mark is called too many times during JSP and tag files compilation. The following screen shot illustrates the excessive number of JspReader.mark invocations and the time they consume.


By using the JspReader buffer to reduce the number of invocations, JSP reading performance can improve fourfold. Tomcat 7.0.31 includes this enhancement; for more details, see

EL code generation

Even with multiple optimizations, EL evaluation is still a big hotspot. It is not only time-intensive but also CPU-expensive. EL resolving is the root cause. Therefore, if we can bypass EL resolving in some cases, we can improve performance.

The idea is to do code generation for simple EL expressions, such as ${model}, ${model.test}, ${empty model}, and ${model.value++}. Such expressions can be interpreted as simple Java code. We have identified three kinds of EL expressions that can use code generation:

  • Simple EL expression containing only one part of the expression -- for example, ${model}. Here is an example of the generated code for this kind of expression :
  • EL expression with two nodes where the type of the first node is specified by a tag file attribute—for example, ${model.location}, where model is specified by the following directive in the tag file:
<%@ attribute name="model" required="true" type="" %>

   Here is the generated code for this example:

(getModel() != null ? getModel().getLocation() : null)
  • EL expression with logic or arithmetic where the value part can be generated. For example:
${(intlExpansion eq 'true' && not empty model.location) || sortType==7}

   The generated code for this example looks like this:

(org.apache.jasper.runtime.ELRuntimeUtil.equals(getIntlExpansion(), "true")&&(! org.apache.jasper.runtime.ELRuntimeUtil.isEmpty((getModel() != null ? getModel().getLocation() : null))))

We have proposed an extensible ELInterpreter solution to Tomcat and implemented an interpreter for these three kinds of EL expressions. After the extensible interpreter has been accepted, we will contribute the code to Tomcat as well. This customized interpreter improves total page performance by 10%. Application developers will be able to inject their own interpreter to replace the default JSPUtil.interpreterCall.

For more about this enhancement, see

JSTL code generation

Tag library performance is another hotspot for page rendering. Among the problems are the numerous method calls in the standard generated code and the high number of context switches. Every tag needs to call doStartTag and doEndTag.

Jasper has a code-generation solution for JSTL, which is based on the TagPlugin interface. Although eBay leverages the implementations in Jasper, there are many bugs. Here are some of the fixes we've contributed:




Fixed version

Bug in JSTL tagPlugin "Out"

Bug 54011



Bug in JSTL tagPlugin "Set"

Bug 54012



Configurable tagPlugins.xml

Bug 54240



NPE in BodyContentImpl

Bug 54241



Bug in ForEach

Bug 54242



Bug in When and Choose

Bug 54314




Tomcat/Jasper is eBay's chosen JSP solution. However, few solutions can fully satisfy eBay's daunting performance requirements, and Tomcat/Jasper is no exception. Our efforts have identified four major performance issues:  EL resolution, JSP compilation, EL code generation, and JSTL code generation. Working with the Tomcat community, we've made substantial progress on some of these issues, and more enhancements and bug fixes are underway.