Testing is an information-gathering process through exploration, experimentation, and studying the product under different constraints and variables.
We, as Testers, gather this information for our clients, customers, stakeholders, and someone to whom this information matters and share it with them in the form of Bugs, Test reports, Test results, Experience reports, etc.
Often, Test reports are presented as pass-fail percentages, bug counts, or some colorful graphs and charts illustrating test-case execution summary.
A few years ago, driven by process, I used to provide such reports to my clients. Later, when a bug occurred in production, they used to ask me whether that particular case was covered in Testing. In addition, many follow-up questions about the product and its testing were redirected to me even after the test report was shared. This process continued until one bright day when I realized, that these reports don’t provide the information that is required by the stakeholders. There is a gap between the information provided to them and the information that they need. This is due to the process-driven unclear reporting that only talks in numbers and percentages.
Have you ever wondered how this information would bring any relevance to the stakeholders to whom the information about the product matters? How would the number of test cases that passed or failed help them understand the product and make important decisions? One might be very good at Testing and finding information, but if it is not articulated and presented well, it wouldn’t help the clients and stakeholders. So how do we do that? How do we present the information that we have discovered in a compelling manner? How do we write clearer test reports that cater to the information needed?
For starters, let's shift the focus from process-driven test reports to purpose-driven test reports.
To build any compelling report, we need to understand the important aspects of a report. They are the report’s
1. Purpose
The first step is to understand the purpose or objective of the report that we are building. If the objective is clear, it helps us to look for the information needed in the right direction and also guides our reporting style.
Tips:
Answers to the following questions would reveal the purpose of our report
- Why are we creating this report?
- For whom are we creating this report?
- What information are they looking for through this report?
- Do I have enough information to furnish through this report?
- How will this information be useful to them?
Ready to Get Started?
Let our team experts walk you through how ACCELQ can assist you in achieving a true continuous testing automation
2. Content
Once we are clear with the purpose of our test report, we will know what information needs to be provided through this report. This constitutes the content of the report. Content is the King of our report, and if our content can answer the questions that our clients have in mind, it establishes our credibility and visibility as a Tester.
Tips:
Following are the points that I have realized were missing from my Test reports and now have inculcated in my Test reports. These points give clarity to the reader (stakeholder/client/team members/managers) on our testing and allow them to form an opinion or make necessary decisions about the product.
Elements to include in the content of your test report if not doing it already:
Product/Feature:
Including a short description of the product/feature helps the reader of the report understand what product/feature we are referring to in this report and would help them relate to it instantly, even if the report is read at a later point in time.
Product Artifacts:
Artifacts include the version/instance of the product that is tested, any developer notes that we got when testing it, Product notes and designs from the Product team, Customer feedback/reviews or surveys from the Sales team, etc. This information helps the reader understand what level and depth of information were available to us (testers) based on which we have carried out our testing.
Testers involved:
Details of the Testers who have tested the product and prepared the report with their conclusions are important. This helps the reader identify the tester with their expertise and also helps the testers to build credibility for their work.
Time spent:
When testing, every minute is important as we keep understanding and uncovering information about the product. Documenting the time invested in testing helps the reader understand how much time was spent gathering all this information.
Be part of the lively group discussions covering various topics around Web, API, Mobile Test automation, and more...
Test Environment Details:
Environment details include the browser details, device details, platforms, and instances that the product/feature has been tested on. This helps the readers understand the platform and device coverage that has been covered through our Testing.
Tools used:
Any tool that has helped us to gather information on this product.
Test Coverage:
The areas/modules of the product that have been covered while testing this product.
Test Approach, Techniques:
The Test approach that we took while testing this product and the test techniques involved help the reader understand the depth of testing performed on this product.
Not Tested areas:
It is essential to include/mention the areas that have not been tested for the product. For example, the product's interaction with other products, a feature still being developed, etc. This helps the reader to have an understanding of which areas of the product have not been covered as part of our test results.
Critical/Showstopper/Blocker bugs:
Critical/Showstopper issues that you want the stakeholders to know, along with their status and the root cause. Also, identify the risk and business impact of such issues and advocate for them.
Enhancements/Suggestions:
You might have felt enhancements, recommendations, and suggestions while testing the product that could be added to increase its value to its users.
Heuristics/references/oracles:
Oracles and Heuristics that you have referred to come to conclusions while testing this product/feature.
“Discovering the unexpected is more important than confirming the known.“ – George E. P. Box.
3. Articulation
Now, we understand the purpose, and we have our content ready. The next step is to articulate this content in a compelling manner. We need to assess our audience and furnish them with the information in a consumable manner. Many a time, people either over-report or under-report information. To strike the right balance, we need to learn what level of product understanding the audience has and what they are looking for.
Tips:
- Be crisp and clear with the information
- Provide supporting information or proofs in the form of links.
- Structure data in a relatable manner
- Emphasize Risks and Business Impact
Format/Template:
There is no standard format or template that one should use for writing test reports. As per the context of your project, the audience, and the time you have at hand for building a report, you can always experiment and find what fits you the best. The format should be easy to understand and accessible to anyone reading it. You can explore mindmaps, documents, sheets, sketch-notes, etc., to present the information.
Once you decide on a format for yourself, you can create a template and reuse it in your future reporting and save a lot of time.
Tips:
Build reports in a format that is
- Shareable
- Readable
- Less complex, easy to understand
- Time-efficient
I have customized my Test report template, and I reuse it whenever required. It saves me a lot of time that I can re-invest in my testing and provide insightful information to my stakeholders.
Hope these tips help you add value to yourself and your stakeholders.
Thanks for reading! Happy Testing!
Apoorva Ram | Senior Software Test Engineer at Qapitol QA
Apoorva is a Test Enthusiast who believes that Software Testing is a Craft. She is an avid reader who is passionate and curious about anything and everything in life. When not Testing, She loves to travel, explore cultures and cuisines, and indulge in deep, meaningful conversations.
