Released by OpenAI on 30th November 2022, ChatGPT garnered over 1 million users in its first week after launch! Why has it taken the online world by storm? For a start, this model enables interactions in a natural conversational mode. With its dialogue-driven model, ChatGPT can answer queries, answer follow up questions, and even admit its mistakes. Powerful, isn’t it?
But do you think ChatGPT can transform software testing? Let’s find out.
So, without further ado, let's get started.
What Is ChatGPT?
ChatGPT is a conversational, natural language processing chatbot that uses the Generative Pre-Trained Transformer (GPT) model. Based on GPT-3.5, ChatGPT possesses the ability to interact using conversational language. It can accurately provide human-like responses to questions.
What makes ChatGPT so popular among online users? For a start, this tool can efficiently answer complex queries, explain technical concepts, and perform advanced tasks like writing an article. Additionally, ChatGPT can enable software testing by writing various test cases using various languages and frameworks.
Here are some of the inherent capabilities of ChatGPT:
- Understand the natural language and analyze words for context
- Use a dialogue model for a chat-like interaction with any user
- Generate creative responses for any topic, along with details about a specific response
- Remember previous responses to develop a more coherent conversation
- Leverage human feedback to follow directions and generate natural responses understood by humans
To that end, let's understand the relevance of ChatGPT in automated testing and test automation, in particular.
ChatGPT and Automated Testing
Among its capabilities, ChatGPT can automatically generate code in response to a natural language request. The good part is that it can create programming code using multiple languages and frameworks. For instance, ChatGPT can write Selenium code using various languages. In fact, it can be probed further if there's any modification needed in the same.
So, can ChatGPT generate code for use in automated testing? Here are some of the possible use cases of ChatGPT in automated testing:
- Writing an automation test case for any scenario
- Creating a complex test automation pipeline using CI/CD
- Testing any application using multiple microservices
- Providing easy-to-understand code and detailed instructions on how to use the generated code
As a low-code tool, ChatGPT can also generate code using the Cucumber testing framework (based on behavior-driven development or BDD). Using the Cucumber framework, test scenarios can now be written in simple English using terms like "given," "when," and "then."
Here is a Google Search scenario code (using natural language phrases):
- Search for a term on Google
- Given I am on the Google homepage
- When I search for “automation testing using ACCELQ”
- Then the results page must include the search term in the page title
With this test case, Cucumber has defined the scenario to search for a term on Google and then implement the subsequent steps. In essence, using ChatGPT, QA teams can now define both positive and negative test cases for complex scenarios in test automation.
Next, let's discuss how ChatGPT can improve software testing.
How To Improve Software Testing with ChatGPT
Here are some of the common QA activities that ChatGPT can perform:
1. Design Test Cases
High-level test cases may miss crucial information that can mislead any software tester. The AI-driven ChatGPT tool can design test cases and improve its results on the repetition of the same question.
Additionally, ChatGPT can create test cases for numerous requests, such as providing:
- Sample data for an online login form
- Ideas for testing a eCommerce transaction
- Test data for a date picker
2. Automate Test Cases
After creating a test case, the next step is to automate the test case. ChatGPT can automate test cases when provided with the right instruction, such as generating code for automating a test case using Java, Selenium, and JUnit. Moreover, ChatGPT provides the required explanation for the generated automation code.
Here is a sample screenshot of the output provided by ChatGPT for Selenium code:
Here is a sample screenshot of the output provided by ChatGPT for Java code:
3. Report Application Errors
Error reporting is an integral part of any software testing process. Besides writing accurate code, ChatGPT provides a user-friendly way of reporting error-related information using natural language.
Here is an example -
4. Make SQL Requests
As part of testing, ChatGPT can write SQL queries to retrieve relevant information from database tables that are not explicitly mentioned. For instance, ChatGPT can provide sample code to an SQL query like "searching customer tables from
Here is a sample screenshot:
Next, let's discuss if ChatGPT is suitable for use as a low-code testing tool.
Using ChatGPT for Low-Code Testing
With low-code development, users with no (or little) programming expertise can create applications. On a similar note, low-code testing makes it easier for product development companies to write testing code. Effectively, product companies can now write test cases without any technical skills.
Can ChatGPT be used as a low-code testing tool? Yes, here are some reasons why ChatGPT is an effective low-code testing tool for creating use cases:
- Support for natural language, thus making it easier for users to write using natural language.
- Support for multiple programming languages, libraries, and frameworks like Cucumber.
- Using the Page Object model to update code whenever there is any change in the element locator or application structure.
- Separating the code from the test cases, thus simplifying the maintainability of the test script.
As a low-code testing tool, ChatGPT makes it easier to change and fix code. For example, QA professionals can request ChatGPT to modify the code that was previously generated to fix any bug. Using its conversational model, users can leverage ChatGPT to execute the exact change in the code.
Next, let’s look at some of the downsides of ChatGPT for automated testing.
Downsides of ChatGPT for Automated Testing
ChatGPT has shown a lot of promise for use as a low-code tool for automated testing.
Despite its benefits, ChatGPT does have its share of limitations, including the following:
- It does not itself execute the application code being tested, hence the tool cannot determine if its generated code is executable (or not).
- Among the common issues, ChatGPT cannot identify identifiers like Xpaths and IDs, which needs manual updates to locate the correct elements.
- ChatGPT cannot identify deprecated methods – neither does it have an efficient way to correct the problem.
- ChatGPT generally uses common page structures in its generated code. Hence, its users must identify the issue to request for an updated code version.
Despite these drawbacks, ChatGPT has tremendous potential as a low-code solution for automation testing. However, QA testers must use ChatGPT only after completely understanding the used programming language and the application under test.
Can ChatGPT Assist Test Automation?
As a code generation tool, ChatGPT is more powerful than previous tools used for generating natural language. Being new to the market, the ChatGPT algorithm will take time to mature and cannot be trusted fully for performing automation testing. Additionally, some of the written test cases from ChatGPT cannot be deemed as 100% accurate.
To summarize, ChatGPT can assist test automation efforts but is not mature enough to replace AI-based test automation. Hence, we believe that AI technology is the “future of software testing.”
If you are looking for a codeless approach to test automation, ACCELQ is the best choice. With its AI-powered cloud-based codeless test automation platform, ACCELQ improves the speed of testing by 3x and reduces testing efforts by 70%. Here is a comparison sheet of ACCELQ against conventional testing tools available in the market.
By partnering with ACCELQ, you can also improve the efficiency of your test automation. To learn more, contact our QA consultants today.
GEOSLEY ANDRADES
Director, Product Evangelist at ACCELQ.
Being a Test Automation enthusiast, Geosley helps provide ACCELQ with innovative solutions to transform test automation to be simpler, more reliable, and sustainable for the real world. As a passionate Community Builder, Geosley drives Q Community – A Global Community of Test Automation enthusiasts, empowering testers to embrace Test Automation and helping them solve real-life complex testing challenges.
