Skip to main content

What is automated testing in Azure DevOps and how is it done?

Automation Testing in Azure Devops
Posted On: 6 June 2024

Microsoft Azure DevOps is a practice that helps develop and deploy quality applications using various services and solutions that simplify the process of distributing software applications to end users. In addition, Azure DevOps offers a selection of built-in test automation features that can help accelerate application delivery and reduce the amount of manual testing required. Automation testing in Azure DevOps can increase software delivery speed and enhance quality and reliability in test execution.

What is Automated Testing in Azure DevOps?

Test automation in Azure DevOps refers to executing test scripts automatically as part of the build and release pipeline. These tests can include unit tests, integration tests, UI tests, etc. Automated testing in Azure DevOps helps teams to efficiently manage the testing process across different environments and stages of the development lifecycle.

Azure DevOps uses Azure Pipelines as its CI/CD service, which automates the build, test, and deployment phases. Automated tests are typically run during the build (CI) phase to validate code changes and during the release (CD) phase to ensure the application behaves as expected in different environments.One powerful feature of Azure DevOps is Azure Test Plans, a feature that helps teams plan, execute, and track their testing efforts systematically.

Understanding Azure DevOps Test Plans

Azure Test Plans is part of the broader Azure DevOps services designed to work with different types of testing, including manual and exploratory tests. Its ability to support automated testing by integrating with Continuous Integration/Continuous Deployment (CI/CD) pipelines stands out. This provides crucial insights into testing outcomes.

The first step in using Azure Test Plans for automated testing is the availability of a DevOps organization and project. Once this is in place, it is possible to quickly create test plans and suites that can be organized into logical groups. This Azur DevOps organization further helps execute related tests collectively and track progress effectively.

Integrating Automated Tests with Azure Pipelines

Now, this allows users to run tests automatically whenever a code is committed or according to a scheduled trigger. Let us go through this step by step:

1. Prepare the testing environment

Before integrating tests, ensure your testing framework and test projects are properly set up. Common testing frameworks compatible with Azure Pipelines include NUnit, xUnit, and MSTest for .NET projects, JUnit for Java, and PyTest for Python. The test projects need to be organized and ready to be executed.

2. Create a Build Pipeline

A build pipeline compiles code runs tests and produces artifacts. Here, Here's it can be set using the:


  • Create a YAML File: In the repository, create a YAML file for the pipeline (e.g., azure-pipelines.yml). This file defines the pipeline’s steps.
  • Define Build Steps: Include steps to install any necessary dependencies, build your solution, and run your tests. Here is an example of a .NET Core project that includes test execution:


yamlCopy code 

trigger: - main  

pool: vmImage: 'ubuntu-latest'  

steps: - script: dotnet restore displayName: 'dotnet restore' 

 - script: dotnet build --configuration Release  

displayName: 'dotnet build' 

 - script: dotnet test --logger "trx;LogFileName=test_results.xml"  

displayName: 'dotnet test' 

In this example, dotnet test runs the tests in the project, and the results are output in the trx format. The script commands can be adjusted based on project requirements and the programming language used.

1)Publish Test Results: To make test results visible in Azure DevOps, use the PublishTestResults task:


yamlCopy code 

- task: PublishTestResults@2 

 inputs: testResultsFormat: 'VSTest'  

testResultsFiles: '**/*.xml'  

failTaskOnFailedTests: true  

Automated Testing in Azure DevOps: in a nutshell

  • The first step is to set up test projects that contain the test scripts and are configured to use a testing framework compatible with Azure DevOps (e.g., xUnit, NUnit, MSTest for .NET applications).
  • Use Azure Pipelines to create a build pipeline triggered automatically whenever code is committed to the repository.
  • The pipeline configuration includes tasks to build the application and run the automated tests.
  • It was analyzing Test Results, which are automatically published to Azure DevOps and can be reviewed in the Pipelines section. These will indicate the number of tests passed, failed, and skipped, along with detailed logs and error messages for failed tests.
  • Automated tests can also be integrated into release pipelines to validate applications in different environments (e.g., staging, production).
  • Pre-deployment and post-deployment gates can also be set up by running automated tests to ensure the deployment meets quality standards.
  • Azure Test Plans provides test management, manual testing, and exploratory testing features. It integrates with automated tests in Azure Pipelines, enabling a unified approach to testing.
  • Automated testing in Azure DevOps facilitates continuous feedback on the software’s quality and readiness. Based on test results, teams can quickly address issues and improve the testing process.


Implementing automated testing in Azure DevOps requires planning and ongoing maintenance to adapt to application and testing needs changes. Azure pipelines are explicitly designed to enable CI/CD to increase code coverage, repeatedly run tests, and deploy code automatically.

ACCELQ provides test automation and test coverage plugins to utilize Azure Pipelines' robust capabilities fully. ACCELQ also enables test automation to be used as the foundation for CI/CD across different stages of product lifecycle management. For more information, contact us for a free demo.

Geosley Andrades

Director, Product Evangelist at ACCELQ

Geosley is a Test Automation Evangelist and Community builder at ACCELQ. Being passionate about continuous learning, Geosley helps ACCELQ with innovative solutions to transform test automation to be simpler, more reliable, and sustainable for the real world.

Discover More

Top 10 cloud testing toolsBlogTestingTop 10 Cloud Testing Tools and Services In 2024
24 May 2024

Top 10 Cloud Testing Tools and Services In 2024

Cloud testing tools are popular in the software sector. The blog lists the top 10 tools and services for cloud based software testing.
Dynamics 365 automation testing toolsBlogTestingTop 9 Dynamics 365 Automated Testing Tools In 2024
16 April 2024

Top 9 Dynamics 365 Automated Testing Tools In 2024

Microsoft automation testing tools are used for Dynamics 365 test automation needs. The blog covers 10 Dynamics 365 automated testing tools

Get started on your Codeless Test Automation journey

Talk to ACCELQ Team and see how you can get started.

Close Menu