Migrating Distributed Tests to Azure DevOps

Prerequisite: TestComplete Test Adapter extension for Azure DevOps/Team Foundation Server

How Network Suite elements translate to Azure DevOps

Distributed test in Azure DevOps – Example

Typical tasks

Questions?

From Microsoft Marketplace (for Azure DevOps)

From Microsoft Marketplace (for Azure DevOps)From Microsoft Marketplace (for Azure DevOps)

1. In a web browser, navigate to your team project web portal.

2. Click , then click Browse marketplace.

3. In the marketplace, locate the TestComplete Test Adapter extension.

4. Click Get if free:

   

   Click the image to enlarge it.

5. Select the team project collection for which you want to make the extension available and click Install.

Locally (for Team Foundation Server 2015–2018)

Locally (for Team Foundation Server 2015–2018)Locally (for Team Foundation Server 2015–2018)

1. Copy the extension file (smartbearsoftware.install-testcomplete-adapter-task-for-tfs-15-63.NNN.vsix ) to a location available to your server.

   By default, the extension resides in the TestComplete folder:

   <TestComplete 15>\TFS Integration\smartbearsoftware.install-testcomplete-adapter-task-for-tfs-15-63.NNN.vsix

2. Connect to your team server. Click , and then click Manage extensions.

3. On the Extensions page, click Browse local extensions.

4. Click Manage Extensions. On the resulting Manage Extensions page, click Upload extension:

   

   Click the image to enlarge it.

5. In the resulting dialog, select the TestComplete test adapter extension file and click Upload:

   

6. Click the extension in the list, and then click Get it free:

   

   Click the image to enlarge it.

7. To make the extension available for your team project, select the collection to which your team project belongs, and then click Install:

   

   Click the image to enlarge it.

There are other ways to control TestComplete test runs from Azure DevOps, for example, you can run tests by using the TestComplete command line or the Session Creator utility. We will not examine these alternatives here.

TestComplete: To manage test runs by using the Network Suite functionality, you use a special project (so-called “master”). It stores the list of computers where your tests will run (hosts), and specifies jobs and tasks to be run on hosts. It also defines synchronization points and Network Suite variables. You use this project to start and monitor the test run and to gather results from remote machines.

Azure DevOps: In Azure DevOps (and other CI/CD systems), a special TestComplete project is not required. You configure and manage test runs directly from Azure DevOps pipelines.

If you are new to Azure DevOps and pipelines, we recommend that you read the pipeline documentation on the Microsoft website:

docs.microsoft.com/en-us/azure/devops/pipelines

TestComplete: In Network Suite, the remote computers, on which your tests are running, are called hosts.

Azure DevOps: Azure DevOps (and other CI/CD systems) support managing a pool of computers on which various tasks are run. Such computers are called agents. In Azure DevOps, agents can be Microsoft-hosted and self-hosted.

For TestComplete tests to be able to simulate user actions over the tested application user interface, the agent must be configured to run in the interactive mode.

For instructions on how to configure agents, see below.

TestComplete: In Network Suite, a job is a collection of tasks that run in parallel on specified hosts. Network Suite can have one or several jobs. Jobs are run in a series.

Azure DevOps: Azure DevOps pipelines also have jobs. A single job runs on a single agent and executes one or several steps. The job always executes the steps in a series, one after another. Jobs can be grouped into stages.

To run tasks simultaneously, you can configure parallel jobs by using various parallel strategies. The pipeline will create several instances of a job and run them concurrently.

To organize parallel job runs, your Azure DevOps organization must have parallel jobs purchased. To learn more, see Configure and pay for parallel jobs in the Azure DevOps documentation.

If you are not interested in parallel runs, you can run your jobs and tasks in a series.

TestComplete: a Network Suite task is a test to be run on a host as part of a job. Task properties specify the TestComplete project that contains the test you want to run, the test to run, the computer on which you want to run this test, and the test runner tool (TestComplete or TestExecute), and the way you would like to handle test results.

Tasks that belong to the same job run in parallel.

Azure DevOps: In Azure DevOps pipelines, a job consists of steps: tasks or scripts that run one after another. To run TestComplete tests as part of a pipeline, use the Visual Studio Test task.

Crucial differences from the Network Suite tasks:

• You cannot assign the Visual Studio Test task to an agent directly. To do this, you use capabilities and demands. See below.

• Unlike the Network Suite tasks that run in parallel, the Azure pipeline runs the tasks one after another in the order the pipeline specifies them. To have several tasks run in parallel, you can use parallel jobs. You can also configure the pipeline to distribute tests among several agents.

1.1. Install the TestComplete Test Adapter extension

1.2. Create an Azure DevOps project

1.3. Configure pools and agents

For Azure DevOps agents to be able to access your TestComplete projects to run them, add the project files to your Azure project’s repository. How you can do it, depends on the repository you use. For detailed instructions on adding files, please see Supported source repositories in the Azure DevOps documentation.

Note: In Azure DevOps, you can create and configure the pipeline either by using the visual editor or as a YAML file.

1. In your Azure DevOps project, go to Pipelines and then click New pipeline.

2. Specify the repository where your TestComplete project files reside.

3. Specify jobs that will run your TestComplete tests. In our example, we will have Job1:

   YAML

   Visual pipeline editor

   Copy Code

   YAML

   name: Build_$(Date:yyyy_MM_dd)
   jobs:
     - job: Job1
       displayName: Job1
       continueOnError: true
   …

   Visual pipeline editor

   

   Click the image to enlarge it.

4. Specify the agent pool and agents on which you want to run tests. In our example, we will run the job on agents that have the TestRunner capability set to TestExecute. It is the capability we have assigned to agents earlier:

   YAML

   Visual pipeline editor

   Copy Code

   YAML

   …
   jobs:
   - job: Job1
     displayName: Job1
     pool:
       name: Test environments
       demands: TestRunner -equals TestExecute

   
   …

   Visual pipeline editor

   

   Click the image to enlarge it.

5. If you do not have Visual Studio installed on your target agents, you can add the Install Visual Studio Test Platform task to the pipeline. The task will install the test platform that allows running tests on computers where Visual Studio is not installed:

   YAML

   Visual pipeline editor

   Copy Code

   YAML

   …
   jobs:
     - job: Job1
       displayName: Job1
       steps:
       - task: VisualStudioTestPlatformInstaller@1
       inputs:
         packageFeedSelector: 'nugetOrg'
         versionSelector: 'latestStable'
   …

   Visual pipeline editor

   

   Click the image to enlarge it.

6. Add the Enable support for TestComplete tests task to the job. This task installs the TestComplete test adapter to your agent and enables the TestComplete test support.

   If needed, set the preferred test runner to use to run tests.

   Specify how you want to handle test logs: upload them only when the tests fails, completed with warnings, or always.

   YAML

   Visual pipeline editor

   Copy Code

   YAML

   …
   jobs:
     - job: Job1
     displayName: Job1
     steps:
       - task: VisualStudioTestPlatformInstaller@1
       …
       - task: InstallTestCompleteAdapter@1
         inputs:
           logsLevel: '0'
   …

   Visual pipeline editor

   

   Click the image to enlarge it.

7. Add the Visual Studio Test task to the pipeline and configure it to run TestComplete tests:

   • Configure the task to select tests by using assemblies.

   • Specify the project suite file (.pjs) that contains your tests and that the task will run.

   • Specify the folder on your agent where the task will search for TestComplete project files. By default, it will search the files in the agent’s working folder.

   • If needed, you can use a filter expression to specify test items of your TestComplete project that you want to run.

     In our example, we will use the expression to run Test1 test item of the TestComplete project.

   • You may also specify the test run name to which test results will be assigned.

   YAML

   Visual pipeline editor

   Copy Code

   YAML

   jobs:
     - job: Job1
       displayName: Job1
       steps:
       …
       - task: VSTest@2
         inputs:
           testSelector : 'testAssemblies'
           testAssemblyVer2: '**\*.pjs'
           searchFolder: '$(System.DefaultWorkingDirectory)'
           testFiltercriteria: 'Name=Test1'
           vsTestVersion: 'toolsInstaller'
           testRunTitle: 'Automated tests'

   Visual pipeline editor

   

   Click the image to enlarge it.

8. To run other test projects, you can add more Visual Studio Test task to the job. To run tests in parallel, you can enable the Parallelism for the created Job1.

   Your organization must have parallel jobs. To learn more about purchasing parallel jobs, see Configure and pay for parallel jobs in the Azure DevOps documentation.

   After enabling parallelism for Job1, configure each instance of Job1 to run a separate TestComplete test:

   1. Create a variable (or variables) that will store values for each Job1 instances:

      YAML

      Visual pipeline editor

      Copy Code

      YAML

      jobs:
      - job: Job1
        displayName: Job1
        strategy:
          matrix:
            Test1:
              test_name: 'Test1'
            Test2:
              test_name: 'Test2'
          maxParallel: 2

      
      …

      Visual pipeline editor

      

      Click the image to enlarge it.

   2. Use the created variable to specify the TestComplete test that the job should run:.

      YAML

      Visual pipeline editor

      Copy Code

      YAML

      jobs:
      - job: Job1
        displayName: Job1
        strategy:
          matrix:
            Test1:
              test_name: 'Test1'
            Test2:
              test_name: 'Test2'
          maxParallel: 2
        steps:
        …
          - task: VSTest@2
            inputs:
              testSelector: 'testAssemblies'
              testAssemblyVer2: '**\*.pjs'
              searchFolder: '$(System.DefaultWorkingDirectory)'
              testFiltercriteria : 'Name=$(test_name)'
              vsTestVersion: 'toolsInstaller'
              testRunTitle: 'Automated tests'
      …

      Visual pipeline editor

      

      Click the image to enlarge it.

Complete pipeline example (YAML)

Typically, you configure the pipeline to run on a specific event. For example, when new file versions are uploaded to the project repository:

YAML

Visual pipeline editor

Copy Code

YAML

…
trigger:
  - main
…

Visual pipeline editor

Click the image to enlarge it.

You can also run the pipeline manually.

After the pipeline run is over, you can view the test results on the Tests page:

Click the image to enlarge it.

Before the testing starts, your test projects must be placed in a location where the test runner (TestComplete or TestExecute) can access them from the remote computers.

TestComplete: this task can be done automatically — you can configure your tasks to copy projects to hosts.

Azure DevOps: Typically, you place your TestComplete projects to your Azure project’s repository and then configure your pipeline to take the project files from that repository and place them to a target location on agents.

For your tests to be able to simulate user actions over the tested application GUI, agents, where the tests will run, must have an interactive user session opened.

TestComplete: you can store user credentials that will be used to open user sessions in Network Suite. For each host you add to your Network Suite, you can specify how user sessions should be opened on them.

Azure DevOps: configure test agents to run as an interactive process, and enable the auto-logon. When the agent starts, it will open the session automatically.

To learn more about configuring agents with the auto-logon enabled, see Self-hosted Windows agents in the Azure DevOps documentation.

In distributed testing, it may be crucial that tests that run on several remote computers interact with each other.

TestComplete: the Network Suite functionality offers the following means of interaction between projects:

• Synchpoints - Can be used to pause the test execution on remote computers until a specific point is reached and then resume the test execution on all remote computers simultaneously.

• Network Suite variables - Can be used to exchange data between all tests that participate in testing. To track the variables state, the WaitForNetVarChange method and the OnNetVarChange event can be used.

Azure DevOps: to implement interactions between TestComplete tests, you can use the following:

• Use various parallel strategies to organize the pipeline run. You can command the pipeline to run several instances of a job simultaneously or you can run several instances of a job with various inputs.

• Use secure files to store resources that the pipeline can access during the build run. To learn more about secure files, please see About pipeline resources in the Azure DevOps docs.

• Use dependencies and conditions to organize the stage run order in your pipeline.

• Use pipeline variables to store, send, and get data from TestComplete tests during the pipeline run.

To run TestComplete tests, you run your pipeline. You can initiate the pipeline run manually, or you can configure your pipeline to start automatically on a specific event, or you can schedule the pipeline run.

TestComplete: you can track the current state of running tests in the Network Suite editor, or you can do this in your tests by checking the State property of appropriate scripting objects.

Azure DevOps: you can track the test progress on the Build page.

TestComplete: Network Suite can retrieve test results from remote computers automatically, or it can store them on remote machines (this was helpful if the test log was too large).

Azure DevOps: in general, when you run TestComplete tests on remote computers from a CI/CD system, the test results will be available only on those remote computers where they were run. In Azure DevOps, the TestComplete Test Adapter extension will accumulate your test results and upload them to the Azure server. You can view them on the Tests page of your pipeline results.

You can configure the test adapter to copy the following results:

• All test results
• Results of failed tests only
• Results of failed tests and tests completed with warnings