Impact Based Testing

TABLE OF CONTENTS

1. Overview

2. Supported Technologies and Application Types

3. Use Case 

4. Approaches in Impact Based Testing

5. Integration of Command Line Interface ( CLI )

1. Overview

During the execution of your test automation scripts, the algoQA solution leverages code coverage tool to evaluate the code coverage for applications. This process generates coverage reports and maps your test scenarios to the corresponding source code files, providing detailed insights into which parts of the code are tested. You can use the algoQA impact-based testing solution to analyze code changes by comparing selected git commits. This helps in identifying which files have been affected by recent changes. Based on the affected files and the collected coverage information, algoQA recommends the most relevant test cases to run and execute, ensuring that your tests are focused on areas impacted by code modifications. If required, the solution can be deployed using Jenkins and the recommended test cases can be executed automatically as part of CI/CD workflow. Note that test scripts must be based on selenium. Additionally, Cypress and Squish has its code coverage tools that can be used for tracking test coverage.

2. Supported Technologies and Application Types

Note: Install the nyc library globally to enable coverage reporting for Node.js applications across your system. This ensures the 'nyc' command is available from any project directory without local installation.

An Istanbul-instrumented application includes code modified by the Istanbul tool to track test coverage. This allows nyc to generate accurate reports on which lines, branches, and functions are exercised during tests.

Add below to package.json file source code

a. In Scripts json

"start:coverage": "nyc webpack serve --mode development"

b. In dependencies json

"nyc": {

"extends": "@istanbuljs/nyc-config-typescript",

"all": true

}

c. In devDependencies

"babel-plugin-istanbul": "^7.0.0",

"@istanbuljs/nyc-config-typescript": "^1.0.2",

"nyc": "^17.1.0"

Add below codes to webpack.config.js below loader: "babel-loader", this line

options: {

presets: [

["@babel/preset-env"],

["@babel/preset-react"],

["@babel/preset-typescript"],

],

plugins: ["istanbul"], // Enables instrumentation for coverage

},

How to run the solution

1. Run the application as localhost

a. Install libraries using npm install

b. Start application using npm run start

2. Run streamlit application

a. Open CMD

b. Run below commands for the first time inside python scripts directory

i. conda create –n impact

ii. conda activate impact

iii. pip install –r requirements.txt

c. If step b successfully completed next time onwards

i. streamlit run app.py

3. Click generate_coverage_report button and run button to get coverage reports

3. Use Case 

Web Application

Step1: Get code coverage information when test automation scripts are executed  The algoQA impact-based testing solution uses Istanbul to check the code coverage of ReactJS-based applications.  Upon execution of the script, coverage reports are obtained, along with the mapping of BDD scenarios to source code files. Step 2: Find the impacted test cases due to changes in code base  The algoQA impact-based testing solution is used to collect affected files between selected git commits.    The affected files and the code coverage information collected is used to recommend and execute impacted test cases using the algoQA solution.

Desktop Application

Step 1: Get code coverage report for desktop applications  The algoQA impact-based testing solution uses JetBrains dotCover command line tool to get source code coverage information for .NET-based desktop applications. The coverage reports are parsed to create a spreadsheet with test cases and source code methods accessed by the test case during its execution. Step 2: Suggest the impacted test cases  The algoQA impact-based testing solution captures changes in the source code by comparing git commit information. The obtained information is parsed to extract the methods impacted between commits, which are then stored as a spreadsheet. Using the above information, coupled with test case-to- method mapping,  the impacted test cases are suggested and executed using the algoQA solution.

4. Approaches in Impact Based Testing

Impact-based testing is a strategic approach to test case selection that ensures efficient software testing by focusing on the affected test cases when changes occur in the application source code.

This approach consists of two key steps:

1. Capturing Code Coverage Information: During test execution, the solution collects source code coverage details and establishes a link between the executed code and corresponding test cases.

2. Tracking Code Changes & Suggesting Tests: When changes are made to the codebase, the system uses version control history to identify affected areas and suggests optimal test cases for execution, ensuring efficient and targeted testing.

5. Integration of Command Line Interface ( CLI )

Note: There is an option to eliminate front end part of Impact Based testing and this can be integrated based on your requirement.

To remove the front-end part of Impact-Based Testing that is, the UI interface for generating the coverage report and suggestions a Command-Line Interface (CLI) tool has to be created. There is a web interface. When you click on “Generate Coverage Report,” the report will be generated. Similarly, clicking on “Generate Suggestions” would execute that functionality and produce the output.

This UI-based workflow can be removed from the folder structure. It can be runned through the CLI tool.

there is a command for this and this will be unique for each client.

For example, consider a case where there are two repositories. The Suggestions module will scan the files within these two repositories only. The repository paths should be provided in the command using a semicolon (;) separator.

You can pass as many repositories as needed using this semicolon-separated format in the CLI command.

Now, regarding commit IDs specifically, commit_one, this represents a GitLab commit ID. Whenever a developer creates a merge request or pushes code, a commit ID is generated. This commit_one is a mandatory field in the command, commit 2 is optional, if you pass both commit ids the comparison wil be between these 2 changes otherwise if only commit 1 is passed. The backend will automatically detect the latest commit relative to commit_one.

Consider it as a tree:

• commit_one is the starting node
• The system will automatically identify the latest commit (leaf node) following from this node

This latest commit will be identified internally, and a comparison will be made between commit_one and this will detect commit to generate the suggestions or impacted test cases.

The system compares the differences and identifies the impacted test cases based on the changes between these commits.

Since the project follows a folder-based structure and all the code is written in Python, a separate executable file must be created in a different folder.

currently, there is a command available in the code to run the functionality and when you run the executable file, it resides in a different folder than the test_case_covered_lines.json file, which is the part of Generate Coverage Report module. To enable access between them, A path will provided explicitly via the command line.

Key Implementation Points:

• Repository Paths:
  Multiple repository paths can be passed through the command line using a semicolon (;) separator.
• Commit IDs:
  At least commit_one (the starting commit ID) is mandatory.
  Optionally, you may also provide commit_two.
  • If both commit_one and commit_two are provided, the system will perform the comparison using these two.
  • If only commit_one is provided, the system will automatically detect the latest commit (the most recent change) and perform the comparison against that.
    
    

Licensing & Security

Since the tool will be CLI-based (and no longer runs in a secure web or desktop environment with built-in authentication), user verification will be added as part of the command.

Every user has a unique user ID (e.g., 56781234). This must be passed as part of the command using the --user flag.

Example:

dist\suggest_test.exe --fe-repo "your/repo/path" --commit1 abc123  --summary-file "your_local_path_to_this file\solution\json_coverage_summaries\testcasewise_covered_lines.json" --user 1234

Behavior:

• The system will first verify the user ID to ensure it’s an active, licensed user.
• Once verified, the tool proceeds with execution.
• This check forms the licensing mechanism for secure access via CLI.