Categories
Automated testing Pytest

Run pytest tests with report publication on GitLab

Configure GitLab CI to run automated tests written on python using pytest framework and to publish allure test results to GitLab pages

This post consists of two parts:
1. How to configure GitLab CI to run automated tests written on python using pytest framework.
2. How to publish allure test results to GitLab pages.

Configuring GitLab CI to run pytest automated tests

Assuming you have already created a test suite and you can run it locally from the console. If not yet, read in this post how to create a test suite with pytest. So now you are ready to add the instructions for Gitlab CI.

GitLab CI reads the instructions from the .gitlab-ci.yml file. You can create this file manually in the project’s root or it will be created automatically when you start configuring CI pipeline. If you want to create it manually then skip it, otherwise pull the repository with your tests into GitLab. Then go to your project’s page and follow the path: CI/CD -> Editor

At the beginning of the pipeline setup, we need to set the names of the stages that will be executed during the process. In this first part, we are configuring only the execution of the automated tests. So we need only one stage. Let’s name it “test”.

stages:
    - test

Now we will create the job that will run the tests. First of all – the job’s name. It can be anything here. Inside the job, there are parameters that we need to set:
– stage
– image
– before_script
– script

Stage

The name of the stage where this job should be run. In our case, it is “test”.

Image

Each job in the pipeline is executed in its container (or instance) that is created from the specified image. It is possible to miss this parameter and a default image will be used. In this case, we will need to add commands to manually configure the environment. To avoid this we will set the image that is suitable for us: “python:3.9” – for example.

Before script

Basically, you can join commands from this section with the commands from the section “script”. But let’s make it neat. Here we will install the dependencies that we need.

Script

Here we will run our tests

Here is the job:

run_tests:
    stage: test
    image: python:3.9
    before_script:
        - pip install -r requirements.txt
    script:
        - pytest -v # I always use ‘-v’ to add verbosity to the input

The whole .gitlab-ci.yml file now looks like this:

stages:
    - test

run_tests:
    stage: test
    image: python:3.9
    before_script:
        - pip install -r requirements.txt
    script:
        - pytest -v

Now you can press the button “Commit changes” and the pipeline will start execution immediately. You can see the logs in CI/CD -> Jobs -> open the last job.

Publish allure test results to GitLab pages

Now we will add jobs that will handle the reporting part.

First of all, we need to change the “run_tests” job.

  1. Add alluredir parameter to the script
  2. Add artifacts section. The artifacts are the files that will be saved as a result of job execution. We will add 3 parameters to this section:
    • paths – the path to the alluredir
    • when – we will set ‘always’ because we want that artifacts are saved after every “run_tests” job execution
    • expire_in – how long the artifacts should be stored. We will set 1 day because the artifacts of this job will be copied by the following job and we will not need them anymore
  3. Set allow_failure to True because in case if some cases are failed all the pipeline will be marked as failed and the results will not be published

Changed job:

run_tests:
    stage: test
    artifacts:
        when: always
        paths:
            - allure_results
        expire_in: 1 day
    image: python:3.9
    before_script:
        - pip install -r requirements.txt
    script:
        - pytest -v --alluredir=allure_results
    allow_failure: true

For the reporting we need to perform the following actions:

  • Copy the results of previous executions
  • Generate the report that consists of historic results and the results from the current tests execution
  • Publish the report to GitLab pages

Copy the results of previous execution

First of all we will add a new stage to the stages

stages:
    - test
    - history_copy

And now we will create the job. I will explain only the script section as far as all the other sections were explained previously.

By the first command, we will copy the results of the last successful pipeline execution. Note that this command will work only in case if you have a public repository. If that’s not your case see the workaround here.

The second command is for unpacking the downloaded contents.

By the third command, we are changing the access rights to the unpacked folder so that we can manipulate it further.

By the fourth command, we will copy the historic data into the alluredir folder.

Also, we need to add “allow_failure:true” because at the first run there will be no historic data to download and the pipeline will fail without this parameter.

history_copy:
    stage: history_copy
    image: python:3.9
    artifacts:
        when: always
        paths:
            - allure_results
        expire_in: 1 day
    script:
        - 'curl --location --output artifacts.zip "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/jobs/artifacts/main/download?job=pages&job_token=$CI_JOB_TOKEN"'
        - unzip artifacts.zip
        - chmod -R 777 public
        - cp ./public/history/* ./allure_results
    allow_failure: true

Generate the report

As usual – new job with new stage:

stages:
    - test
    - history_copy
    - generate_report

The job will use another image that has the allure installed. The script will do only one thing – it will generate the report from the alluredir folder into a new folder “allure_report”.

generate_report:
  stage: generate_report
  image: frankescobar/allure-docker-service
  script:
     - allure generate -c allure_results -o allure-report
  artifacts:
    paths:
      - ./allure-results
      - ./allure-report
    expire_in: 1 day
  rules:
    - when: always

Publish the report to GitLab pages

Add a new stage:

stages:
    - test
    - history_copy
    - generate_report
    - report

Here the name should be exactly “pages”. This will tell GitLab CI that it needs to publish the artifacts to the pages.

In the script section, we are creating a new directory named public and then copying the generated report there.

In the artifacts section, we set the ‘expire_in’ parameter to ‘never’ as we want to have the access to the results of every execution forever.

Also, we set that the job needs that the previous job be completed successfully. We point it out in the ‘needs’ parameter of the job.

pages:
  stage: report
  script:
    - mkdir public
    - cp -r ./allure-report/* public
  artifacts:
    paths:
      - public
    expire_in: never
  needs:
    - job: generate_report

Finally the .gitlab-ci.yml file should look like this:

stages:
    - test
    - history_copy
    - generate_report
    - report

run_tests:
    stage: test
    artifacts:
        when: always
        paths:
            - allure_results
        expire_in: 1 day
    image: python:3.9
    before_script:
        - pip install -r requirements.txt
    script:
        - pytest -v --alluredir=allure_results
    allow_failure: true

history_copy:
    stage: history_copy
    image: python:3.9
    artifacts:
        when: always
        paths:
            - allure_results
        expire_in: 1 day
    script:
        - 'curl --location --output artifacts.zip "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/jobs/artifacts/main/download?job=pages&job_token=$CI_JOB_TOKEN"'
        - unzip artifacts.zip
        - chmod -R 777 public
        - cp ./public/history/* ./allure_results

generate_report:
  stage: generate_report
  image: frankescobar/allure-docker-service
  script:
     - allure generate -c allure_results -o allure-report
  artifacts:
    paths:
      - ./allure-results
      - ./allure-report
    expire_in: 1 day
  rules:
    - when: always

pages:
  stage: report
  script:
    - mkdir public
    - cp -r ./allure-report/* public
  artifacts:
    paths:
      - public
    expire_in: never
  needs:
    - job: generate_report

Now, after you commit the results and the tests execution is finished you can find the link to the report here: Settings -> Pages

By Eugene Okulik

Follow me on twitter