Skip to main content

How to Perform Unit Testing for AWS Glue Jobs in an Azure DevOps Pipeline

Unit testing AWS Glue jobs presents challenges due to the complexities involved in replicating the Glue environment locally. Fortunately, AWS offers a solution in the form of Glue container images available at Glue container images. These images allow us to perform unit tests effectively, as outlined in detail in the official documentation here. In this blog post, we will delve into the process of running AWS Glue job unit tests within an Azure DevOps pipeline and discuss how to calculate and publish code coverage for these tests.

To begin with, the Glue container image operates under a special user named GLUE_USER, which is referenced in the associated dockerfile.

USER glue_user

Assuming you have developed your Glue job in a Python script named myawesomegluejob.py, which is stored in an Azure DevOps (AzDO) Git repository, creating a pipeline for this purpose might initially seem straightforward. However, executing build steps directly within the Glue container is not feasible due to permission constraints with the GLUE_USER.

To overcome this limitation, our approach involves leveraging Docker commands in the pipeline to fetch the Glue image and subsequently mounting the Azure DevOps pipeline's file structure inside the Glue container. This facilitates the sharing of test results and code coverage data back to the Azure DevOps pipeline for future utilization.

By default, the Azure DevOps pipeline file system is not writable by the GLUE_USER. To address this, we must grant access to all users by executing the command chmod -R 0777 $(Build.SourcesDirectory).

Next, we can execute the following command:

docker run -v $(Build.SourcesDirectory):/home/glue_user/workspace -w /home/glue_user/workspace public.ecr.aws/glue/aws-glue-libs:glue_libs_4.0.0_image_01 -c "pip install pytest pytest-azurepipelines pytest-cov; python3 -m pytest test --doctest-modules --junitxml=junit/test-results.xml --cov=main --cov-report=xml"

This command effectively mounts $(Build.SourcesDirectory) into the /home/glue_user/workspace folder within the container. By setting the working directory to /home/glue_user/workspace, we proceed to execute a series of commands that install the necessary Python libraries and perform the unit tests. Consequently, a coverage.xml file is generated at $(Build.SourcesDirectory). However, as this file is created within the container, it contains relative paths of the container in its sources node. To rectify this, we conduct a string replacement using the sed command.

Here's the relevant snippet encompassing the aforementioned steps in the Azure DevOps pipeline:

- job: 'Scan_and_Build'
  steps:
      - script: |
          docker pull public.ecr.aws/glue/aws-glue-libs:glue_libs_4.0.0_image_01
      displayName: 'Pull Glue Image'

      - script: |
          chmod -R 0777 $(Build.SourcesDirectory)
          docker run -v $(Build.SourcesDirectory):/home/glue_user/workspace -w /home/glue_user/workspace public.ecr.aws/glue/aws-glue-libs:glue_libs_4.0.0_image_01 -c "pip install pytest pytest-azurepipelines pytest-cov; python3 -m pytest test --doctest-modules --junitxml=junit/test-results.xml --cov=main --cov-report=xml"
          sed -i "s|/home/glue_user/workspace|$(Build.SourcesDirectory)|g" $(Build.SourcesDirectory)/coverage.xml
      displayName: 'Run tests'

      - task: PublishTestResults@2
      condition: succeeded()
      inputs:
          testResultsFiles: '**/test-*.xml'
      displayName: 'Publish unit test results'

Conclusion

This blog post has effectively demonstrated how to perform unit tests for AWS Glue jobs within an Azure DevOps pipeline. By leveraging Glue container images and integrating Docker commands, it becomes possible to seamlessly execute unit tests and publish code coverage data, thus ensuring the reliability and stability of your Glue jobs.

Comments

Popular posts from this blog

Integrating React with SonarQube using Azure DevOps Pipelines

In the world of automation, code quality is of paramount importance. SonarQube and Azure DevOps are two tools which solve this problem in a continuous and automated way. They play well for a majority of languages and frameworks. However, to make the integration work for React applications still remains a challenge. In this post we will explore how we can integrate a React application to SonarQube using Azure DevOps pipelines to continuously build and assess code quality. Creating the React Application Let's start at the beginning. We will use npx to create a Typescript based React app. Why Typescript? I find it easier to work and more maintainable owing to its strongly-typed behavior. You can very well follow this guide for jsx based applications too. We will use the fantastic Create-React-App (CRA) tool to create a React application called ' sonar-azuredevops-app '. > npx create-react-app sonar-azuredevops-app --template typescript Once the project creation is done, we

Creating a Smart Playlist

A few days earlier I was thinking that wouldn't it be nice if I had something which will automatically generate a playlist for me with no artists repeated. Also, it would be nice if I could block those artists which I really hate (like Himesh Reshammiya!). Since I couldn't find anything already available, I decided to code it myself. Here is the outcome -  This application is created entirely in .NET Framework 4/WPF and uses Windows Media Player Library as its source of information. So you have to keep your Windows Media Player Library updated for this to work. It is tested only on Windows 7/Vista. You can download it from here . UPDATE : You can download the Windows XP version of the application here . Please provide your feedback!

Add Git Commit Hash and Build Number to a Static React Website using Azure DevOps

While working on a React based static website recently, there was a need to see exactly what was deployed in the Dev/Test environments to reduce confusion amongst teams. I wanted to show something like this: A quick look at the site's footer should show the Git Commit Hash and Build Number which was deployed and click through to actual commits and build results. Let's see how we achieved this using Azure DevOps. Git Commit Hash Azure DevOps exposes a variable called  $(Build.SourceVersion) which contains the hash of the commit. So I defined a variable in the Build Pipeline using it. Build Id and Build Number Azure DevOps also exposes two release time variables  $(Build.BuildId) and  $(Build.BuildNumber) which can be used to define custom variables in the pipeline. So we have a total of 3 variables defined: Next we use these variables in our React App. I created 3 global variables in index.html and assigned a token value to them. < script   type = "text/JavaScript&quo