Optimize Jenkins build time for parallel tests
The approach to optimize Jenkins builds go hand in hand with test parallelization. Running Parallel tests requires separate Jenkins build containers. The Time consuming issue is how to distribute the build artifacts across the build containers.
How to optimize Jenkins build by replacing stash/unstash
In projects with an high quality approach, every change has to be checket by using test automatization. To satisfy the quality approach, a so called test pyramid is used. Basically it means that a big number of tests (unit tests) with low system resource usage is been used to cover as much as possible test cases. On the other hand side a smaller number of tests with an hight quality approach and high system resource usage is used (Mock Tests and System Tests).
Running Mock tests means that a Front End (FE) application send its requests (rest calls) to an simulates backend (BE) returning a predefined response. With systemtests a real running backend is used to send a response to a requesting frontend service.
Test parallelization
In many projects the test suite playwright originally developed by Microsoft is used for mock- and system tests. The way how the tests are divided to a number of testifies is basically an issue solved by the developers. One easy option is to split the tests to a fix number of test files an than running every test file on a separate Jenkins build container.
When ever Jenkins steps are running on separate build containers, it is an issue that the build artifact must be available on every build container. With stash and unstash Jenkins provides internal functions to move artifacts to an internal storage.
stash includes: ‘dist/**/*’, name: ‘buildApplication’ unstash ‘buildApplication’
The issue with Jenkins stash and unstash is that it is ineffective in case of big artifacts.
(…) Note that the stash and unstash steps are designed for use with small files (…) https://www.jenkins.io/doc/pipeline/steps/workflow-basic-steps/
In addition it might be useful to keep the build artifacts for a longer time then just for one build.
Architecture Diagram
Example
A frontend application is generated with npm ci. The package.json is fetched from a git repository, the corresponding package-lock.json can be used to create a hash. The hash value can be used as a directory name on the data storage where the build artifact can be copied to.
Jenkins stash / unstash can be replaced by using s3 storage. Other alternatives are nexus or artifactory. The aws s3 cli and s3fs can be installed on the build container. Using s3fs is an easy way to include the s3 storage into the local filesystem just by creating a month point.
This is an example how it can be done
export AWS_ACCESS_KEY_ID=<AWS_ACCESS_KEY_ID> export AWS_SECRET_ACCESS_KEY=<AWS_SECRET_ACCESS_KEY> export S3_BUCKET_NAME=<S3_BUCKET_NAME> export S3_ENDPOINT_URL=<S3_ENDPOINT_URL>
echo $AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY > ~/.passwd-s3fs chmod 600 ~/.passwd-s3fs [ -d ~/s3-mount ] || mkdir ~/s3-mount
s3fs $S3_BUCKET_NAME ~/s3-mount -o passwd_file=~/.passwd-s3fs -o use_path_request_style -o url=$S3_ENDPOINT_URL
_PACKAGE_DIR=$(shalsum package-lock.json) tar - cf node_modules.tar node_modules mkdir ~/s3-mount/$_PACKAGE_DIR rsync -av node_modules.tar ~/s3-mount/$_PACKAGE_DIR