As a Senior Software Engineer for JUMO, Anes Djebbar had the opportunity to lead the implementation of an End-to-End (E2E) testing framework for our microservice architecture.
We all know the absolute importance of testing code, so we won’t get into the details of the advantages of well tested applications, however, we will share Anes’s experience of automating the often arduous process.
Why E2E Testing?
Back in 2019, we were in the process of migrating from a monolithic architecture to a microservice architecture, and we needed a way to verify that the integration between services would remain stable at all times.
It was also crucial to test our services in production-like deployment clusters, adjacent to actual infrastructure, to gain confidence about progressing changes to production, and to ensure the quality of our code as well as the quality of the financial products our code supports.
Continuous Deployment often comes with a microservice architecture, which means that we want to deploy code changes to production as soon as they are ready, which requires a high level of trust in our code.
That’s when we realised: what we need is an automated end-to-end (E2E) testing framework.
What Framework?
The mission was clear: build a framework that could take advantage of existing tech skills in the company, be simple to use and maintain, easy to integrate with a deployment pipeline, and open to everyone in engineering.
We considered several third-party options, but ultimately we decided to build our own using Kotlin so we could tailor it to our specific needs.
Long story short, we went with a single repository containing all the tests, divided by suites, and a few packages for common code, network calls, infrastructure tools etc. Each team would have their own tests to maintain, and due to the nature of the E2E tests, each test can reuse other components instead of having to re-invent the wheel every time. The tests would be executed by a job on Kubernetes, after the deployment to our development Kubernetes cluster. Each job would run a specific suite of tests based on the service being deployed and either fail the deployment or let it go through if successful.
How
After the design came the hardest challenge of all: how to convince all the engineering teams to take part in the project instead of relying on their own solutions, if any.
We were basically asking teams with different priorities and working processes to change over and to start doing more. It required a cultural shift within the organisation. We had to make a strong case for the benefits of E2E testing and how it would ultimately make the development process more efficient and reliable.
They say that a picture is worth a thousand words, and a proof of concept a million!
So, we decided to actually build a first version of the tool, and test the interaction with a new service planned for release soon, integrate it in the new deployment pipeline and eventually make it the first service to have a full Continuous Integration/Continuous Deployment pipeline in the company.
The service works by ingesting some data from AWS S3 files, then persists them in PostgreSQL and makes them available through a REST API.
The E2E test was quite simple:
- Generate and upload a file on an AWS S3 bucket containing some data
- Wait a few seconds
- Query the API of the service
- Assert that the response is correct and contains the data previously uploaded
We ran it, and… epic fail! While the file on S3 was ingested by the service, the data could not be persisted in the database due to a mis-configuration of the permissions in AWS.
The first automated E2E tests in JUMO helped us discover an infrastructure bug before reaching production. Massive win!
Who
Once we proved it was useful, not that we ever doubted it, people started adding new tests, which was great! But what about ownership? As mentioned previously, this was a single repository, owned by everyone in engineering (so Pull Requests/merge permissions were not restricted). And if everyone owns it, then generally nobody is responsible.
We quickly started to have situations where one team makes a change to a reusable component that inadvertently breaks another team’s use-case, making their test fail and blocking their pipeline. Not cool.
We therefore decided to create a virtual team, with at least one representative member from each engineering team, to be responsible for approving and merging new PRs. This helped to ensure that each team was held accountable for their own tests, while also providing a forum for collaboration, feedback and improvement. There are two simple rules:
- Anybody in JUMO can create PRs
- Merge is only allowed for members of the virtual team, if approved
This acts as an open-source model within the company, having a set of maintainers with full context and contributors that branch the code and open PRs.
Final thoughts on E2E testing
I think the most important lesson gleaned from this exercise is that there are no one-size-fits-all solutions. We had to make a model that could evolve on a regular basis. And ultimately, implementing an E2E testing framework for our microservice architecture has been a significant improvement in our development process. It has helped us achieve Continuous Deployment and to ensure that our system is functioning correctly before deploying to production ✅.’
Connect with Anes on LinkedIn: https://linkedin.com/in/anesdjebbar
