A few months ago, I started a POC about testing and automation testing and find a good way to improve the current test flow that we have and find a good way to have E2E testing as well. firstly I’ll talk about the different kinds of tests and then we will talk about other big companies’ test flow, etc.
Introduction
Automation testing is vital to ensure that software is effective. Further, it increases test execution’s speed and test coverage. The main benefit of automation testing is that it does not require human intervention, saving a tremendous amount of time.
As we move towards the top layers of the pyramid, the scope of the tests increases, and the number of tests that must be written decreases.
Services communicate with each other over a network, Therefore Unit testing is all the more important in this context to validate each business logic aka microservices separately.
BUT, UNIT TESTING IS NOT ENOUGH!
Unit Test/ Integration Test
- size of the unit test is not defined anywhere, consider writing them at a class level or a group of related classes.
- keeping the unit testing as small as possible.
- The integration test collects microservices together to verify that they collaborate as intended to achieve some larger piece of business logic.
- Exercises communication paths through the subsystem to check for any incorrect assumptions each module has about how to interact with its peers.
TestDoubles
Test Double is a generic term for any case where you replace a production object for testing purposes.
In the microservices, It is a big deal, which requires its own build tools, environments, and network connections.
Notice: For services, such tests may run against an in-process test double, using something like mountebank
Why We Need To Use TestDoubles?!
- services are being maintained by a different team.
- they may be subject to slow.
- and unreliable networks.
- and ……
That’s why a test double is handy, it stops your tests from being slow and unreliable.
Question: TestDoubles are truly faithful? => YES, IF WE HAVE ContractTest
ContractTest: One of the most common cases of using a TestDouble is when you are communicating with an external service.
Question: what happens if the external service changes its contract? There is a failure in our system! so Run a separate set of contract tests, These check that all the calls. A failure in any of these contract tests implies you need to update your test doubles... Notice: communication with the external service supplier is even more important if a service is being used as part of a production application.
Integration testing in depth
According to our description of TestDoubles, and an article created by Martin Fowler, we have two types of integration tests:
Narrow Integration Tests
- Exercise only that portion of the code in my service that talks to a separate service.
- Uses test doubles of those services.
- Consist of many narrowly scoped tests (Usually run with the same test framework that’s used for unit tests)
Broad Integration Tests
- Require live versions of all services, requiring substantial test environment and network access.
- Exercise code paths through all services, not just code responsible for interactions.
Broad Integration Tests == “system test” or “end-to-end test”.
By using both narrow integration tests and contract tests, I can be sure that I’m integrating with an external service correctly without needing to run tests against an actual instance of the service. This makes it much easier to build and deploy my software with confidence.
Other Companies’ experience
In this section, I’ll walk through some big companies’ experiences regarding Test flow implementation.
Soundcloud
They have more than 300+ services and they use Contract tests for them.
1. “As a consumer of an API, you write a “contract”. This contract states what you expect from the provider (of an API). Based on this contract generates a mock of the provider.
2. As a consumer, you can test your application against the mock of the provider.
3. The contract can be sent over to the provider who can validate if the actual implementation matches the expectations in the contract.”
How does it work?
The best solution would be to cooperate on the contract and find the solution which works best for everyone.
They are using Pactflow to implement their contract testing, and the below pictures is the flow of the works on Pactflow and their flow of contract testing.
Spotify
It’s an interview with Spotify Engineer:
how do you do the tests in Production System?
When I say, we do testing in the Production System; it means that we are using our production data. We use test accounts. So, we don’t use anybody’s real user accounts. We are only using our accounts, and those test accounts are the ones used in production.
Do you like to create dummy users and then do the tests?
They are not dummy users; they are real test accounts.
As far as I know, Facebook does this, too but they have different clusters. So if something happens with the production data, something goes wrong then they change the cluster. Do you do something like this?
We always have an open discussion and thoughts on what problems we might get with this methodology. We are not doing any writing. We don’t destroy anything. We are not capable of interacting with banking data or anything like that. That’s off the table. It’s not there. What we do is create a playlist, listen to tracks, and check the amount of traffic. We all live in the same ecosystem. I mean from the production point of view we are just another user. We don’t need to do that yet as Facebook does. I don’t know how they do it.
What we should aim for instead of Integration Tests, which verify the correctness of our service in a more isolated fashion while focusing on the interaction points and making them very explicit.
Let’s use some services we have at Spotify as real-life examples.
Having Implementation Detail Tests there would just be in the way. Now we can refactor the internals without touching any tests. We could even replace the database from PostgreSQL to NoSQL without having to modify the actual test methods. The only thing that would need to change is the test setup.
End to End Test (E2E)
In the beginning, I thought Robot Framework which is an open-source framework is one of the good choices because with it we can use Selenium and other resources in it, but after doing some research about the other tools like Cypress, Playwright, Puppeteer … and also when I hear from my colleagues in other teams, I’ve changed my decision to use another framework.
One of my colleagues said Cypress is no longer an option for us so we need to compare Playwright and Puppeteer together and determine which one works best for us. Playwright is a refactoring of Puppeteer by developers who joined Microsoft to develop it. These two frameworks have very similar syntax, so migrating between them is easy!
The advantage of Playwright is providing API Testing as a black box, so the back-end side can be tested without requiring E2E and UI testing.
- Playwright vs Puppeteer: A Tester’s Perspective
- Cypress vs Playwright: Which is best for E2E testing?
In Playwright:
- We can set up several configurations to test the local domain, staging domain, and production domain separately.
- Multi-user testing is possible.
- We can have a nice report (by adding –reporter=html to the playwright command) and code coverage (by using the NYC package) This package makes it easy to mock data.)
- here is the List of the devices which the Playwright support.
- Playwright has experimental support for Android automation.
Example of Real Use Case: I want to run a test on a shop on a Galaxy Tab S3 with Android 10 and Chrome for Android… Without having a Galaxy Tab S3 ourselves…
If we want to have Automation tests on different OS:
Conclusion
Automation tests help us a lot to have enough confidence when we want to deploy a new feature. It’s better to use Narrow Integration Tests during our CI pipeline and then we can speed up the CI/CD flow in our platform. And also use Contract Testing as well as communicating with External Services, such as Payment, etc.
In Broad Integration Testing we can use a framework like Playwright, Cypress, etc. There are many tools that we can use to implement our Automation Tests, and as I researched most of them used Selenium, such as Katalon, Kobiton, etc. The choice of framework for your platform depends on your project and team. It’s up to you to consider the options and decide which one is best suited for your needs. in our case, because we needed to have several users at the same time to test, Playwright was a good option for us.