Deploy with Speed and Confidence Using Contract Testing and Pact

Rate this content
Bookmark

It’s almost 2021 and we still rely on integrated environments and large end-to-end test suites to release complex, distributed applications called "software". In this talk, Matt breaks down the arguments for such nonsense and shows how a better, faster, safer alternative.

Matt Fellows
Matt Fellows
32 min
15 Jun, 2021

Comments

Sign in or register to post your comment.

Video Summary and Transcription

The Talk discusses the cost and issues with end-to-end integration tests and the benefits of using contract testing with PACT. It explores the challenges of scaling teams and components and highlights the advantages of using PACT for testing microservices. The PACT framework is demonstrated, showcasing its ability to provide fast and reliable feedback, independent testing, and versioned contracts. The Talk also covers topics such as testing compatibility, safe removal of fields, and integrating PACT with Cypress.

Available in Español

1. Introduction and Agenda

Short description:

In this part, Matt Fellowes introduces himself as a core maintainer of PACT and co-founder of PACTFlow. He discusses the agenda for the talk, which includes the problem with end-to-end integration tests, how PACT works, and the principles of contract testing. He also mentions the cost associated with the current way of testing microservices.

Well, thanks, everyone, for coming for my talk to deploy with speed and confidence using Contract Testing at PACT. My name is Matt Fellowes. I'm a core maintainer of PACT. I'm also the co-founder of PACTFlow, which is a continuous delivery microservices platform. And you know, if I wasn't working in IT, I'd probably be working in sports and fitness to get away from it all. If you want to contact me after this talk, you can follow me on my handles below.

So the agenda for today, we're gonna be talking about how to release software, and particularly distributed systems. We're gonna start with talking about the problem with end-to-end integration tests. Then we're gonna talk about how PACT works, and the principles of contract testing. And then we're gonna finish off with a bit of a demo.

So the old way, or the current way a lot of companies test their microservices, is to do what we call end-to-end integrated tests. And what that requires is you to stand up your entire platform, something like this, and use a functional API testing tool, like Postman or whatnot, and drive requests for the entire system. So for example, you push it through...maybe not a user interface with Postman, but you push it through the platform, and it's gonna pass through all the layers of the system. So Microservice A, Microservice B, Microservice C, and the request is gonna make its way through all real systems, it's gonna pass over real networks, it's gonna talk to real databases, send emails, whatever it is that your actual application is going to be doing. Now this is great, if the system works, and the tests pass, it does give you some level of confidence that your system's working as expected, but this kind of testing comes at a huge cost.

2. Issues with End-to-End Integration Tests

Short description:

End-to-end integration tests have several issues. They are slow, fragile, and costly to maintain. Identifying and resolving issues can be time-consuming and challenging. Achieving full coverage is difficult, and the tests don't scale well. They require deploying everything together, leading to dependencies and delays between teams.

The first issue is that it's slow, testing to pass with real layers and need to do real things and this can be slow, of course. But also oftentimes they can't be run in parallel, and the reason for that is the stateful nature of these types of interactions.

The second issue with these types of testing is that they're fragile and they can be nondeterministic. So this property of flakiness is very present in these types of testing. So even if they do pass, they may take multiple runs to get there.

For example, you need every version of every service in the system to be lined up. If any of those change, the test could break. If you've got the wrong version of data, the wrong version of the tests or the wrong configuration for your environment or just that somebody has tampered with the environment in advance, it's possible that your test will fail. They're very costly to maintain.

And when you do find issues, or when you do have an issue, finding the actual problem and the source of the issue can actually be really costly to do. So, for example, if you have a failure that's causing Microsoft's B here, well, it may not be visible from the outside why that test actually failed. So you might need to go digging through your log platforms like Splunk or Simulogic, you have to trace some correlation IDs through the system to actually find out what the problem is. And then you need to find the code version for that particular service, go to the repository, and go digging.

It's basically like finding a production issue. So it can be quite costly just to find the bug itself. Often times it fails just because of those flaky reasons I mentioned earlier. Similarly, it's difficult to achieve full coverage this way. So what I mean by that is you've got multiple systems here and you've got a lot of different potential scenarios that can go on. And so running your tests this way, it's very possible that you're not going to get all the tests you want. Because a, they take so long to run. B, they're costly to maintain. And also you just literally cannot run that many tests in the amount of time, because the combinations spiral out of control.

Because you test everything together this way, well, then, you really have to release everything together this way. Because you don't have confidence if I deploy just a single component that things will continue to work at the end of it. So you now need to deploy things together. And doing that means you've got teams coupled with each other at release time. And that means teams are waiting on other teams to get things done. And we know from Agile theory, that's not very good. And so these types of tests don't scale well. They tend to get worse and worse over time.

3. Challenges of Scaling Teams and Components

Short description:

Increasing the number of teams and components leads to a nonlinear relationship with the number of environments or their complexity. Build time, build complexity, failure rate, and risk associated with change all increase. This results in developers being idle, queues forming, and delays in completing work. These issues come with a significant cost.

So let's say you increase the number of teams and components over time in this linear fashion. What you see is this nonlinear relationship with the number of environments or the complexity of environments, that you need to manage. You see that the build time goes up, or the complexity of those builds, and the failure of those builds starts to go up. You also see the risk associated with change, moving up exponentially. And of course, we now have developers idle, we have a lot of queues, we have, you know, cards on the wall, that are referencing other teams waiting for their work to be done, before we can get ours done. And this all has a huge cost associated with it.

4. Benefits of Pact for Testing Microservices

Short description:

Good tests are fast, isolated, easy to reason about, and easy to maintain. Writing separate unit tests for the consumer and provider can be effective, but they may not reflect reality. Contracts provide a solution by allowing consumers to specify their needs and providers to implement them. This approach ensures that changes don't break consumers and provides documentation. Pact is an open-source tool that combines mocks, unit tests, and contracts to prevent drift. It's designed for testing microservices and distributed systems and aims to eliminate the need for end-to-end tests and complex test environments.

Of course, good tests have the exact opposite properties of what we just talked about. They're fast, they're isolated, they're easy to reason about and they're easy to maintain. Could mocks come to the rescue here? Well, as you've probably all written, it's very simple to write two separate unit tests on either side of the service boundary. We can write a unit test for the consumer, mocking out the provider, and we can write a unit test for the provider, simulating the consumer. And these are great. They can run really fast, they can be easy to fix, easy to find bugs with. Of course, they may not represent what actually happens in production, and so it's quite possible for you to put an assumption in there that's not valid, and because it's a unit test, we're not checking that assumption later on. And so, we have this nice property of, you know, all these great properties, but the new problem is that they're not actually representing reality. It's hard to keep both sides in sync.

And this is where we can talk about contracts. So, you're probably familiar with specification first design, where an API producer specifies or creates a contract using Swagger or something else, and they publish that document to all of its consumers. And there's a number of great properties that come with this. But one of the downsides is when you move and change and modify that version of the contract or the specification, it's easy to accidentally break a consumer because you don't know what parts of the API they're using. And it requires a lot of diligence to ensure that you don't push out backwards compatible changes. This is where we can talk about consumer driven contracts, which inverts that relationship. Consumers can specify what they need of the provider. And write those in their own form of a contract and give it to the provider, each consumer having its own potential subset of the API. And then the API just needs to implement the superset of all those contracts, and then it can get its job done. This has some really interesting side effects, or consequences. The first one is that you'll know when you break a consumer, because the consumers are telling you what they use. You get a form of documentation, because the consumers are giving you exactly what they're using every time they push a build, and you can test things independently.

So this is where the tool pact comes in. It combines the idea of fast isolated mocks and unit tests with contracts to ensure that they don't drift. It's an open source, consumer driven contract testing tool, and it's designed to help you test your microservices and distributed systems independently. Its main use cases are in things like JavaScript web applications, native mobile apps, talking to restful services, JSON or XML, or talking of message queues. So think of Kafka or SNS and these kinds of things. Its goals are to reduce or remove entirely the need for those end-to-end integrated tests and reduce reliance on those complicated test environments. Pax benefits are basically, because you've got focus for a single integration point, you're only looking at one thing at a time. You don't need to deploy. So because you don't need to deploy, you don't need a test environment to do this form of testing.

5. Pact Framework and Demo

Short description:

Fast and reliable feedback. Tests run fast and scale linearly. Independent testing allows for independent releases. Pact framework: consumer writes test to define provider expectations. Pact mocks out the provider and simulates the provider API. Interactions are recorded as a contract. Contracts are shared and versioned. Contracts are replayed against the provider to ensure symmetry. Demo using React product catalog website and ExpressJS backend. Testing the product ID endpoint. Consumers drive API design. Provider workflow and release gating with Can I Deploy?

You get fast and reliable feedback because of it. The bug is always going to be found on your machine. You don't need to go digging through logs. This means those tests run really fast and they scale linearly. And last, because you're testing things independently, you can now release them independently.

Okay, let's quickly talk about how Pact works and then we'll show how it works in action. So we have a consumer website... Sorry, we have a website that's talking to a product API. We call the website a consumer and we call the API the provider. And the messages that pass between that, the sum of those, we call that the contract. So it's a consumer-driven contract testing framework and so the first thing we're going to do is we're going to write a test from the consumer side first to define the expectations of the provider. So what Pact will do is mock out the provider. We never make them talk to each other. But Pact will simulate the provider API. And the consumer can say, given that I make a request to get 1, 2, 3, 4 for the product endpoint, I expect to get back some response. And we do this for all the things the consumer needs of the provider. At the end of that session, we're going to record those interactions into what we call a Pact file or a contract. We're going to share that with a tool like the Pact broker or Pact flow, which will help us exchange the contract and version the contract across our ecosystem. And then finally, on the provider side, what we're going to do is we're going to pull down the contracts from Pact flow, we're going to replay them back against the provider. And Pact is now going to simulate the consumer. It's going to replay this request, check the responses, and if they match what the consumer does, we now have symmetry on both sides of this interaction. We have two fast mocks, and we've now got a contract that's ensuring that those two mocks are actually valid.

Okay. So, we're going to get into our demo. We're going to use a React product catalog website that talks on an ExpressJS backend as our example, we're going to test the product forward slash ID endpoint and show how consumers can help drive the API design. We're going to look at the provider workflow, we're also going to look at how we can gate releases with a tool called Can I Deploy? So, let's look at the application. So here it is, it's a very uninteresting website, I'm sorry, it's not the greatest React website of all time. But you can see here, the home page just lists the products and we can drill down into an individual product to get the data for it. And they're going to hit two different endpoints. We're going to test the endpoint for this page.

6. Testing the Get Product Method

Short description:

We'll focus on testing the get product method of the API class. The provider side also has a similar setup with routes and a Pact test. To run the Pact test, we stand up the provider, specify the Pact files, and replay them. The contract is shared with a Pact broker, showing the current state of the interaction. We can make changes and promote them through environments.

Okay. Looking at our code for this. We'll jump into the consumer test. So, if we look over here at our product page, this is our React component. You can see here, to populate this page, we need to get some data from an endpoint. Now, instead of actually loading this in, in other way, we're going to talk to an API endpoint and we're going to use a class to do that. So, this API class has got a method called get product, and that's what's going to get the product data for us.

So, from a Pact point of view, we can test this method, this is what we care about, this is the target of our Pact test, we don't need to test anything with React to do this form of testing. And, you can see here, it's just going to hit the product forward slash id endpoint, it's going to send some headers and it's going to then convert the data it comes back into a product class, and the product class looks like this.

On the provider side, we have a similar thing, we've got a product definition over here, we have our routes that deal with the different endpoints, and we have a Pact test over here as well. We're not going to get too much into the pact test here, because it's a lot of config, but basically to run the Pact test on this side, we're just going to stand up the provider, tell Pact how to find the Pact files, and we'll replay them against the provider. Lastly, we're going to share the contract with a Pact broker, in this case, it's a posted Pact flow. And it's going to show us the current state of the interaction over time. So you can see here that the current version of the consumer is in the master, and it's been deployed to production and the provider has also been deployed to production as well. We can drill in to the Pact, and we can see the various interactions that are supported by this contract. In this case, getting a product with ID 10, we get the ID, the type and the name back in the body. But most importantly, we're going to use this to show how we can make changes into the system and then promote them through environments.

7. Consumer Pact Test

Short description:

Let's examine the consumer pact test, where we follow the arrange, act, and assert model. We specify the expected product with ID 10, type credit card, and name 28 degrees. The like matcher ensures that the keys exist and are of the same type, allowing flexibility in the values. We have a library that simplifies matchers for future use.

Okay, so let's look at the consumer pact test first, because this is where we start. This is our pact test here. We're going to follow the standard arrange, act and assert model just to see how this all works together. So first up, we need to tell Pact, we need to tell our unit test what our code is about to do. As I said, Pact is a marking tool. It's going to validate what we actually do. So given that a product with ID 10 exists, making a call to get that product, using the get verb at this path. We expect to get back an HTTP 200 with some headers and a body that looks like line 19. You can see the expected product is ID 10, type credit card, name 28 degrees. The like matcher here basically says, we don't care about the values here, we just care that the keys exist and they're of the same type. So later on when the provider verifies this, we're not going to fail if different IDs come back or even different products come back. We're not going to get into matchers today either, but we have a flexible library we can use to make this much simpler.

8. Calling the API and Adding Price Field

Short description:

We configure our API client to talk to the PACKT mock service. We call the method, which talks to PACKT instead of the real API. We write unit test assertions for this call. We add a new field, price, to display in the React component. We simulate a CI process, but the provider hasn't implemented the new field, so it's not safe to release. The test passes, and the contract is published to PACKFLOW. A new contract called FeetAddPrice is created, but it's yet to be verified. The Consumer version can't be deployed until the provider implements the new features.

The second thing too is to actually call the API. We configure our API client and rather than talk to the production client, we're going to talk to the PACKT mock service. Then on line 42, we call the method. That method there is going to call a real HTTP endpoint, but in this case, it's going to talk to PACKT instead of the real thing. And then on line 45, all we need to do is write our unit test assertions for this call.

So, let's pretend like PACKT doesn't exist. Well, what should we test in this unit test to make sure our code did what we thought it did? So, this test is already passing, as you saw before. It's already published a PACKT flow. What we want to do is add a new field. What happens when we evolve this API? So, this is a product website. It would be nice to actually display a price for the product, right? So, let's add price into the mix. So, we'll add this new expectation on product here, we'll add it to our product class as well, and then we'll have price available to us to be able to display in the React component. And what I'm going to do is I'm going to check out a new branch, my code, and I'm going to shut down those processes.

Let's create a new field, feet, add price. What I'm going to do here is simulate a CI process as if we're doing a continuous deployment. We're in a branch, so this will be a pull request flow, and what I can do is I can run a fake CI. And what this is going to do, this is going to run the test, it's going to publish the contract to PACKFLOW. We're then going to run a tool called can I deploy and say, is it safe to release this change? And the answer will be no, because this is a new field, the provider has never verified the contract, in fact, the provider hasn't implemented it yet. So we're going to be told it's not safe to release this yet. So back to my terminal, you can see the test passed up there. We've published the packs, and we've said, can we deploy, and we can't deploy. So the build has bailed out with a non-zero exit code. You can see, though, in our code base there's a PACKFLOW that's being created, and you can see it's now got the price property captured in that contract file. We're not going to talk too much about the contract file for now. Just know that it exists, and that's what we use to mediate this whole process. So now that we've added the price property, let's have a look at the PACKFLOW and see what it sees. Cool. So we can see a new contract has been created up top here called FeetAddPrice. It's yet to be verified. So this version of the Consumer can't be deployed anywhere yet because no provider has implemented its features.

9. Deploying Provider and Moving to Production

Short description:

Let's add the price to the Product class and repository. Push the change to the provider and run tests. Deploy the provider to production. Verify the new feature branch and merge it into the mainline. Run the CI process and deploy to production. We are now in production.

So let's go ahead and do that to the provider now. Now I've got some stash changes to avoid Demo Gods. But what we're going to do is we're going to add the price to the property, the Product class, and add it to the repository as well, so there's data.

So there we go. We can see the price has been added. What we can do now is, in theory, we can push this into master. So let's see what it looks like when we commit and push this change. The provider should run the test by pulling the contracts down from PacFlow. It'll share the results of that verification back to PacFlow to say if it passed or succeeded. It will then run the check saying can I deploy this version of the provider to production? The answer should be yes, because it already supports the current version of the consumer. And it's also got the new functionality for this new branch. So we deploy it to production.

Okay, so you can see the provider's run. A whole bunch of assertions have been happening by the PAC framework. It's published the results back to the broker, and it's run the can I deploy check. Can I deploy says yes. This version of the provider satisfies the current production version of the consumer that it would be deploying to, and so it deploys to production. So if I go back to PacFlow and refresh this page, you'll see that this new feature branch has now been verified by the provider. And the version of the provider that satisfies this contract is the same one that satisfies the production contract of the consumer. And so basically it's now safe to merge this change into the mainline and push it to production. So I'm going to pretend like I've merged this. So I'm going to check out back, I check out master. And I'm going to run the CI process as if I've just merged this into master now. And so what should happen is now the consumer will run its tests, it will publish the contract. The contract hasn't changed. So any verifications that have happened before are still valid. It will run a Can I Deploy check. It says, can I move this change to production? And because there is a version of the provider that satisfies these needs, it's safe to do so. And then we deploy it. And then we go, we are now in production.

10. Testing Compatibility and Safe Removal of Fields

Short description:

The prod version of the contract now includes the price property. Removing a field from the provider can be tested locally without pushing any changes. PACT can detect if a consumer is using a property and whether its removal would cause a failure. PACT can also identify properties that are safe to remove if no consumers are using them.

So refresh this, the prod version of the contract now has the price property in it, as you can see here. What happens if we remove a field from the provider? What happens then? Will we catch a bug? Well, for example, if I remove the ID field, sorry, go to provider, and comment that out, I can now run NPMT locally without pushing any change and find out if I'm going to break any of my consumers. There you go. It correctly found that a consumer's using this property, and if we removed it, it would fail. But if I remove the version property, what's going to happen here? Well, as we know, no consumers are currently using this property, so it's actually safe to release it, to remove it. There we go. That's an interesting property of PACT. PACT is able to pick up changes this way at attribute level to find out compatibility.

11. Cost of End-to-End Tests

Short description:

Today we discussed the cost of end-to-end integrated tests, the benefits of contract testing, and how PACT works. We also addressed the challenges of integration testing and the common pain points associated with it. Additionally, we answered a question about using PACT with different consumer and provider technologies, highlighting its ability to support polyglot environments.

Okay. So today we talked about the cost of end-to-end integrated tests. We saw that there's a high cost of maintenance, and that they scale poorly. We saw how contract testing can help with integration testing by combining the approaches of fast and isolated unit tests with contracts to prevent drift. We saw how PACT works and how we can gate releases using Can I Deploy?

So I hope that talk was useful. Really appreciate you coming on. And feel free to contact me on my details there, if you'd like to talk further. Thank you very much.

Hi Matt. Hello everyone. Thanks for having me. Is the poll result surprising to you, or did you expect this? No, that sounds about normal. Look, not everyone's suffering so bad they need to buy bulk coffee and buy themselves a coffee machine like, like I do at home. But, you know, most people find the integration testing, or at least end-to-end integration testing, challenging enough that they have to spend a bit of time on it and generally need something to get through that pain because, you know, the flaky tests, because it takes time to manage because chasing down the issues. You know, there's always an excuse to not want to look into those tests and write those tests and maintain those tests, so it's not entirely surprising, but it's also good to hear that not everyone is in so much pain that they need to, you know, have caffeine hooked into their veins just to better get through the day.

Yeah, but flaky tests are always a good excuse to get another cup of coffee.

There are a few questions. Manuel Zambrano asked, can we use PacFlow with different consumer and provider technologies? I mean, for example, a React frontend and a Python backend serving the API? That's a good question. It's probably the main reason people choose Pact over other alternative sort of contract testing frameworks, so it's worth giving a quick shout out to Spring Cloud Contract. So if you're using Java here, because this is a testing conference for JavaScript, but if you do use Java, Spring Cloud Contract is a decent choice, it's putting the folks out of Spring. They do have support for other languages, but the way it works is you still write groovy scripts, so I guess technically you're probably still writing JVM stuff. But one of the benefits of Pact is that it uses a specification that enables you to work across languages. So really early on in its design, the early maintainers recognized that this would be a challenge, that polyglot architectures were a thing, and that contract testing needed to support polyglot environments from the beginning. So there's actually a specification that governs the way matching happens in a language-independent way. That means that different consumers can be written for different providers and sort of in this environment, but the way the matching happens and the verification happens, so transcends languages. So yes, it's entirely possible to have a JavaScript front-end, an iOS front-end, a Swift front-end, talking to a Java API back-end or Ruby back-end or a .NET back-end. You can mix and match languages kind of as you please.

Amazing. Steve, I'm probably pronouncing that wrong. I'm sorry.

12. Contract Testing vs. Applying JSON Schema

Short description:

Contract testing and applying a JSON schema to every request have some differences. Schema testing focuses on the request bodies and often doesn't cover other HTTP elements like the verb, path, query string, and headers. Ensuring compatibility between the consumer and provider schemas can be challenging. Contract testing with tools like PACT allows versioning and tagging of contracts, ensuring smooth transitions between versions. While you can build your own schema testing tool, contract testing frameworks already provide many necessary features. This strategy can be effective for most use cases.

I was asked how different, slash, related is contract testing with applying a JSON schema to every single request you perform using, for example, Postman? Yeah, that's a great question. We actually get that question a lot. I've got some articles. I can point people to afterwards that sort of talk about the difference between schema testing and contract testing. It goes sort of more skin deep. One of the first things that with schema testing, for starters, is that you're only normally looking at their bodies. You're not looking at the request, the other HTTP things. So the verb, the path, the query string, the headers, as an example, they usually don't get covered in your contract test. They're also obviously a very important part of that contract. But obviously, you can do that. The second thing is even if you're using JSON schema on one side of the contract, what's guaranteeing you that that is exactly what the provider needs on the other side. So you do need to make sure there's a way of pinning the schema you use on both sides of the contract and ensuring those schemas are compatible. So for example, let's say you've got version one of the schema on your consumer side, then the provider updates its schema, you need to make sure that those schemas are now in sync. So essentially, you could do that, but you just need to make sure that the schemas are always the same. And then the second challenge with that is if you think about something like contract testing with PACT, the way it wants to work is actually using your intermediates through something like the PACT broker or PACT flow. And what that does is it lets you version and tag your contracts just like you would with Git code. So let's say you've got code, right? You're about to deploy this into production, version two of your code is in Git, as well as version one, and you need to migrate production from version one to version two. So you need a process to make sure that you can smoothly transition from one to two. And so, again, if you're going to use your own schematic testing tool, you're going to need to come up with your own process for evolving that schema from version one to version two. When you've got multiple components in the system. Again, so it's not that you can't do that. But you're gonna need to build a lot of things that the contract testing frameworks have in them already. What I will say, though, is it's a strategy that people seem to be employing and you know, you can probably get 70% of the way there and that might be good enough for most people's use cases.

QnA

Contract Testing in the Testing Pyramid

Short description:

Contract testing fits in two places in the testing pyramid. On the consumer side, it is closer to a unit test, focusing on a single function. On the provider side, it usually sits in the middle of the pyramid, overlaying the contract test into the service or integration test layer. Contract testing removes the collaboration and communication testing from end-to-end tests, shrinking them. Consumer trust in the provider is addressed by moving relevant tests to the provider's code base. The remaining tests can be placed in production synthetic tests or elsewhere.

Thank you so much for this detailed answer. Richard Fordshow is asking, Is contract testing mainly for unit tests or for end to end tests like with Cypress? I guess the question asks a little bit about where would contract testing stand in the testing pyramid? Yeah, so it's a good question. So probably, I'd say there's two questions. One is kind of where it fits in the pyramid. Can you use tools like end to end testing tools, which I think is, I think as a community, we need to fix that, end to end testing means too many things. We gotta fix that. I think we can't have a situation where Cypress means end to end test, and is really just testing the UI layer. And then we also can refer to end to end testers passing through the entire platform. So I'll put that challenge out there for someone to try and fix. But let's talk about the first one. So where contract testing fits in the pyramid, it sort of fits in two places on the consumer side. Pardon me, it's much closer to a unit test. So basically you should be picking a single function as you saw in the talk, and running a unit test for that function. So that's usually pretty straightforward. On the provider side, you've got a few more options about how far up and down the pyramid you go. Usually it sort of sits in the middle. So we would normally overlay the contract test into the middle of that pyramid, the service test or the integration test layer, where basically you would run up your provider as a bit of a black box, you'd start out any third party dependencies, and you start the service up and then PACT would talk to that service, you know, through HTTP mechanisms, passing through probably multiple layers of your application, but usually they still run very quickly. So they're kind of closer to a unit test, but they're not really a unit test, because they've got to pass through a few layers. So that's usually where they fit in the pyramid. And, you know, in addition to that, what we normally do is we, we supplement, you know, in order to remove end-to-end tests, we supplement those contract tests. Or if you think about it, we got to end-to-end integration tests we want to get rid of. Contract testing removes the tests from the end-to-end tests that have the testing for the collaboration and the communication. Can they talk to each other? All the contract bits of that go away. So we shrink the end-to-end tests a bit. We then look in there and go, well, actually, usually what you find is at end-to-end tests, a lot of them is actually the consumer not trusting the provider. And so the consumer is writing end-to-end tests to check the behavior of the provider. So what you do is you take those tests out and go, nope, they shouldn't belong there. They belong in the provider's code base. Their job is to make sure their code works, not the consumer's job. And then you whittle it down to just a few tests and that's when you can go, well, what's the value in having these few tests left behind? Maybe we can put those in production synthetic tests, or we can put them somewhere else.

Integrating Pact with Cypress

Short description:

Yes, you can integrate Pact with Cypress, but there are challenges due to potential overlap in scenarios. The Pact team is working to optimize this process and plans to release a Cypress plugin in the future.

Or we can remove them altogether because they're not adding value and we can move faster. Now the second part of that question I think is around, could you integrate it with something like Cypress? The short answer is yes, you could integrate it with Cypress. There's some challenges with doing that and we're currently sort of making it less problematic to do that for testing with Cypress. But the short of the answer is, because typically with Cypress tests, you're going to have a lot of overlap. You don't really want to be capturing too many overlapping scenarios in your contract tests because they need to be replayed against the provider. And so that can become a burden on the provider side testing if you add too many. But again, we're doing some work to try and optimize that process to make it easier to use Cypress to write tests and to verify the provider side. And I should mention, we literally at PackFlow, we are writing tests using Cypress now for some of our user interfaces and we're experimenting with generating contracts through that as well. And I suspect in the next couple of months we'll probably release a plugin for Cypress as an example that could be replicated for others.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Network Requests with Cypress
TestJS Summit 2021TestJS Summit 2021
33 min
Network Requests with Cypress
Top Content
Whether you're testing your UI or API, Cypress gives you all the tools needed to work with and manage network requests. This intermediate-level task demonstrates how to use the cy.request and cy.intercept commands to execute, spy on, and stub network requests while testing your application in the browser. Learn how the commands work as well as use cases for each, including best practices for testing and mocking your network requests.
Testing Pyramid Makes Little Sense, What We Can Use Instead
TestJS Summit 2021TestJS Summit 2021
38 min
Testing Pyramid Makes Little Sense, What We Can Use Instead
Top Content
Featured Video
Gleb Bahmutov
Roman Sandler
2 authors
The testing pyramid - the canonical shape of tests that defined what types of tests we need to write to make sure the app works - is ... obsolete. In this presentation, Roman Sandler and Gleb Bahmutov argue what the testing shape works better for today's web applications.
Full-Circle Testing With Cypress
TestJS Summit 2022TestJS Summit 2022
27 min
Full-Circle Testing With Cypress
Top Content
Cypress has taken the world by storm by brining an easy to use tool for end to end testing. It’s capabilities have proven to be be useful for creating stable tests for frontend applications. But end to end testing is just a small part of testing efforts. What about your API? What about your components? Well, in my talk I would like to show you how we can start with end-to-end tests, go deeper with component testing and then move up to testing our API, circ
Test Effective Development
TestJS Summit 2021TestJS Summit 2021
31 min
Test Effective Development
Top Content
Developers want to sleep tight knowing they didn't break production. Companies want to be efficient in order to meet their customer needs faster and to gain competitive advantage sooner. We ALL want to be cost effective... or shall I say... TEST EFFECTIVE!But how do we do that?Are the "unit" and "integration" terminology serves us right?Or is it time for a change? When should we use either strategy to maximize our "test effectiveness"?In this talk I'll show you a brand new way to think about cost effective testing with new strategies and new testing terms!It’s time to go DEEPER!
Playwright Test Runner
TestJS Summit 2021TestJS Summit 2021
25 min
Playwright Test Runner
Top Content
Everyone Can Easily Write Tests
TestJS Summit 2023TestJS Summit 2023
21 min
Everyone Can Easily Write Tests
Let’s take a look at how Playwright can help you get your end to end tests written with tools like Codegen that generate tests on user interaction. Let’s explore UI mode for a better developer experience and then go over some tips to make sure you don’t have flakey tests. Then let’s talk about how to get your tests up and running on CI, debugging on CI and scaling using shards.

Workshops on related topic

Designing Effective Tests With React Testing Library
React Summit 2023React Summit 2023
151 min
Designing Effective Tests With React Testing Library
Top Content
Featured Workshop
Josh Justice
Josh Justice
React Testing Library is a great framework for React component tests because there are a lot of questions it answers for you, so you don’t need to worry about those questions. But that doesn’t mean testing is easy. There are still a lot of questions you have to figure out for yourself: How many component tests should you write vs end-to-end tests or lower-level unit tests? How can you test a certain line of code that is tricky to test? And what in the world are you supposed to do about that persistent act() warning?
In this three-hour workshop we’ll introduce React Testing Library along with a mental model for how to think about designing your component tests. This mental model will help you see how to test each bit of logic, whether or not to mock dependencies, and will help improve the design of your components. You’ll walk away with the tools, techniques, and principles you need to implement low-cost, high-value component tests.
Table of contents- The different kinds of React application tests, and where component tests fit in- A mental model for thinking about the inputs and outputs of the components you test- Options for selecting DOM elements to verify and interact with them- The value of mocks and why they shouldn’t be avoided- The challenges with asynchrony in RTL tests and how to handle them
Prerequisites- Familiarity with building applications with React- Basic experience writing automated tests with Jest or another unit testing framework- You do not need any experience with React Testing Library- Machine setup: Node LTS, Yarn
How to Start With Cypress
TestJS Summit 2022TestJS Summit 2022
146 min
How to Start With Cypress
Featured WorkshopFree
Filip Hric
Filip Hric
The web has evolved. Finally, testing has also. Cypress is a modern testing tool that answers the testing needs of modern web applications. It has been gaining a lot of traction in the last couple of years, gaining worldwide popularity. If you have been waiting to learn Cypress, wait no more! Filip Hric will guide you through the first steps on how to start using Cypress and set up a project on your own. The good news is, learning Cypress is incredibly easy. You'll write your first test in no time, and then you'll discover how to write a full end-to-end test for a modern web application. You'll learn the core concepts like retry-ability. Discover how to work and interact with your application and learn how to combine API and UI tests. Throughout this whole workshop, we will write code and do practical exercises. You will leave with a hands-on experience that you can translate to your own project.
Detox 101: How to write stable end-to-end tests for your React Native application
React Summit 2022React Summit 2022
117 min
Detox 101: How to write stable end-to-end tests for your React Native application
Top Content
WorkshopFree
Yevheniia Hlovatska
Yevheniia Hlovatska
Compared to unit testing, end-to-end testing aims to interact with your application just like a real user. And as we all know it can be pretty challenging. Especially when we talk about Mobile applications.
Tests rely on many conditions and are considered to be slow and flaky. On the other hand - end-to-end tests can give the greatest confidence that your app is working. And if done right - can become an amazing tool for boosting developer velocity.
Detox is a gray-box end-to-end testing framework for mobile apps. Developed by Wix to solve the problem of slowness and flakiness and used by React Native itself as its E2E testing tool.
Join me on this workshop to learn how to make your mobile end-to-end tests with Detox rock.
Prerequisites- iOS/Android: MacOS Catalina or newer- Android only: Linux- Install before the workshop
API Testing with Postman Workshop
TestJS Summit 2023TestJS Summit 2023
48 min
API Testing with Postman Workshop
Top Content
WorkshopFree
Pooja Mistry
Pooja Mistry
In the ever-evolving landscape of software development, ensuring the reliability and functionality of APIs has become paramount. "API Testing with Postman" is a comprehensive workshop designed to equip participants with the knowledge and skills needed to excel in API testing using Postman, a powerful tool widely adopted by professionals in the field. This workshop delves into the fundamentals of API testing, progresses to advanced testing techniques, and explores automation, performance testing, and multi-protocol support, providing attendees with a holistic understanding of API testing with Postman.
1. Welcome to Postman- Explaining the Postman User Interface (UI)2. Workspace and Collections Collaboration- Understanding Workspaces and their role in collaboration- Exploring the concept of Collections for organizing and executing API requests3. Introduction to API Testing- Covering the basics of API testing and its significance4. Variable Management- Managing environment, global, and collection variables- Utilizing scripting snippets for dynamic data5. Building Testing Workflows- Creating effective testing workflows for comprehensive testing- Utilizing the Collection Runner for test execution- Introduction to Postbot for automated testing6. Advanced Testing- Contract Testing for ensuring API contracts- Using Mock Servers for effective testing- Maximizing productivity with Collection/Workspace templates- Integration Testing and Regression Testing strategies7. Automation with Postman- Leveraging the Postman CLI for automation- Scheduled Runs for regular testing- Integrating Postman into CI/CD pipelines8. Performance Testing- Demonstrating performance testing capabilities (showing the desktop client)- Synchronizing tests with VS Code for streamlined development9. Exploring Advanced Features - Working with Multiple Protocols: GraphQL, gRPC, and more
Join us for this workshop to unlock the full potential of Postman for API testing, streamline your testing processes, and enhance the quality and reliability of your software. Whether you're a beginner or an experienced tester, this workshop will equip you with the skills needed to excel in API testing with Postman.
Testing Web Applications Using Cypress
TestJS Summit - January, 2021TestJS Summit - January, 2021
173 min
Testing Web Applications Using Cypress
WorkshopFree
Gleb Bahmutov
Gleb Bahmutov
This workshop will teach you the basics of writing useful end-to-end tests using Cypress Test Runner.
We will cover writing tests, covering every application feature, structuring tests, intercepting network requests, and setting up the backend data.
Anyone who knows JavaScript programming language and has NPM installed would be able to follow along.
Best Practices for Writing and Debugging Cypress Tests
TestJS Summit 2023TestJS Summit 2023
148 min
Best Practices for Writing and Debugging Cypress Tests
Workshop
Filip Hric
Filip Hric
You probably know the story. You’ve created a couple of tests, and since you are using Cypress, you’ve done this pretty quickly. Seems like nothing is stopping you, but then – failed test. It wasn’t the app, wasn’t an error, the test was… flaky? Well yes. Test design is important no matter what tool you will use, Cypress included. The good news is that Cypress has a couple of tools behind its belt that can help you out. Join me on my workshop, where I’ll guide you away from the valley of anti-patterns into the fields of evergreen, stable tests. We’ll talk about common mistakes when writing your test as well as debug and unveil underlying problems. All with the goal of avoiding flakiness, and designing stable test.