This folder contains services that connect via REST. Currently, this is reduced to showcasing resilience patterns.
A good background read is this InfoWorld article 3 common pitfalls of microservices integration—and how to avoid them
This sample REST (micro-)service effects payments in response to a PUT call. It requires an upstream REST service that charges credit cards.
This simple call-chain is perfect for demonstrating important resilience patterns.
The following technologies are used:
- Java 17
- Spring Boot 3.1.x
- Resilience4j 2.1
- Camunda 8.x
First you have to startup the "Stripe Fake Server", as this handles the credit card payments:
mvn -f java/stripe-fake/ exec:java
Now you can run the Payment Service itself:
mvn -f java/payment/ exec:java
Now the different versions of the payment service are available:
You now can issue a PUT with an empty body:
curl \
-H "Content-Type: application/json" \
-X PUT \
-d '{}' \
http://localhost:8100/api/payment/v1
All examples from version 3 and above use Camunda as a workflow engine, so you need a Camunda instance to run those. The easiest way to play around is to get a test account in the Camunda SaaS offering, create a cluster and API credentials and add them to the application.properties.
See Fail fast is not enough: https://blog.bernd-ruecker.com/fail-fast-is-not-enough-84645d6864d3
Let's assume a scenario where the upstream credit card service still responds, but its very slow. With no resilience pattern in place, this is the worst thing that can happen - as now the payment service will call the credit card service and block until it gets a response. As this take a long time, all threads from the payment service are held hostage, and the payment service will eventually time out for its clients. Tiny failures somewhere in your system might blow up your whole system:
A simple mitigation is to apply a fail fast pattern like circuit breaker. In this example I use Resilience4J. If a service responds too slowly, the circuit breaker interrupts and the payment service gets a failure right away. This way you make sure the overall system is still responding, even if functionality degrades (meaning: we cannot charge credit cards).
There is also an extended version adding some stateless retrying to the mix also using Resilience4J. This can be usefuly to mitigate problems with a flaky network:
Failing fast is good, but it is not enough. Frequently, a retry after the credit card service has been fixed resolves the situation (if the service was in a hard failure mode) - or a retry may discover that the earlier attempt succeeded, but took an abnormal amount of time (if the service is in a degraded performance mode). This retry needs to be stateful to not only retry right away but again in a couple of minutes, hours or even days. Keeping this stateful retry local to the payment service reduces overall architectural complexity.
In the example, I use the Camunda workflow engine to do the stateful retry reliably.
The processing just got asynchronous, which is often not wanted. In this scenario you could very well return a synchronous response whenever the credit card service is available, but switch to asynchronicity when it is not.
HTTP supports this via return codes: 200 OK means "all OK", 202 ACCEPTED means "I'll call you back later".
An alternative to synchronously calling an upstream service is to communicate asynchronously. The default would be messaging.
This example shows a successful approach taken by many customers: using the workflow engine as work distribution, behaving like a queue. This leverages the External Tasks pattern.
- Java
- Workflow model: payment5.bpmn (hint: use the free Camunda Modeler to show this model graphically)
- Worker Node.js: index.js
- Worker in Java (alternative): CustomerCreditWorker.java
The last part of the example adds compensation to the game. In distributed systems, ACID transactions are not applicable (or at least do not scale well). Using compensation is the alternative - meaning that you reliably undo already executed work if something later on fails.
See payment6.bpmn / Java and PaymentV6.bpmn / C# for the workflow