Photo by Omar Flores on Unsplash

Serverless Event Gateway Pattern

Example of enabling legacy systems and 3rd parties to publish and consume events to/from your Amazon EventBridge central event bus; with supporting code repository written in TypeScript and the AWS CDK.

Contents

✔️ Introduction.
✔️ What are we building?
✔️ Talking through the code.
✔️ Additional considerations.
✔️ Wrapping up.

The code for the article can be found below:

GitHub - leegilmorecode/serverless-event-gateway-pattern: Example of enabling legacy systems and…

👇 Before we go any further — please connect with me on LinkedIn for future blog posts and Serverless news https://www.linkedin.com/in/lee-james-gilmore/

Introduction

There are many times when developing our serverless solutions when we need to interact with existing legacy systems or 3rd party services within our wider domain; and they are unable to publish and consume events from your central Amazon EventBridge bus directly due to technology or framework limitations.

💡 This could be simply that they can't utilise the aws-sdk or iam in 
code, but can perform REST API requests.

This quick article discusses using the ‘Event Gateway Pattern’ alongside Amazon EventBridge and Amazon EventBridge API Destinations to allow this flow between systems both ways i.e. consuming and publishing based on domain events.

💡 This is a basic example to show the pattern and is not production ready

What is the ‘Event Gateway Pattern’?

The ‘Event Gateway Pattern’ is essentially putting an Amazon API Gateway REST API in front of your central Amazon EventBridge bus, allowing any legacy or 3rd party publisher to publish their own events via REST (API) with a direct integration with Amazon EventBridge.

Example of our legacy system using REST to publish events to our shared EDA account

We can then utilise Amazon EventBridge API destinations to also allow the flow both ways i.e. based on particular domain events and rules in our internal cloud services we can then invoke REST endpoints on the legacy/3rd party systems, passing them the full contents of the event (event-carried state transfer).

This ensures that these 3rd party systems can always publish and consume from/to our dedicated central event bus, as almost all modern languages and frameworks allow for interactions using REST.

In the next section let’s see what we are going to build out.

What are we building? 👟

We are going to build a fictitious sneaker online store called ‘Sneakers — Lee Gilmore Sneaker Store’ which has a direct integration with a 3rd party manufacturer who supply them:

💡 Note: In our example, the 3rd party external service is simply an API 
Gateway REST API, backed with Lambda and DynamoDB.

Our fictitious online sneaker store called ‘Sneakers’.

We can see below what we are building at a high level (HLD):

Example two way integrations between our internal and external systems via our ESB.

Let’s talk through the steps above:

1. Customers place new sneaker orders through the API Gateway API (internal system API).
2. The sneaker order is processed via AWS Lambda and stored in DynamoDB.
3. An ‘OrderPlaced’ event is published to the central Amazon EventBridge event bus.
4. Using API Destinations we notify the Legacy external system of the new order through their own REST API. At this point they populate the order in their system to (persisted in DynamoDB).
5. At a later point the Legacy system publishes an ‘OrderCancelled’ event to the central event bus via the central EDA account REST API. This is because the sneakers have now been discontinued, and they won’t be supplying anymore to the website, and therefore cancelling all orders with this product ID.
6. The ‘OrderCancelled’ event is routed to the Order Service via a target rule and the internal database is updated accordingly (status set to cancelled), as this product has been discontinued, and the order can not be fulfilled. At this point the customer would get an email stating that unfortunately the sneakers have been discontinued.

💡 As this is a basic example we are not utilising resilience patterns
and services such as dead letter queues, archive and replay etc.
We will discuss this in the next section however.

Talking through the code

Now let’s talk through the design and code in detail.

Amazon API Gateway direct to Amazon EventBridge

How do we setup the integration with Amazon API Gateway and Amazon EventBridge as a direct integration using the Amazon CDK, allowing 3rd parties to publish their domain events to our shared ESB? Let’s see the specific code below:

This setup ensures that when an external consumer (service) posts a new event utilising the following REST endpoint:

https://rest-api.execute-api.your-region.amazonaws.com/prod/events/

We pass through the following shaped JSON object on the POST request to the ‘events’ resource:

const externalDomainEvent = {
  metadata: {
    source: 'com.external.orders',
    eventSchemaVersion: '1',
    eventName: 'OrderCancelled',
  },
  data: {
    ...order,
  },
};

As shown below in the external ‘delete-order.ts’ handler we use Axios to POST the request to the Shared ESB API:

This means that the following code on the Shared ESB Amazon API Gateway Integration Request parses the body of the REST POST event (request) and passes this through to Amazon EventBridge directly with a PutEvent:

const eventBridgeOptions: apigw.IntegrationOptions = {
  credentialsRole: apigwRole,
  requestParameters: {
    'integration.request.header.X-Amz-Target': "'AWSEvents.PutEvents'",
    'integration.request.header.Content-Type':
      "'application/x-amz-json-1.1'",
  },
  requestTemplates: {
    'application/json': `{"Entries": [{"Source": "$util.escapeJavaScript($input.path('$.metadata.source'))", "Detail":"$util.escapeJavaScript($input.json('$'))", "DetailType": "$util.escapeJavaScript($input.path('$.metadata.eventName'))", "EventBusName": "${sharedEventBus.eventBusName}"}]}`,
  },
  integrationResponses: [
    {
      statusCode: '200',
      responseTemplates: {
        'application/json': 'Created',
      },
    },
  ],
};

This means our third party external system has been able to publish their events directly to our shared event bus via our REST API, allowing our other domain services to consume.

The following image shows this in action:

Example of our external service publishing events to our shared event bus via an Amazon API Gateway REST API, in this example when a product is cancelled.

💡 Note: We dont have any authorisation or authentication on this API for
the demo, but lets cover this in the next section.

Amazon EventBridge to external Amazon API Gateway API

OK, so how do we reverse this, and ensure that on domain events being published on the shared event bus we directly notify the external systems?

Let’s look at the code below using Amazon EventBridge API Destinations:

As you can see from the code above we:

1. Create a new connection. This is where we can add an API Key, Basic Auth OIDC etc to setup the authentication to the external system. (In our example we simply use an API Key which should never be used for auth on its own).
2. Create the API Destination. This is where we setup the URL we want to invoke, the method type (e.g. POST), rate limiting etc.
3. Create the target rule. Finally we create the rule target that states when we get an event on the shared event bus with the detail type ‘OrderCreated’ from the source ‘com.internal.orders’ we invoke the API in point 2.

The following image shows this in action:

Example diagram showing our direct integration with an external API using Amazon EventBridge API Destinations, in this example when a new order is created in the internal system we publish this to the external service.

Additional Considerations

OK, now that we covered the very basic code example above to showcase the ‘Event Gateway Pattern’, what other additional considerations might we have? Let’s cover some of them below.

✔️ Storage First Pattern

The following part of the design is widely known as a ‘Storage First’ pattern:

The blue circle shows a ‘storage first’ pattern, where we persist the message/event/payload as soon as we can when performing asynchronous processing; ensuring that we don’t lose it.

The ‘storage-first approach’ is to increase resilience and durability in serverless solutions through persisting the requests as soon as viably possible through direct service integrations before performing any business logic, and then subsequently processing them including retry mechanisms for failures. This pattern is very different to the typical approach of processing requests as soon as possible through compute services like AWS Lambda.

This is discussed at length in the following article if you would like to understand more:

Storage-First Serverless Solutions 🚀

✔️ Authorisation and Authentication

Authorisation and authentication on your APIs are a massive consideration when communicating between systems.

The blue circles indicate where we would want to look at key areas like authentication, authorisation, usage plans, api keys, IP whitelisting etc

I won’t go into detail in this article around authorisation and authentication, but the following article covers this in detail:

Serverless Authorisation 🚀

Another consideration is the use of Usage Plans and API Keys, as well as IP whitelisting using AWS WAF. Again, I won’t cover this in detail in this article as the following article covers this in detail (with code base example):

Serverless B2B Authentication

✔️ Event Validation

Another key consideration which we have not covered in this article is event validation, be that when publishing events, or when consuming them.

This is covered in detail in the following article if you would like to know more:

Amazon EventBridge Schema Validation

In our example we would also likely want to ensure that that there is a lookup between the 3rd parties API Key on the usage plan, and the source and detail type on the event. This would ensure that we could reject any REST calls for events we are not expecting from specific 3rd party domains sources. In our example we would most likely add this to a Custom Authorizer with a lookup on parameter store.

Summary

I hope you found that useful as a basic concept of allowing 3rd party and legacy systems to publish and consume events from your cloud domain services. If you did find it useful, please feel free to share with others on LinkedIn or Twitter ✔️

Wrapping up 👋

Please go and subscribe on my YouTube channel for similar content!

I would love to connect with you also on any of the following:

https://www.linkedin.com/in/lee-james-gilmore/
https://twitter.com/LeeJamesGilmore

If you enjoyed the posts please follow my profile Lee James Gilmore for further posts/series, and don’t forget to connect and say Hi 👋

Please also use the ‘clap’ feature at the bottom of the post if you enjoyed it! (You can clap more than once!!)

About me

“Hi, I’m Lee, an AWS Community Builder, Blogger, AWS certified cloud architect and Global Enterprise Serverless Architect (GESA) based in the UK; currently working for City Electrical Factors (UK) & City Electric Supply (US), having worked primarily in full-stack JavaScript on AWS for the past 6 years.

I consider myself a serverless advocate with a love of all things AWS, innovation, software architecture and technology.”

*** The information provided are my own personal views and I accept no responsibility on the use of the information. ***

You may also be interested in the following:

Serverless Content 🚀