Enterprise Serverless 🚀 Hints & Tips
This section in the series on Enterprise Serverless specifically covers some hints and tips based off working on large scale serverless projects. You can read the ideas behind the series by clicking on the ‘Series Introduction’ link below.
The series is split over the following sections:
- Series Introduction 🚀
- Tooling 🚀
- Architecture 🚀
- Databases 🚀
- Hints & Tips 🚀
- AWS Limits & Limitations 🚀
- Security 🚀
- Useful Resources 🚀
Hints & Tips
YAML vs JSON
When working with the Serverless Framework you have the option of using JSON or YAML when defining your resources, both of which are fairly limited when you need to perform some ultra bespoke logic (on large projects this happens a lot in my experience!).
One useful way of getting around this is having a JavaScript function which essentially exports the serverless JSON on the fly. How is this useful? Well, say you wanted to only deploy certain resources depending on environment variables for locale and stage, you can now use a common JavaScript function that wraps these resources, and spreads the values into the JSON when required dynamically. I have found this approach extremely useful in past projects where we have used this a large number of times for varying bespoke scenarios.
Tagging Resources
One of the key things to consider and monitor with distributed event driven serverless architectures is cost — and how to link that cost to particular workflows or larger domain services through the system (from API through to datastores such as databases or S3 for example). Tagging your AWS resources is a best practice and allows you too quickly monitor and determine where associated services have incurred overall costs.
Split out CloudFormation
The serverless framework is fantastic when managing serverless resources, but on any large scale serverless project you will no doubt quickly find yourself using the resources section in CloudFormation to manage other resources such as AWS Parameter Store, S3, IAM etc
One key consideration is to split each resource type, such as IAM, into its own file per serverless project, which can then be imported into the main JSON/YAML. If not you will soon find that your main serverless file grows massively and becomes hard to manage, and you will constantly have merge conflicts when working with multiple scrum teams.
Correlation IDs
One of the most difficult things to do in a distributed system is to track through logs how an API call, for example hitting API Gateway or AppSync, has made its way via various SQS queues, Step Functions and lambdas, and how far it has got before potentially failing or being successful. Times this by 500–200m+ API calls per month and its an issue tracking individual calls..
A key approach to all large scale serverless projects I have worked on is to generate a correlation ID at the earliest possible opportunity, and ensuring that this correlation ID is passed throughout the entire flow (including SQS messages for example). This allows collating the log entries per correlation ID using search as if it was one synchronous call, rather than trying to search through individual logs over many services for one API call which would be a nightmare!
💡 I typically set the correlation ID to the request ID when using API Gateway, or by generating a UUID in the before step of a pipeline in AppSync using a shared VTL.
Share expensive resources
Based on the first example of exporting JSON dynamically above; one key approach I have used in the past for sharing expensive resources across many ephemeral environments and QA is to export the resources dynamically based on stage.
Examples of these resources have been AWS Elasticsearch, VPCs and Nat Gateways, and AWS DocumentDB for example where it made no sense for each short lived environment to have its own dedicated resources, as well as the slow down in deploying them individually each time.
💡 The key with this approach is to ensure that you build the DocumentDB database names or Elasticsearch indexes dynamically per stage so this works seamlessly on the fly.
Next section: AWS Limits & Limitations 🚀
Previous section: Databases 🚀
Wrapping up
Lets connect on any of the following:
https://www.linkedin.com/in/lee-james-gilmore/
https://twitter.com/LeeJamesGilmore
If you found the articles inspiring or useful please feel free to support me with a virtual coffee https://www.buymeacoffee.com/leegilmore and either way lets connect and chat! ☕️
If you enjoyed the posts please follow my profile Lee James Gilmore for further posts/series, and don’t forget to connect and say Hi 👋
About me
“Hi, I’m Lee, an AWS certified technical architect and polyglot software engineer based in the UK, working as a Technical Cloud Architect and Serverless Lead, having worked primarily in full-stack JavaScript on AWS for the past 5 years.
I consider myself a serverless evangelist with a love of all things AWS, innovation, software architecture and technology.”
** The information provided are my own personal views and I accept no responsibility on the use of the information.
