Serverless Threat Modelling Part 3 🚀
Advanced Serverless Threat Modelling with guest writer Marek Šottl, discussing attacks such as Lambda Injection and Black Infra as Code.
Introduction
When discussing Serverless Threat Modelling, it is key to also get the objective view of Serverless security experts. For that very reason, I am delighted that Marek Šottl is covering more of the advanced attack vectors in this article!
Marek is part of the AWS Community Builders programme, and AWS and DevSecOps hands-on architect. He runs the YouTube channel Hackitect’s Playground. He loves learning and exploring new technologies (occasionally breaking them to cyberpieces).
To read part one and two please find the links below:
Other Threat Modelling Techniques
In previous parts of this series, Lee James Gilmore covered very well the STRIDE model (Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Escalation of privileges).
The same techniques can also be mapped to the MITRE ATT&CK framework. Common elements of generic threat models are very limited (boundary, storage, process, data flow, actor).
I usually recommend using the official AWS icons and notation to map the current infrastructure and serverless services.
AWS official notation is the best way to communicate your design, deployment, and topology for threat modelling sessions. AWS visualisation guidelines are more than detailed enough to provide you a comprehensive way to build understandable diagrams for everyone on the threat modelling session (developers, security, architects, product).
Don’t forget that STRIDE is not the only methodology for Threat modelling, as there are others like Attack tree (tree-based decisions on attack paths).
Gamification! 👾
If your architecture consists of AWS Step Functions, Lambdas, ECS (Fargate serverless deployment) or EKS; consider using the Serverless OWASP TOP 10 mapping.
For this purpose, there are two great games (yes, we talk about gamification!). OWASP Cornucopia and Elevation of privilege games. You still need your architecture diagram, and to prepare some award for the best virtual attacker and threats!
Players are trying to trump each other on attack types from OWASP TOP 10 or STRIDE.
OWASP Cornucopia has better coverage according to the recommended cheat sheet and coding techniques. It references application security verification standards and the OWASP testing guide. So, after the threat modelling session you have some ideas where to go and write better optimised Lambda code or how to verify Lambda inputs.
Recommended parts of the diagram are storage, serverless, compute, security, identity and compliance, network and content delivery, organisation management and containers. Reflecting current trends in AWS, also GameTech, Blockchain services or IoT are suitable too.
Mitre ATT&CK
The MITRE framework (MITRE ATT&CK Techniques, tactics, and procedures) is perfect not only for incident response, but for feeding your ideas for threat modelling!
In threat modelling we focus on tactics and techniques. Take your architecture diagram and with the Cloud, IaaS or Containers, use the MITRE ATT&CK matrices as guides so you can start your session.
Filtered attack navigator for Cloud, IaaS and Containers from MITRE are shown below:
I recommend starting with Trust Relationships (VPC peering, Cross account roles), cloud storage discovery, dumping, and account manipulation, in order to steal application AWS Secret key and Access key IDs.
These can be obtained from a misconfigured metadata service running on http://169.254.169.254. With lack of validation in your application or Lambda function input events, a remote attacker can use RCE (Remote code execution), or Server-side injection, to gather AWS credentials used by the AWS service role.
“These can be obtained from a misconfigured metadata service running on http://169.254.169.254. With lack of validation in your application or Lambda function input events, a remote attacker can use RCE (Remote code execution), or Server-side injection, to gather AWS credentials used by the AWS service role.”
OWASP Threat Dragon 🐉
You are still not happy with the tips? Looking for some visual tooling?
OK, if you are OK with Azure (the Microsoft approach); then you can try the MS Threat modelling tool. You can also consider the generic open source tool instead called OWASP Threat Dragon.
📘 Threats however must be defined by you.
📘 If you want nicer diagrams and more fun, use AWS official icons and your favourite modelling tool!
Selected threats deep dive 👿
The following sections below discuss selected threats in more detail.
CloudFormation Resource Injection 💉
Many attack scenarios anticipate AWS credentials leakage. It’s a very realistic scenario because many repos have credentials in commit history.
“Many attack scenarios anticipate AWS credentials leakage. It’s a very realistic scenario because many repos have credentials in commit history.”
Applications can be also vulnerable to ECR (Elastic Container Registry) with environment variable dumping. Many developers across various companies use locally stored credentials, and the only thing that separates you from the credential theft is a well targeted phishing campaign.
“only thing that separates you from the credential theft is a well targeted phishing campaign”
The remote attacker can use the notification after the template upload to an S3 bucket. They can download the template, modify it after notification, and upload to controlled S3 bucket.
Then the CloudFormation engine will deploy the malicious updated CloudFormation Template. This can lead to creating Black infrastructure, Denial of wallet attacks or creating new permissions for privilege escalation by adding new inline policies in AWS IAM which are hard to find (and also, not recommended to use).
Also never name your bucket cf-*, because it’s easy to enumerate such a bucket with tools like Pacu.
“Also never name your bucket cf-*, because it’s easy to enumerate such a bucket with tools like Pacu.”
📘 Same story goes for a CloudFormation resource injection attack.
It works with the reality that CloudFormation uses the TemplateURL parameter in the ‘CreateStack’ API call with a one second delay. This attack also has the ability of the attacker to use any role/policy /user to ‘s3:PutObject’, ‘s3:PutBucketNotification’, and ‘s3:GetObject API’ calls. This needs to be done to put the new malicious template into the S3 bucket!
Detailed information from Rhinolabs about the Cloud formation injection attack : https://rhinosecuritylabs.com/aws/cloud-malware-cloudformation-injection/
Deep dive into Denial of Wallet attacks (DoW)💰
DoW seeks to cause the victim financial loss due to lack of billing alarms and FinOps practices. Learn how to setup the billing alarm using the AWS documentation below:
“DoW seeks to cause the victim financial loss due to lack of billing alarms and FinOps practices”
For better financial planning and spend control you can use AWS Budgets. Also, when you are using the billing console in a single account or when multiple users have access to your billing account, restrict the access only to necessary users and services.
How to restrict the users access in the billing console:
Setup above allows full access to the AWS services, but denies IAM users access to the ‘Billing and Cost Management’ console.
Access to billing data also helps remote attackers to enumerate the services that they do not have access to, but from spend reporting they can see all the services used by your account!
Denial of Wallet attacks can be performed via offensive Terraform libraries (black infrastructure), or Lambda looping attacks. With bad input validation you can construct the payload that will cycle lambda for a long time before the timeout. This will increase the overall spend in your AWS account for Lambda execution time.
“Denial of wallet can be performed via offensive Terraform libraries (black infrastructure), or Lambda looping attacks. With bad input validation you can construct the payload that will cycle lambda for a long time before the timeout.”
One example is an enumeration and time-consuming operation such as password validation. Based on how long you evaluate the input tokens, remote attackers can estimate if the token is correct or not.
Example of vulnerable code
This piece of code (below) I have seen in several services out there in the Serverless wild, mainly in Lambdas processing authentication tokens
The token, password or secret processing must be designed in a way that the evaluation of how long the string is always returns the same response time.
There are two aspects here; time based enumeration, and length of the run per token evaluation.
This is covered in the article below:
How to avoid this issue? Firstly, you must avoid Lambda anti-patterns :
Great developers must optimise the code to run as designed in the minimum execution time, and that takes time to master.
Are you looking for some guidance for execution time optimisation?
Semantic reasoning in AWS Code Guru can help a lot. Code Guru offers automated recommendations in security, quality, and profiling of your code.
“The cloud and development team should cover Threat in early stages of development and cover the best practices for run time of the code”
You can find there OWASP TOP 10, AWS best practices, LDAP injection, and also detection and help for Log4j-type vulnerabilities.
The cloud and development team should cover threat in early stages of development and cover the best practices for run time of the code. For example, the sample python code:
client = boto3.client(‘s3’)
When used in an initialisation block this can last milliseconds compared to being used in the definition of the function itself. The difference for Lambda cold start will be not be so visible, but for the warm start you can observe significant savings on code execution time.
Dive in Lambda injection 💉
Typical SQL injection or Cross site scripting injection types is the same for Lambdas. The RCE (Remote code execution) will differ a lot.
RCE targets the Lambda credentials and temporary data stored in /tmp. The problem of poorly validated inputs is the same as in web applications.
“RCE targets the Lambda credentials and temporary data stored in /tmp. The problem of poorly validated inputs is the same as in web applications”
Outdated or Legacy libraries used in dependencies provide another attack vector if you do not run continuous Lambda Security composition analysis with tools like Snyk, Whitesource etc.
Before a lambda is deployed you should use AWS Code Build with dependency scanning and testing for known security issues.
Notice that the threats in the diagram can be ideally numbered, and when you created the list of threats don’t forget to describe them well, and number them as to not get lost and also keep track. Threats in our case:
- SQL injection
- Secrets leakage.
Lack of Serverless protection mechanisms is another threat worth adding to the list in your DevOps pipelines. You should also focus on AWS CodeBuild, AWS pipeline and AWS CodeDeploy. Missing pieces of CI/CD are very dangerous gaps.
With leaked credentials with Lambda environment variables when misconfigured, even the whole code can be downloaded with all of the layers.
“With leaked credentials with Lambda environment variables when misconfigured, even the whole code can be downloaded with all of the layers”
With access to the code, attackers can easily discover flaws in your code and use them to inject the parts of the apps where they do not have access like the RDS database for the board game shop above.
SQL injection often allows you to get full admin access to the underlying database. RDS usually has access to AWS Secrets manager. Secrets Manager is the perfect place to look after your elevated role to AWS RDS admin.
“SQL injection often allows you to get full admin access to the underlying database. RDS usually has access to AWS Secrets manager. Secrets Manager is the perfect place to look after your elevated role to AWS RDS admin.”
Why is all of this is possible? Unvalidated inputs! During the threat modelling session, you should also ask how the user input is validated, and where is the Web Application Firewall?
Black Infra as code 🏴
‘Black infra’ as I call it is malicious infrastructure created by remote attackers with your infiltrated pipeline or leaked AWS credentials. (Imagine offensive CloudFormation or Terraform).
You don’ need to imagine this. The whole thing exists out there in the waters of GitHub.
📘 Offensive Terraform is a set of malicious Terraform modules.
How often do you review your Terraform dependencies and how often do you scan your terraform with SAST tools for known misconfigurations?
Black infra modules offer various options like copy of public snapshots, malicious backdoor security groups, cross account persistence or creating the Kali Linux instance for hacking in your infra.
Now, another question for threat modelling. Do you have infrastructure bill of materials, to know what you produce with CDK, CloudFormation or Terraform? I want to mention one of the great tools for BOM generation for k8s, sigs-bom.
When we are talking a lot about serverless, what about Black infra with current role. Offensive Terraform can create Lambda functions with the current IAM role. The module invokes it automatically to exfiltrate AWS temporary credential from environment variables and outputs it back to attacker. Imagine that you can do the same for example for AWS Code Build or CodeDeploy. So many new Threats to identify!
IAM Chaining Threads ⛓️
IAM threats are for a separate article. I will try to be fast and straight forward. IAM review is a complex task to identify exfiltration, dumping, privilege escalation or resource exposure capabilities as you need serious tooling.
During threat modelling try to identify at least the most important roles in your accounts. Don’t forget to use tools like IAM access analyser or Cloudsplaining.
Try to think what will happen after any developer assumes a specific role. Are they able to use any service role to create a new service and then change his own IAM permissions? For example, what if the developer can assume role to the role that allows CreateAccessKey for another user?
IAM analyser helps you to identify threats automatically in your IAM roles. It uses zelkova, which is an AWS algorithm based on semantic reasoning.
Semantic reasoner is an engine that works with axioms (facts). This engine is able to derive new facts from existing data based on engine rules.
ECR Backdooring 🔑
ECR, EKS and general container threats are very big and special category!
Don’t forget to focus on it during your sessions. ECR IAM permissions are something that you should carefully watch. If you are not using GitOps with AWS CodeCommit, GitHub or GitLab, you can run in serious trouble.
Containers should only be pushed from your repository CI/CD; not by developers. Allowing developers ecr:PutImage or ecr:InitiateLayerUpload can lead to the ability of remote attackers being able to push a malicious image with latest tag, and you will not even notice that there is a new layer added to the same image as you have in your repository.
“Allowing developers
ecr:PutImageorecr:InitiateLayerUploadcan lead to the ability of remote attackers being able to push a malicious image with latest tag, and you will not even notice that there is a new layer added to the same image as you have in your repository.”
Threat Modelling as Code 🧑🤝🧑
It’s undeniable that threat modelling as code is the present and future of speeding up threat modelling exercises.
If you are mature enough to generate the threat models from code or codify them, don’t hesitate to push your models and code into a dedicated repo. Versioning will also help you to observe the evolution of your threat models.
If you are looking for dedicated tooling in this area, I highly recommend looking at threatspec or pytm (A Pythonic framework for threat modelling).
Let’s look deeper at threatspec
To generate models from your code you need to have high maturity coding standards as it requires dedicated security comments and annotations.
Your model data sits in the code and then the framework dynamically generates threat models from code. You need to use annotations like @control, @threat or @transfers, and many others reflecting the part of the targeted threat model.
Threat modelling tooling generates meta-data in JSON or YAML. This means you can generate a model for a single monorepo, or multiple repos per configuration.
Next step is using CI/CD to generate the reports for you. This pipeline generated artefacts can be used by AppSec teams in testing process or code reviews by developers.
When you decide to use tools like pytm you have limited amount of Threat modelling elements (Server, Datastore, Dataflow, Actor etc.) but if it’s enough for you, feel free to go. If you want to get better coverage for AWS element modelling you can use diagrams as code with diagrams.py.
Example of a codified diagram is below:
Thank you for reading the article to this place (+10 experience points) and hope that you enjoyed the advanced threat modelling tips!
Please feel free to follow me below using the following links:
🔵 YouTube
🔵 LinkedIn
🔵 Ultimate DevSecOps library
Summary
I hope you found that useful!
Go and subscribe to my Enterprise Serverless Newsletter here for more of the same content:
Wrapping up 👋
Please go and subscribe on my YouTube channel for similar content!
I would love to connect with you also on any of the following:
https://www.linkedin.com/in/lee-james-gilmore/
https://twitter.com/LeeJamesGilmore
If you enjoyed the posts please follow my profile Lee James Gilmore for further posts/series, and don’t forget to connect and say Hi 👋
Please also use the ‘clap’ feature at the bottom of the post if you enjoyed it! (You can clap more than once!!)
About me
“Hi, I’m Lee, an AWS Community Builder, Blogger, AWS certified cloud architect and Global Serverless Architect based in the UK; currently working for City Electrical Factors & City Electric Supply, having worked primarily in full-stack JavaScript on AWS for the past 6 years.
I consider myself a serverless advocate with a love of all things AWS, innovation, software architecture and technology.”
*** The information provided are my own personal views and I accept no responsibility on the use of the information. ***
You may also be interested in the following:
