On the fast lane. @Credits: Photo by toine G on Unsplash
This is the second and final post of a two-part series on DevSecOps practice.
Before we launch into this week’s article, let’s quickly recap the three must-have critical controls in a DevSecOps practice that we covered in the first part.
- Build a security-conscious culture: Human error and ignorance can bring down your system and subsequently, your business in one fell swoop. To mitigate this risk, inculcate a sense for cybersecurity among your people by using relevant and relatable content. Establish guard rails that make it easy for people when they follow the right processes, and find it difficult (but not impossible) to carry on when they attempt to deviate. Appoint a Security Champion to bridge the communication gap between DevOps folks and the cybersecurity team.
- Visibility and sense-making: Knowing exactly which direction you are headed can make all the difference between ending up in a ditch and crossing the finish line. Keeping an inventory of everything, like container images, secrets and software dependencies, is the first step to managing your assets. Develop metrics to know the state of security throughout your development lifecycle so that you can make the necessary improvements.
- Key management: Unmanaged keys can lead to data breaches or even a complete wipeout of all your assets. Secure your keys in a vault that allows you to control who can use the keys, and rotate your keys frequently to reduce any potential damage caused by stolen keys.
Now, let’s move on to the remaining four must-have critical controls for a DevSecOps practice.
4. Software supply chain management
Risks: When building your application, how do you know if you are using quality components or cheap knock-offs? You need a consistent way to manage and assess the quality of your suppliers and the components they supply. Knock-offs can introduce serious vulnerabilities to your system, which can, in turn, become a field day for attackers.
Controls:
- Create an inventory of the software dependencies in your application. Use inventories of software dependencies in your application to identify what needs to be patched and to hunt for threats that lay dormant. A popular term in the market to describe this is software composition analysis (SCA), where tools are used to analyse and identify the dependencies that are used in your application, and how they are used.
- Identify vulnerabilities in dependencies. Check your dependencies for vulnerabilities against publicly available vulnerability repositories, like cvedetails.com, cve.mitre.org, and nvd.nist.gov. Many software supply chain management platforms can cross-check against such databases for you, and flag vulnerabilities very early in the development lifecycle. Some can even prevent a developer from using a vulnerable software library in his code right within the Integrated Development Environment (IDE) before he checks in the code.
- Establish a “software dependency firewall”. Permit the use of only whitelisted dependencies. Considerations for a whitelisting policy include the severity score of identified vulnerabilities, whether the application uses the vulnerable portions of the dependency, and the risk profile of the application in which the dependency will be used. For example, CVE-2018–11040 is a Medium-rated vulnerability in the Spring Framework that allows cross-domain requests through a particular method name. However, if your application is not using the vulnerable method, you could still whitelist the dependency on the basis that your application would not inherit that vulnerability.
- Use distroless Docker images. This minimises available attack surfaces. Standard Docker images may have lots of binaries or dependencies that are irrelevant to your application, but if these binaries are vulnerable or can be potentially harmful, they may become the weakest link in your system. Distroless images are a great way to eliminate unnecessary binaries or dependencies and can result in a more efficient and secure container image with reduced attack surface.
5. Security acceptance tests
Risks: Without a consistent way to make sure that every component in a car works as intended, we would put our lives at risk every time we drive. There would be no assurance that the wheels on a car are going to stay on, or that the engine isn’t going to explode. Automobile manufacturers like BMW put their vehicles through a rigorous and detailed battery of tests because they recognise that performance, comfort and safety need to be consistently validated. Why should software development be any different?
Controls:
- Assemble a full-stack automated testing suite for your application. This ensures that all necessary security controls are in place for every layer of the technology stack across your infrastructure and applications. For example, a typical application stack for containerised workloads would look like this: base operating system, Kubernetes platform, container operating system, and application logic. In each layer, there are specific security-related issues to test. Misconfigurations, application logic flaws and vulnerabilities should be flagged out as part of a pipeline validation of the stack’s security.
- Construct security test cases based on threat models. Threat modelling is a structured way to represent risk and impact to a particular function of the system. It lays out the risk consistently and concisely so that the right decisions can be made. After identifying the relevant security controls to implement as a result of threat modelling, develop security tests to validate the controls. One commonly used threat modelling tool is the OWASP Threat Dragon.
- Augment your current application security testing strategy with Interactive Application Security Testing (IAST). I have personally seen developers overwhelmed by the effort required to filter through scan reports from Static and Dynamic Application Security Testing tools (SAST and DAST). At the time, the number of false positives was so large that we started to wonder whether such tools had become more of a hindrance than a help. Combined with the improper use of tools and the lack of know-how to fine-tune them, this led to delayed timelines and hasty risk acceptance of discovered vulnerabilities. It was apparent that such tools were losing their effectiveness in an environment of fast-paced development. In contrast, IAST can be more effectively used to augment your overall application security testing strategy because:
- It provides greater accuracy in identifying vulnerabilities. This leads to a reduced number of false positives, and is achieved by monitoring the behaviour of low-level application activity, such as Java bytecode. This results in a more deterministic method to discover vulnerabilities, hence resulting in higher accuracy.
- It provides higher-quality, actionable information, aiding in more timely remediation when required. Some IAST tools can even report the exact offending line of code and suggest remediation. Developers do not need to tediously look through the stack trace to identify the offending function or line of code, and can focus their efforts on actually fixing the vulnerability.
- It is easy to integrate with existing workflows without requiring advanced knowledge to fine-tune or configure. Since attack patterns can be more accurately identified with low-level activities, there is less need to customise rules to detect attacks. Without even writing specific security test cases to validate protection against vulnerabilities like SQL injections, IASTs can detect vulnerabilities when you run your regular functional test cases.
- IASTs are generally very lightweight and do not severely impact overall performance. They are designed to run alongside the application to detect and even block threats in real-time. This helps developers reap its benefits with ease, in an almost plug-and-play fashion.
- Conduct resilience tests. This aims to validate the resilience controls put in place, like failover mechanisms, to ensure that the system is operationally ready. Site reliability engineering (SRE) is a discipline that addresses the ability to maintain a system in a reliable state all the time, and describes the collection of controls and architecture decisions that make it happen. Understandably, finding someone with extensive knowledge from such a discipline can be hard to come by, but we can still take certain steps to achieve a degree of system resilience. For example, developers can actively test the functionality of failover mechanisms especially for critical components like the database. Then, make sure that your applications know how to respond when such failovers and swing backs happen. Another control to effectively monitor the health of the system is to periodically conduct end-to-end tests, say every five minutes or so, that simulate real user interaction with the system, like following through the usual steps to update the test user’s particulars in the system, which in turn tests everything from the front end to the backend database and even caches. Proactively get on top of things, instead of reactively waiting for your users to complain about a system outage.
6. Release management
Risks: Especially in the context of Agile development, multiple releases can be made over a short period. If there is no way to track all these releases, you may end up unknowingly deploying the wrong (and perhaps even vulnerable) versions of your application.
Controls:
- Establish a structured versioning nomenclature, and apply policies on upgrading or downgrading versions. For example, only allow the secured and tested versions of your applications to be deployed, by maintaining information about a release with all its dependencies. Cross-reference every deployment with a list of allowed release versions, to avoid deploying outdated or vulnerable versions of your application.
- Mitigate accidental or malicious tampering of codes and binaries that flow throughout the pipeline, with code and binary signing. The integrity of codes and binaries will be assured because any tampering attempts would cause the signature validation to fail. Validate signatures at every phase of the pipeline to achieve the assurance of integrity.
7. Runtime protection
For this control category, I would like to make a special mention on runtime protection for containerised workloads. While many have already started using containerised workloads and microservices, adoption of the relevant runtime protection is only just catching up. It has great potential to thwart critical vulnerabilities in containerised workloads.
Risks:
Endpoint protection (EPP) and endpoint detection and response (EDR) solutions protect the operating system space because it has visibility over system activities. But with the rise of modern application architecture and platforms like microservices-based applications running as containers orchestrated by Kubernetes, most EPP and EDR solutions today are not capable of protecting individual containers and their orchestrator just as they would an operating system, as they do not have visibility of activities within containers. Without such visibility, attacks that come through containers could potentially remain undetected and even spread to the underlying operating system despite having EPP or EDR running. Attackers could also choose to steer away from the watchful eyes of EPP and EDR solutions by choosing to operate mostly in the container platform space.
An attacker could potentially cause much damage in the container platform space without having to directly attack the underlying operating system, in the likely scenario that a majority of the business logic lives in containers. In that space, the attacker could pivot from one container to another without triggering the EPP or EDR. They can then gain control of other containers to subvert business rules for fraudulent activity. For example, they can bypass an authentication flow to impersonate another user, thus gaining access to that user’s account to conduct fraudulent transactions.
Controls:
- Consider container security solutions with behaviour profiling capabilities. Effective solutions profile and block ‘low-level’ activities such as syscalls made to the underlying Linux operating system, which are necessarily triggered if an attacker tries to create processes which require increased privileges or access to the OS kernel. Hence, any malicious actions performed in the container would result in a deviation from the normal set of syscalls, which will stick out like a sore thumb. Monitoring the integrity of files in the container’s filesystem also acts as an additional tripwire to detect potentially malicious modifications to critical system files.
- Behaviour profiling requires a time to form a baseline of ‘good behaviour’ in a controlled environment, before the container is run in a production environment. The more complex an application is, the longer the learning duration and also reduced effectiveness to block bad behaviour. To maximise the benefits of runtime protection aided by behaviour profiling, keep microservices, well, micro, to reduce learning periods and increase effective prevention. However, this requires significant discipline in architectural decisions at the early stages of DevSecOps.
Next steps
You can achieve some quick wins right now by doing these few, simple things:
- Build on existing controls. You might already have some of the controls in one form or another. You can start right now by improving existing controls, then progress to implement high-impact controls.
- Identify the weakest links in your development process. Assess which of the seven controls mentioned would give you the greatest value through a cost-benefit analysis, and start by addressing the weakest links.
- Start by building the foundations right. Visibility and sense-making are the foundational pieces for a successful DevSecOps, because every stage of the development lifecycle needs to close the feedback loop for decision making, enabled by the sense-making piece. Then, proceed to address potential downstream issues by securely managing the software supply chain. This will save you a lot of headaches at the onset and even further down the road.
Author: Josias Teo
Originally published on
CSG @ GovTech
For more deep-dive cybersecurity tools and techniques, please visit
CSG @ GovTech