Drawing on my experience in securing technology startups and software companies, I wrote a guest blog for ISACA on how to embed security in the modern product development. You can check it out here.
Bug bounty programmes are becoming the norm in larger software organisations but it doesn’t mean you have to be Google or Facebook to run one for continuous security testing and engaging with the security community..
Setting it up can be easier than you might think as there are multiple platforms like HackerOne, BugCrowd or similar out there that can help with centralised management. They also offer an option to introduce it gradually through private participation first before opening it to the whole world.
At a minimum, you can have a dedicated email address (e.g. security@yourexamplecompanyname) that security researchers can use to report security issues. Having a page transparently explaining the participation terms, scope and payout rate also helps. Additionally, it’s good to have appropriate tooling to track issues and verify fixes.
Even if you don’t have any of the above, security researchers can still find vulnerabilities in your product and report them to you responsibly, so you effectively get free testing but can exercise limited control over it. Therefore, it’s a good idea to have a process in place to keep them happy enough to avoid them disclosing issues publicly.
There is probably nothing more frustrating for a security researcher than receiving no response (apart perhaps from being threatened legal action), so communication is key. At the very least, thank them and request more information to help verify their finding while you kick off the investigation internally. Bonus points for keeping them in the loop when it comes to plans for remediation, if appropriate.
There are some prerequisites for setting up the bug bounty programme though. Beyond the obvious budget requirement for paying researchers for the vulnerabilities they discover, there is a broader need for engineering resources being available to analyse reported issues and work on improving the security of your products and services. What’s the point of setting up a bug bounty programme if no one is looking at the findings?
Many companies, therefore, might feel they are not ready for a bug bounty programme. They may have too many known issues already and fear they will be overwhelmed with duplicate submissions. These might indeed be problematic to manage, so such organisations are better off focusing their efforts on remediating known vulnerabilities and implementing measures to prevent them (e.g. setting up a Content Security Policy).
They could also consider introducing security tests in the pipeline as this will help catch potential vulnerabilities much earlier in the process, so the bug bounty programme can be used as a fall back mechanism, not the primary way for identifying security issues.
Using abstractions to think about risks is a useful technique to identify the ways an attacker could compromise a system.
There are various approaches to perform threat modelling but at the core, it’s about understanding what we are building, what can go wrong with it and what we should do about it.
Here is a good video by SAFECode introducing the concept:
The Software and Security Engineering course taught at the University of Cambridge is available for free online. It includes video lectures, slide decks, reading materials and more.
Whether you are new to information security or a seasoned professional, this course will help you build solid foundations.
Lecture 9 covering critical systems is my favourite. It bring together previous discussions on psychology, usability and software engineering in the context of safety. It adds to the array of the case studies from Lecture 6, focusing on software failures and what we can learn from them. It also offers a fascinating analysis of the Therac-25 accidents and Boeing 737 Max crashes.
If you are following my blog, you’ve probably noticed that I’ve been focusing on security-specific AWS services in my previous several posts. It’s time to bring them all together into one consolidated view. I’m talking, of course, about the AWS Security Hub.
You can group, filter and prioritise findings from these services in many different ways. And, of course, you can visualise and make dashboards out of them.
Apart from consolidating findings from other services, it also assesses your overall AWS configuration against PCI DSS and/or the CIS Amazon Web Services Foundations Benchmark, which covers identity and access management, logging, monitoring and networking, giving you the overall score (example below) and actionable steps to improve your security posture.
Similar to the many other AWS services, Security Hub is regional, so it will need to be configured in every active region your organisation operates. I also recommend setting up your security operations account as a Security Hub master account and then inviting all other accounts in your organisation as members for centralised management (as described in this guidance or using a script).
If you are not a big fan of the Security Hub’s interface or don’t want to constantly switch between regions, the service sends all findings to CloudWatch Events by default, so you can forward them on to other AWS resources or external systems (e.g. chat or ticketing systems) for further analysis and remediation. Better still, you can configure automated response using Lambda, similar to what we did with Inspector findings discussed previously.
I wrote about automating application security testing in my previous blog. If you host your application or API on AWS and would like an additional layer of protection agains web attacks, you should consider using AWS Web Application Firewall (WAF).
It is relatively easy to set up and Amazon kindly provide some preconfigured rules and tutorials. AWS WAF is deployed in front of CloudFront (your CDN) and/or Application Load Balancer and inspects traffic before it reaches your assets. You can create multiple conditions and rules to watch for.
If you’ve been configuring firewalls in datacentres before the cloud services became ubiquitous, you will feel at home setting up IP match conditions to blacklist or whitelist IP addresses. However, AWS WAF also provides more sophisticated rules for detecting and blocking known bad IP addresses, SQL Injections and Cross Site Scripting (XSS) attacks.
Additionally, you can chose to test your rules first, counting the times it gets triggered rather than setting it to block requests straight away. AWS also throw in a standard level of DDoS protection (AWS Shield) with WAF at no extra cost, so there is really no excuse not to use it.
If you rely on EC2 instances in at least some parts of your cloud infrastructure, it is important to reduce the attack surface by hardening them. You might want to check out my previous blogs on GuardDuty, Config, IAM and CloudTrail for other tips on securing your AWS infrastructure. But today we are going to be focusing on yet another Amazon service – Inspector.
To start with, we need to make sure the Inspector Agent is installed on our EC2 instances. There are a couple of ways of doing this and I suggest simply using the Inspector service Advance Setup option. In addition, you can specify the instances you want to include in your scan as well as its duration and frequency. You can also select the rules packages to scan against.
After the agent is installed, the scan will commence in line with the configuration you specified in the previous step. You will then be able to download the report detailing the findings.
The above setup gives you everything you need to get started but there is certainly room for improvement.
It is not always convenient to go to the Inspector dashboard itself to check for discovered vulnerabilities. Instead, I recommend creating an SNS Topic which will be notified if Inspector finds new weaknesses. You can go a step further and, in the true DevSecOps way, set up a Lambda function that will automatically remediate Inspector findings on your behalf and subscribe it to this topic. AWS kindly open sourced a Lambda job (Python script) that automatically patches EC2 instances when an Inspector assessment generates a CVE finding.
You can see how Lambda is doing its magic installing updates in the CloudWatch Logs:
Or you can connect to your EC2 instance directly and check yum logs:
You will see a number of packages updated automatically when the Lambda function is triggered based on the Inspector CVE findings. The actual list will of course depend on how many updates you are missing and will correspond to the CloudWatch logs.
You can run scans periodically and still choose to receive the notifications but the fact that security vulnerabilities are being discovered and remediated automatically, even as you sleep, should give you at least some peace of mind.
Committing passwords, SSH keys and API keys to your code repositories is quite common. This doesn’t make it less dangerous. Yes, if you are ‘moving fast and breaking things’, it is sometimes easier to take shortcuts to simplify development and testing. But these broken things will eventually have to be fixed, as security of your product and perhaps even company, is at risk. Fixing things later in the development cycle is likely be more complicated and costly.
I’m not trying to scaremonger, there are already plenty of news articles about data breaches wiping out value for companies. My point is merely about the fact that it is much easier to address things early in the development, rather than waiting for a pentest or, worse still, a malicious attacker to discover these vulnerabilities.
Disciplined engineering and teaching your staff secure software development are certainly great ways to tackle this. There has to, however, be a fallback mechanism to detect (and prevent) mistakes. Thankfully, there are a number of open-source tools that can help you with that:
You will have to assess your own environment to pick the right tool that suits your organisation best.
If you read my previous blogs on integrating application testing and detecting vulnerable dependencies, you know I’m a big fan of embedding such tests in your Continuous Integration and Continuous Deployment (CI/CD) pipeline. This provides instant feedback to your development team and minimises the window between discovering and fixing a vulnerability. If done right, the weakness (a secret in the code repository in this case) will not even reach the production environment, as it will be caught before the code is committed. An example is on the screenshot at the top of the page.
For this reason (and a few others), the tool that I particularly like is detect-secrets, developed in Python and kindly open-sourced by Yelp. They describe the reasons for building it and explain the architecture in their blog. It’s lightweight, language agnostic and integrates well in the development workflow. It relies on pre-commit hooks and will not scan the whole repository – only the chunk of code you are committing.
Yelp’s detect-secrets, however, has its limitations. It needs to be installed locally by engineers which might be tricky with different operating systems. If you do want to use it but don’t want to be restricted by local installation, it can also run out of a container, which can be quite handy.
In the previous blog, I wrote about how you as a security specialist can succeed in the world of agile development, where the requirements are less clear, environment more fluid and change is celebrated not resisted.
Adjusting your mindset and embracing the fact that there will be plenty of unknowns is the first step in adopting agile security practices. You can still influence the direction of the product development to make it more resilient, safe and secure by working with the Product Owner and contributing your requirements to the product backlog.
Simply put, product backlog is a list of desired functionality, bug fixes and other requirements needed to deliver a viable product. There are plethora of tools out there to help manage dependencies and prioritisation to make the product owner’s job easier. The image at the top of this post is an example of one of such tools and you can see some example requirements there.
As a security specialist, you can communicate your needs in a form of user stories or help contribute to existing ones, detailing security considerations. For example, ”Customer personal data should be stored securely” or “Secure communication channels should be used when transmitting sensitive information”. Below are a couple more examples from different categories.
When writing security user stories, you should try and elaborate as much as possible on the problem you are trying to solve, what value it will provide if solved and the acceptance criteria. Each story will then have points assigned which signifies how much effort a particular functionality will require. The process of arriving to the final number is quite democratic and usually involves playing planning (sometimes also called Scrum) poker in which every developer will estimate how long each story is going to take with some discussion and eventual consensus. You can do it with an app as on the image below, or the old school way with a deck of cards.
You don’t have to use the above number pattern, and opt-in instead for the Fibonacci sequence or T-shirt sizes.
It’s important that the security team is involved in sprint planning to contribute to the estimates and help the product owner with prioritisation. Other Scrum meetings, like backlog refinement and daily stand-ups are also worthwhile to attend to be able clarify your requirements (including value, risk, due dates and dependencies) and help remove security related impediments.
A culture of collaboration between teams is essential for the DevSecOps approach to be effective. Treating security as not something to workaround but as a value adding product feature is the mindset product and engineering teams should adopt. However, it’s up to security specialists to recognise the wider context in which they operate and accept the fact that security is just one of the requirements the team needs to consider. If the business can’t generate revenue because crucial features that customers demand are missing, it’s little consolation that security vulnerabilities have been addressed. After all, it’s great to have a secure product, but less so when nobody uses it.