I remember conducting a detailed security assessment using the CSET (Cybersecurity Evaluation Tool) developed by the US Department of Homeland Security for UK and US gas and electricity generation and distribution networks. The question set contained around 1200 weighted and prioritised questions and resulted in a very detailed report.
Was it useful?
The main learning was that tracking every grain of sand is no longer effective after a certain threshold.
The value add, however, came from going deeper into specific controls and almost treating it as an audit where a certain category is graded lower if none or insufficient evidence was provided. Sometimes this is the only way to provide an insight into how security is being managed across the estate.
What’s apparent in some companies – especially in financial services – is that they conduct experiments often using impressive technology. But have they made it into a standard and consistently rolled it out across the organisation? The answer is often ‘no’. Especially if the company is geographically distributed.
I’ve done a lot of assessments and benchmarking exercises against NIST CSF, ISO 27001, ISF IRAM2 and other standards since that CSET engagement and developed a set of questions that cover the areas of the NIST Cybersecurity Framework.
I felt the need to streamline the process and developed a tool to represent the scores nicely and help benchmark against the industry. I usually propose a tailor-made questionnaire that would include 50-100 suitable questions from the bank. From my experience in these assessments, the answers are not binary. Yes, a capability might be present but the real questions are:
- How is it implemented?
- How consistently it is being rolled out?
- Can you actually show me the evidence?
So it’s very much about seeking the facts.
As I’ve mentioned, the process might not be the most pleasant for the parties involved but it is the one that delivers the most value for the leadership.
What about maturity?
I usually map the scores to the CMMI (Capability Maturity Model Integration) levels of:
- Quantitatively managed and
But I also consider NIST Cybersecurity framework implementation tiers that are not strictly considered as maturity levels, rather the higher tiers point to a more complete implementation of CSF standards. They go through Tiers 1-4 from Partial through to Risk Informed, Repeatable and finally Adaptive.
The key here is not just the ultimate score but the relation of the score to the coverage across the estate.