Compliance Reporting Metrics: Moving Away from Emojis

Published August 31, 2017 by 5 min read

Using compliance reporting metrics often feels like using emojis to answer a math problem. While putting together the compliance program may be time-consuming, it has clear directions. Assessing the compliance program, however, often seems more qualitative than quantitative. Fortunately, ISO 27004: 2016 helps make compliance reporting metrics more clear.

Why do compliance reporting metrics matter?

Anyone who has a smartphone has learned the value of the all-powerful emoji. The little pictures that express ideas or emotions in shorthand have created a language all their own. However, they work to express an idea only if the person receiving the message understands them.

For example:

Courtesy PlayBuzz Quizzes

How many of those would you have figured out on your own?

If a reader is not well versed in the descriptive language, then using emojis to express an idea conveys no meaning.

Compliance reporting metrics work similarly. While qualitative reporting can offer insight into how a program works, truly assessing the stance means offering meaningful numbers to illustrate the value of controls and processes.

How can quantitative compliance reporting metrics strengthen audit results?

Auditors know the emojis. They understand the shorthand and the language. In fact, one of the main reasons for choosing compliance with various standards lies in the ability to have a well-referenced stance that offers confidence to customers and vendors.

Qualitative compliance reporting helps show specific cases within the organization.  A large part of business value relies on intangibles related specifically to a given company. For example, any peer group reviews will be based less in specifics than in descriptions. This makes sense since an organization does not have insider information about its competitors.

Within the information technology area, one of the easiest examples is the idea of reputation. Reputation cannot be quantified mathematically. Since reputation is based on how large groups of people subjectively view an organization, a company can argue they are in the top 10% of customer confidence scores. However, those scores are defined by qualitative measures, not “hard” data.

Quantitative measurements, by contrast, offer an objective look into how an organization operates. Instead of relying on the capricious whims of others, qualitative measures rely on data gathered through verified measures.

When auditors are checking off boxes, they want to see data that represents repeatedly verifiable information. Therefore, compliance reporting metrics may lead to better audit outcomes because there is more consistency and less room for subjective interpretation.

How did ISO 27004: 2016 impact compliance reporting metrics?

While ISO 27001:2013 set out prescriptive standards in many areas, it set forth non-prescriptive methods for evaluating an information security management system (ISMS). While these non-prescriptive specifications created best practices and offered flexibility, they did not necessarily provide a strong basis for evaluating effectiveness.

With this in mind, ISO established ISO 27004:2016 Information technology — Security techniques — Information security management — Monitoring, measurement, analysis and evaluation.  This new standard supplements rather than supplants ISO 27001. The goal of ISO 27004:2016 is to help organizations quantify their processes and controls to create statistically sound compliance reporting metrics.

Section 5, titled, “Rationale,” explains the importance of incorporating measurements and validation to ensure information safety. In section 5.3, the ISO guidance notes,

5.3 Validity of results ISO/IEC 27001:2013, 9.1 b) requires that organizations choose methods for measurement, monitoring, analysis and evaluation to ensure valid results. The clause notes that to be valid, results should be comparable and reproducible. To achieve this, organizations should collect, analyse, and report measures, taking the following points into consideration:

  1. a) in order to get comparable results on measures that are based on monitoring at different points in times, it is important to ensure that scope and context of the ISMS are not changed;
  2. b) changes in the methods or techniques used for measuring and monitoring do not generally lead to comparable results. In order to retain comparability, specific tests such as parallel application of the original as well as the changed methods can be required;
  3. c) if subjective elements are part of the methods or techniques used for measuring and monitoring, specific steps can be needed to obtain reproducible results. As an example, questionnaire results should be evaluated against defined criteria; and
  4. d) in some situations, reproducibility can only be given in specific circumstances. For example, there are situations where results are non-reproducible, but are valid when aggregated.

This discussion regarding comparable results and reproducibility underlies the importance of compliance reporting metrics. To prove compliance, organizations need to not only explain but also quantify their success.

To continue to succeed, companies need reliable measurements that show why their programs meet compliance standards. This type of objective specificity requires more than narrative responses.

With this in mind, ISO 27004:2016 offers three specific guidances to help firms move forward towards a more analytical approach to compliance.

How does Annex A offer insight into compliance reporting metrics?

Annex A sets forth a measurement information model to help organizations determine how to link information to math. For example, Annex A discusses how to link employee knowledge of information security policy to the relevant ISMS entities.

In other words, to fully evaluate a compliance program, companies need to create a process to quantify needs and outcomes. The first step is to determine what information is necessary to measure compliance and then find a measurable concept. From there, the organization needs to create a base measure for analyzing data.

For example, if the goal is to ensure that employees understand an information security policy, your organization could require annual reading of and testing on the policy. If this is done through an online program, your organization can require a minimum score and track the number of attempts it took each employee to meet that score. This data provides measurable, repeatable information for analysis.

How does Annex B build upon Annex A?

Annex B offers greater detail to assist with implementing the overall guidelines of Annex A. Within Annex B, ISO has incorporated tables that cross-reference objectives to ISO 27001:2013. This helps organizations follow the more prescriptive goals of ISO 27004:2016.

For example, Section B.12 offers suggestions regarding Information Security Training metrics as discussed above. It states that information is needed  “to evaluate compliance with [the] annual information security awareness training requirement,” and suggests the “percentage of personnel who received annual information security awareness training” as the coordinating measure.

However, instead of merely suggesting these as goals, Annex B provides a prescriptive formula to show compliance:

[Number of employees who received annual information security awareness training/number of employees who need to receive annual information security awareness training] * 100

Further, it provides a Target Range to determine how best to analyze whether or not the organization is in compliance:

0-60% – Red;

60-90% – Yellow;

90-100% Green.

For Yellow, if progress of at least 10% per quarter is not achieved, rating is automatically red.

Red – intervention is required, causation analysis must be conducted to determine reasons for non-compliance and poor performance.

Yellow – indicator should be watched closely for possible slippage to Red.

Green – no action is required.

By establishing guidelines based on mathematical principles, Annex B provides a way to specify compliance outcomes and prove that a program’s controls work.

What does Annex C provide that helps clarify the other two?

Annex C offers an example of how to write a free-text form measurement construction. In the event that an organization does not like to use charts, Annex C gives a “word problem” that narratively explains the mathematical process of determining the metrics.

How does GRC automation eases the process of gathering compliance reporting metrics?

GRC automation offers a way to streamline the data collection process. Using an automated platform not only offers a centralized location for the collection of data, but also provides an easier tracking method.

When collecting and organizing the necessary quantitative data, smaller organizations start with spreadsheets and emails. As organizations grow, they add employees and applications that lead to a more complex posture, making it increasingly cumbersome to coordinate the people and information to provide quantitative data.

GRC automation streamlines this process. Using a SaaS platform, an organization can schedule tasks and reminders that trigger employee responses. In addition, the information is more easily accessible when an auditor requests documentation.

As technology increasingly drives business outcomes, compliance reporting metrics will become more commonplace. ISO 27004:2016 is the first step in the standardization of qualitative audit performance standards.

For more information on how GRC automation can help streamline the process of engaging in qualitative compliance, read our ebook,”Compliance Management Best Practices: When Will Excel Crush You?

Learn how we can fit into your business.

Schedule a demo to learn how we can help guide your organization to confidence in infosec risk and compliance.

Help us get to know you.

Get a demo