Your Security Assessment May be Accurate…and Still Miss the Biggest Risks

Posted in Governance & Accountability, Michael’s Perspective, People, Places & Behaviour, Preparedness & Continuity, Risk & Decision-Making, Security in Practice

Why better security decisions begin long before the risk analysis starts.

Organizations invest considerable time, resources, and expertise in security risk assessments. Security professionals review policies, examine security technologies, identify vulnerabilities, and develop recommendations to strengthen the organization’s security posture.

Yet following a security incident, a familiar question often emerges:

How did we miss that?

The immediate assumption may be that the assessment was inadequate, a control failed, or the assessor overlooked an identifiable vulnerability.

But there is another possibility worth considering.

What if security professionals conducted the assessment competently, the analysis was technically sound, and the recommendations were reasonable—yet the information they collected did not fully represent how the organization actually operated?

This raises a fundamental question about the way we approach security risk assessment.

Are we assessing how security should function, or how it actually functions under everyday operating conditions?

The Difference Between Documented Security and Operational Reality

Most organizations have an established understanding of how their security programs should operate.

Policies establish expectations. Procedures define responsibilities. Technologies provide protective capabilities. Training communicates required behaviours.

Collectively, these elements describe an organization’s intended security posture.

However, the existence of a documented control does not necessarily establish how consistently or effectively that control operates in practice.

A controlled entrance may be secure by design but routinely left open to accommodate deliveries. Organizations may thoroughly document their visitor management procedures but apply them differently during peak operating periods. Employees may develop informal workarounds because established security requirements conflict with operational demands.

These conditions do not necessarily indicate negligence or deliberate non-compliance.

Often, they reflect the way people adapt to the realities of their working environment.

The challenge for security assessors is that these adaptations may not be readily apparent through documentation reviews, structured interviews, or scheduled demonstrations.

And if assessors do not discover these conditions, they may never become part of the subsequent risk analysis.

Organizations rarely fail because they don’t know what their policies say. They fail because they don’t always know how people actually perform their work.

The Audit Paradox

This challenge led me to explore a concept I refer to as the Audit Paradox.

The premise is straightforward:

The more formal and evaluative an assessment environment becomes, the greater the possibility that individuals will demonstrate intended practices rather than their routine operational behaviour.

When individuals know an assessment is underway, their behaviour may change.

Employees may follow procedures more carefully or temporarily suspend informal practices. Supervisors may become more attentive, and employees may demonstrate how they are expected to perform their activities rather than how they typically perform them.

This does not mean individuals are intentionally misleading the assessor. Nor does it mean conventional security assessments are inherently unreliable.

It means that the assessment process itself can influence the environment that assessors observe. For security professionals, the implications deserve consideration.

A security assessment may accurately describe the controls, conditions, and behaviours present during the assessment without fully capturing the operational variations that exist outside that period.

The distinction is important.

An assessment can be accurate in what it records while remaining incomplete in what it discovers.

Why Information Fidelity Matters

Security risk assessment depends upon information.

Threat identification, vulnerability analysis, likelihood estimation, consequence assessment, and treatment recommendations all rely on the quality and completeness of the information available to the assessor.

Security professionals have devoted considerable attention to improving the analytical components of risk assessment. Established methodologies provide structured approaches to identifying, evaluating, and treating risk. However, even a sophisticated analytical framework cannot independently compensate when assessors fail to identify important operational conditions.

This raises a question that deserves greater consideration:

How faithfully does the information an assessor collects represent the operational environment from which it originates?

Researchers have long examined information quality through established concepts such as accuracy, completeness, contextual relevance, and faithful representation. These considerations are particularly important in security risk assessment, where information collected during discovery becomes the foundation for subsequent analysis and decision-making.

Building upon these established concepts, I propose that we must also consider information fidelity at the point of acquisition not simply whether we accurately record or preserve information, but whether that information adequately represents the operational conditions we seek to understand.

From Information Accuracy to Operational Representation

This distinction matters because information can be entirely accurate without being sufficiently representative of everyday operations.

An assessor may accurately document a facility’s access control procedures, confirm the functionality of its security technologies, and verify that established policies are in place. Yet if routine operational workarounds, informal practices, or variations in employee behaviour remain undiscovered, the resulting assessment may present an incomplete picture of the organization’s actual risk exposure.

Within the Observational Security Assessment framework, I refer to this relationship as the Information Fidelity Principle of Security Assessment:

The fidelity of information acquired during discovery constrains the reliability of security risk assessment outcomes.

This proposed principle does not suggest that better discovery eliminates uncertainty or guarantees more accurate risk predictions. Rather, it recognizes that analytical rigour cannot independently compensate when assessors fail to identify significant operating conditions.

The implication is straightforward.

Better analysis remains essential. But better analysis begins with better information.

Observational Security Assessment: A Different Starting Point

These considerations led me to develop Observational Security Assessment (OSA), a structured discovery methodology intended to improve how security assessors identify, understand, and validate the operational conditions that influence risk.

OSA does not claim to introduce observation, interviewing, or contextual inquiry as new assessment techniques. Its proposed contribution lies in how OSA deliberately sequences, integrates, and documents these established practices to strengthen discovery before formal risk evaluation.

Rather than replacing recognized threat and risk assessment methodologies, OSA seeks to improve the quality of the information upon which those methodologies depend.

How do we develop a more reliable understanding of the environment we are about to assess?

OSA places greater emphasis on observing ordinary operational conditions, understanding the reasons behind behaviour, and comparing actual practices with documented expectations.

Its approach includes observation-first discovery, contextual inquiry, examination of operational workarounds, comparison of multiple evidence sources, and clear distinctions between what assessors directly observe, what others report, and what assessors infer from the available evidence.

The objective is not to identify employee shortcomings or create an atmosphere of surveillance.

It is to understand how security functions within the broader operating system.

An employee bypassing a procedure may represent a compliance concern. But the same observation may also reveal an impractical workflow, conflicting organizational priorities, insufficient resources, or a security control that no longer reflects operational requirements.

Those possibilities can lead to substantially different recommendations.

OSA helps assessors distinguish among them.

From Observation to Better Decisions

Consider a facility where employees repeatedly leave a secured access point open during busy delivery periods.

A conventional control-focused assessment might identify the condition as a failure to enforce access procedures and recommend additional training, signage, or supervisory oversight.

Those measures may be appropriate.

But observation and contextual inquiry could reveal that employees use the access point to compensate for a receiving process that no longer accommodates the facility’s operational demands.

In that situation, the underlying issue may extend beyond employee behaviour.

The more effective recommendation could involve redesigning the receiving workflow, modifying the access arrangement, or aligning security controls with actual operating requirements.

The difference is not simply the identification of an open door.

It is understanding why the door is open.

That distinction illustrates the potential value of an observation-led discovery process.

It can shift the assessment from identifying an apparent control deficiency toward understanding the conditions that produce it.

And that understanding can lead to recommendations that are more practical, sustainable, and responsive to the organization’s actual risk exposure.

A Different Conversation for Security Leaders

For executives, boards, and security leaders, this discussion extends beyond assessment methodology.

It concerns confidence in the information used to make decisions.

Organizations routinely receive reports describing their security posture, identifying vulnerabilities, and recommending investments.

But how frequently do decision-makers ask whether the assessment captured the conditions under which their security controls actually operate?

Did assessors conduct the assessment during representative operating periods?

Were informal practices and operational exceptions explored?

Did assessors understand employee behaviours within their operational context?

Did assessors compare documented procedures with routine activities?

Did assessors independently corroborate their observations?

And, perhaps most importantly, how confident are we that the assessment reflects operational reality rather than simply documented intent?

These questions do not diminish the importance of conventional risk assessment.

They strengthen the foundation upon which it depends.

The Future of Security Assessment Begins with Discovery

Security assessment has evolved considerably through improved methodologies, more sophisticated technologies, and increasingly structured approaches to risk analysis.

Those advances remain important.

However, the next opportunity for improvement may lie earlier in the process.

Before assessors rate vulnerabilities.

Before they complete risk matrices.

Before they prioritize recommendations.

Before executives make investment decisions.

There is an opportunity to improve how we discover and understand the conditions that give rise to risk.

Observational Security Assessment is my contribution to that conversation.

It is a developing methodology that draws upon practical assessment experience and invites further examination, critical evaluation, and refinement through continued application and research.

Its central proposition is simple:

Security decisions are only as reliable as the information collected before the analysis begins.

Perhaps the most important question we can ask at the beginning of an assessment is not whether the organization has the right security controls.

Perhaps it is whether we truly understand how those controls function when no one is conducting an assessment.