When the Evidence Changes, the Threat Assessment Must Change Too

By Ian Bradbury

Editor’s note: In Part One of this series, former Canadian defence and national-security practitioner Ian Bradbury examined Mark Carney’s U.S. military contingency and the distinction between a possibility, a threatening communication and evidence that an attack is likely. Part Two looks at recently reported threats against Islamic schools in Canada, the difference between assessing a threat and managing risk, and why institutions must be willing to change their conclusions as new evidence emerges.

OTTAWA — New information does not simply add detail to an assessment.

It can change the assessment itself. An initial report may contain little more than a communication alleged to contain a threat. Investigators may not yet know whether the material is authentic, who sent it, whether the sender possesses meaningful capability, what their intent may be, whether they have access to the target or how likely the threatened action is to occur.

Further investigation can establish some of those things, disprove others and raise entirely new questions. New information can change the weight assigned to what was already known, undermine an assumption, strengthen or weaken an attribution, alter an assessment of intent or change how previously disconnected information is understood. Confidence in one conclusion may increase while confidence in another decreases. A relatively small piece of independently derived technical or investigative evidence can materially alter an assessment, while a new allegation attracting considerable public attention may add little or nothing to the evidentiary picture.

New evidence can also change the relationship between information already available. A fact that originally appeared significant may become less so once its context is understood. Information treated as supporting one explanation may ultimately support another. An assumption that allowed several pieces of information to fit together may collapse, forcing the assessment to be reconsidered more broadly. Assessment cannot therefore be reduced to accumulating more information. Volume alone does not necessarily improve understanding. What matters is whether new information changes the weight, reliability or meaning of what is already known and what it actually contributes to the assessment.

A working hypothesis is a necessary part of assessment. The problem begins when that hypothesis becomes harder to change than the evidence warrants. Investigators and institutions bring experience, expectations and existing assumptions to an event. An initial attribution can begin to appear settled. A suspected motive becomes the accepted motive. Two incidents that appeared connected continue to be treated as connected even as the evidentiary basis weakens. Information supporting the original explanation can receive greater weight while contradictory information is discounted, rationalized or given less weight.

Sometimes we want a connection to exist because it provides a coherent explanation for events that otherwise remain uncertain. There is an attraction to an explanation that makes disparate information fit together, particularly where the alternative is accepting that important parts of the picture remain unresolved. That does not make the connection false, but neither does it establish it. Once an analytical frame becomes too narrow, or an institution becomes invested in a particular interpretation, blind spots develop. Information that should be testing the assessment can instead begin to reinforce it, and the analytical process can shift from determining whether the explanation remains supported to finding ways of preserving it despite information that should challenge it. At that point, the analytical frame itself may no longer be fit-for-purpose.

Threat and risk are also routinely conflated when visible protective measures are introduced. Threat assessment is concerned with the nature of the threat, the evidence supporting it, attribution, capability, intent and likelihood. Risk assessment asks what that threat means for a particular target, including its exposure, vulnerability, existing controls and potential consequences. Decisions are rarely made on likelihood alone. A relatively unlikely event can still warrant action where the consequences of failing to act are sufficiently serious, while a more likely threat may require a different response where exposure is limited, vulnerabilities have been reduced or the consequences are more manageable. The threshold for taking precautionary action is therefore not necessarily the same as the threshold for concluding that a threatened action is likely to occur.

Protective action is often taken against the consequence of being wrong, not because the underlying assessment has reached certainty. The investigator, police commander, school administrator, security manager and political leader may all be looking at the same underlying information while making different decisions because their responsibilities differ, as do the consequences of error. An investigator may reasonably conclude that attribution remains uncertain while a police service increases its presence and a school introduces additional security measures. None of those decisions requires the underlying threat assessment to have reached certainty.

The recent threats against Islamic schools across Canada provide a useful example. The Muslim Association of Canada reported that nine of its schools in Ontario, British Columbia and Quebec received the same anonymous email threatening violence over a 24-hour period. Multiple police services opened investigations, schools increased security and police increased their presence around affected institutions.

Waterloo Regional Police confirmed that it received a report on September 21 involving a threatening, hate-motivated email sent to a Muslim Association of Canada school. Officers attended, conducted safety planning with school representatives and offered support through Victim Services. Police also said the threat formed part of a nationwide investigation.

Those facts establish that threatening communications were received, that multiple jurisdictions were investigating and that protective measures were taken. They do not establish who sent the communications, whether the sender possessed the capability or intent to carry out the threatened violence or whether a specific and imminent threat existed against any particular school. Ottawa police, for example, said there was no indication of a “specific or imminent threat” while increasing police visibility at affected locations.

Police can increase their presence because important questions remain unanswered. Schools can increase security because the potential consequences of failing to act are significant. Parents can reasonably expect institutions to take precautions even where investigators have not established that an attack is likely. These are risk-management decisions made in the presence of uncertainty.

Problems arise when the response itself is treated as evidence about the assessment. A visible police presence can be interpreted as confirmation that authorities believe an attack is imminent. Additional security can be taken as proof that investigators have established meaningful capability or intent. An evacuation can be treated as evidence that a threat was judged likely to occur. Yet each action may have been taken precisely because those questions remained unanswered and the consequences of waiting for certainty were considered unacceptable. Response and assessment are related, but they are not interchangeable.

The response can also change the operating environment. Protective measures can redirect resources, alter institutional behaviour and shape public perception before investigators establish whether violence was ever intended. Once the response begins changing behaviour, consuming resources and shaping the information environment, those effects become part of the wider assessment.

Public communication creates another failure point. Authorities often have to explain what they are doing before they can explain with confidence what has happened. Communities want reassurance, journalists want explanations, political leaders are asked to respond and institutions need to demonstrate that an incident is being taken seriously. Those demands frequently move faster than the investigative process.

Once a provisional explanation has been communicated publicly, changing it becomes harder. An institution revising a position it has already communicated is operating under different pressures from an investigator changing an internal assessment. Political statements may have been made, resources committed, communities told that events are connected and media reporting built upon the original explanation. Institutional reputation can then become tied, deliberately or otherwise, to the assessment remaining correct. Changing the assessment becomes institutionally more difficult even where the evidentiary case for changing it has become stronger.

Sometimes the accurate answer is that important questions remain under investigation. There is no analytical failure in uncertainty where uncertainty genuinely exists. The failure begins when gaps in the evidence are filled with assumptions and those assumptions acquire the appearance of fact through repetition, authority or public acceptance.

Institutional culture matters as much as analytical methodology. A sound methodology can still fail if analysts are discouraged from challenging an established conclusion, if leaders treat revision as institutional embarrassment, or if contradictory information is viewed primarily through its implications for the organization rather than for the assessment. An assessment process that cannot accommodate revision when the evidence changes is not functioning as it should.

Later evidence also creates the problem of hindsight. Early decisions are frequently judged using information that was not available when those decisions were made. An institution may have made an entirely reasonable decision to evacuate even if later investigation establishes that a threat was fabricated. Police may have deployed resources because they could not initially rule out a serious threat. Security measures may have been increased because the consequences associated with uncertainty justified them. A later determination that the threat was less serious than initially feared does not automatically make those early decisions unreasonable. They have to be judged against the information and consequences understood at the time.

The reverse is equally important. A decision that was reasonable at the beginning should not be preserved simply because changing it might imply that the initial assessment was incomplete. Later evidence does not change what decision-makers knew at the time, but it should change what they know now. The quality of an earlier decision and the accuracy of the current conclusion are different questions. An early decision may have been justified and a later assessment may still require substantial revision. Treating revision as an admission that the original decision was wrong can create precisely the institutional resistance that prevents an assessment from changing when it should.

New evidence may strengthen an attribution, weaken it or undermine it entirely. A suspected connection may become established or disappear. Apparent intent may be reinforced or weakened by additional evidence of planning, preparation or behaviour, while new information about means, access or opportunity may materially change the assessment of capability or likelihood. Information considered central at the beginning may become peripheral, while something initially dismissed as insignificant may become important once additional evidence provides context. What matters is whether the methodology and institutional culture allow the assessment to change when the evidentiary picture requires it.

There is a significant difference between defending an assessment because the evidence continues to support it and defending it because too much has become invested in the conclusion. In the latter case, contradictory evidence is no longer testing the assessment in the way it should.

Good threat assessment is not demonstrated by being right at the beginning or by defending an initial conclusion to the end. It requires sufficient analytical discipline to distinguish what the evidence supports from what we believe, suspect or want to be true; to recognize when new information has materially changed the evidentiary picture; to test assumptions rather than protect them; and to correct conclusions when the evidence no longer supports them.

That discipline cannot stop when the assessment leaves the analytical process. Institutions have to be capable of correcting themselves. Political leaders have to be capable of acknowledging that an earlier understanding has changed. Media reporting has to distinguish between an assessment that changes because new evidence emerges and one that was unsupported in the first place. An assessment becoming more precise, less certain or materially different as information develops is not necessarily evidence of failure. Refusing to change it when the evidence requires otherwise is.

The public sees the headline. The real work is figuring out what sits behind it.