Why summative is not a pass fail activity

Continuing the “Basic Human Factors Concepts for Medical Devices” series, this time I want to talk about why summative evaluation is not a pass/fail activity. Thanks to @Andreas for suggesting the topic.

Let’s start with some basic knowledge about summatives. Traditionally, summative studies are conducted using simulated-use techniques. This is one of the techniques HF professionals have in their toolbox to represent how users may interact with the device in the intended use setting.

Usually, a simulated-use study includes, at minimum, two key elements. First, an observation session, where users perform the tasks by themselves and the researcher observes performance and documents use errors, close calls, and difficulties. Then, a root cause investigation, where the researcher tries to understand the reason behind those observations.

You cannot run these evaluations with the same approach as a quantitative activity. There is no magic error threshold that equals success. And the reason is simple. You need to know what a use error means to take proper action, and you do not know what a use error means until you understand why it happened. So counting use errors is not enough.

A use error may reveal a serious design issue that could lead to harm. Another use error may be related to a misunderstanding that is already adequately controlled. The number alone does not tell you whether the device is safe and effective. The root cause does.

That is why the real value of a summative is not about counting use errors or setting a number to decide if your device “passes.” Before you can say your Human Factors work is complete, you have to understand the root causes of each use error and know whether those findings are reasonably controlled, whether further risk reduction is needed, or whether bigger changes are still required.

But theory aside, the more important point is that a summative is just the end of a process that should have been running for a while. By the time you get there, the device and its instructions should already fit the real world well enough to support safe and effective use. It is not an exam you either pass or fail with a single score. It is a confirmation step after all the real work is done.

I have seen summative evaluations with a small number of use errors lead to a “not acceptable” conclusion and summative evaluations with more observed errors still support approval. What matters is whether you can show that the critical use-related risks were properly understood, addressed, and supported with solid evidence through development.

A good summative is not about zero errors. It is about a clear story behind every design choice, every observed use error, and every residual risk. Regulators want proof that you identified the critical risks early, acted on them, and did not just rely on hoping for a perfect study result at the end.

#humanfactors #summative #medtech