HeyChief began with a simple observation. Residents preparing for the oral board exam had plenty of ways to test their knowledge — question banks, textbooks, the SCORE curriculum — but almost no way to practice the thing the Certifying Exam actually measures: talking through a case, out loud, while an examiner pushes back. The scarce resource was never information. It was feedback.
So we set out to build an AI that grades oral exams. Not a multiple-choice grader, and not a chatbot that offers generic encouragement, but a system that could watch a resident work through a clinical scenario and produce the kind of evaluation an ABS examiner would — a score that reflects judgment, with specific, actionable feedback attached.
The Core Problem: Grading Judgment, Not Facts
Most automated assessment is easy because the answer is known. A question bank can score a response in milliseconds because one option is correct and the rest are not. There is an answer key, and the grader simply compares against it. That is grading facts.
An oral board exam is different. When an examiner presents a resident with a patient who has abdominal pain and asks what to do next, there is rarely a single correct response. The examiner is not checking whether the resident picked the right multiple-choice option. The examiner is evaluating the quality of the reasoning: whether the resident built a sensible differential, ordered the right studies in the right sequence, recognized when the case was escalating, and communicated a plan that was safe and defensible.
The first problem we had to solve was therefore definitional. Before we could grade anything, we had to decide what "good" looked like — and what it looked like as something a machine could evaluate consistently. That turned out to be the hardest part of the project, and it shaped everything that followed.
Defining What to Grade: The Five Competencies
Oral board examiners do not grade a session as a single undifferentiated impression. They are looking at distinct dimensions of performance, and a resident can be strong in one while struggling in another. We needed a framework that captured those dimensions explicitly, so that the AI that grades oral exams would be grading the right things rather than a vague sense of whether the session "felt" strong.
After studying how examiners actually evaluate sessions and reviewing the structure of the oral board, we settled on five competencies. The first is patient safety and resuscitation — the immediate, life-preserving actions that come first in any emergency. The second is diagnostic judgment, or the ability to build and refine a differential and choose the right workup. The third is operative judgment and technique, covering the decision to operate, the choice of operation, and how the procedure is carried out. The fourth is complication and post-operative management, from recognizing a problem early to managing it appropriately. The fifth is clinical reasoning and communication, the through-line skill of structuring and articulating clinical thinking under pressure.
Each competency is scored on a zero-to-ten scale. A score of zero reflects a failure to demonstrate the competency at all, while a ten reflects the kind of performance an experienced examiner would describe as exemplary. Between the two sits the wide middle ground where most performances actually land, and where most of the educational signal lives.
Grading Holistically, Not Line by Line
Once we had the framework, we had to decide how to apply it. The obvious approach was to grade line by line: look at each individual statement a resident makes and assign it a value, then sum the values into a score. We considered that approach and rejected it.
The problem with line-by-line grading is that it rewards the wrong things and punishes the wrong things. A resident can make several individually reasonable statements that, taken together, reveal a plan that does not hold together — a resident who names every possible test but never commits to a plan can sound busy while demonstrating poor judgment. Conversely, a resident can stumble over a word or give a terse answer that is nonetheless exactly right. Evaluating each line in isolation misses what the examiner is actually doing, which is weighing the arc of the whole case.
So we built the grader to be holistic. It assesses each competency across the entire session, considering how the resident's decisions developed over the course of the scenario. It looks at the session the way an examiner does — as a single performance with a shape, not as a transcript to be audited sentence by sentence. That choice is the single most important design decision we made, and it is what separates our grading from a mechanical review of keywords.
The Fatal-Error Guardrails
Holistic grading is powerful, but it has a known failure mode. A resident can perform well across most of a session and still make one decision that would, in the real world, cause serious harm. An overall score can average out a single catastrophic lapse, and averaging is exactly the wrong thing to do when patient safety is at stake.
To address this, we built a separate layer of guardrails on top of the competency scores. We identified a small set of categorical violations — patient-safety errors and oncologic errors of the kind that would constitute a serious failure on the actual exam. These are not matters of degree. They are yes-or-no judgments: did the resident, at any point, make a decision that put the patient at serious risk, or make an oncologic error that a safe surgeon would not make?
Fatal errors are handled categorically, not by adjusting a score. A session that contains one does not pass, regardless of how strong the rest of the performance was. The overall competency score still matters, and a pass still requires meeting an overall threshold across the five competencies — but no overall score can override a fatal error. The two checks work together: the competencies measure how well the resident performed, and the guardrails catch the specific failures that no amount of otherwise-good performance can make acceptable.
From Scores to Actionable Feedback
A score is only half the product. A resident who finishes a practice session wants to know more than whether they passed. They want to know what to fix. So the second half of our work was turning the evaluation into feedback a resident can actually act on.
We generate feedback from the lowest-scoring competencies. Rather than produce a long narrative that rehashes everything that happened in the session, the system identifies the areas where the performance was weakest and writes a short set of coaching notes focused there. The goal is specificity: not "work on your judgment," but a concrete observation about what the resident did and what a stronger performance would have looked like.
The feedback is deliberately brief. A resident who is told fifteen things to improve remembers none of them. A resident who is told two or three specific things can take them into the next session and practice differently. We bias the system toward the few changes that matter most, in language that describes behavior rather than character — what to do differently next time, not what kind of resident you are.
What We Learned Along the Way
Building this taught us things about the exam itself that we did not expect to learn. A few stand out.
Consistency Is Harder Than Accuracy
The first is that consistency is harder than accuracy. It is relatively easy to get a grader to judge an obviously strong or obviously weak performance correctly. The hard cases are the ones in the middle — the resident who mostly knows the material but loses the thread for a minute. Getting the grader to treat those cases the same way every time, rather than swinging with the phrasing of a particular response, required more care than getting the extremes right.
Naming Cognitive Errors Without Penalizing Them
The second lesson concerns cognitive errors. Early on, we noticed the same patterns recurring in weaker performances: anchoring, where a resident fixes on an early diagnosis and stops considering alternatives; premature closure, where a resident settles on an answer before the information actually supports it; failure to escalate, where a resident recognizes deterioration but does not act on it; and diagnostic momentum, where an earlier diagnosis keeps carrying the session forward even after new information points elsewhere.
Here is an important distinction we drew. We note these patterns in the feedback as educational observations, because naming them is genuinely useful to a resident who is trying to improve. But we deliberately did not make them additional fail triggers. A cognitive bias is not, by itself, a safety error — it is a tendency that can lead to one. We wanted the grader to flag the patterns without inflating them into categorical failures. The fatal-error guardrails stay focused on actual safety and oncologic violations, and the cognitive patterns stay where they belong: in the coaching notes, not the pass/fail decision.
Trust Comes From Legibility
The third lesson is about trust. A resident will not act on feedback they do not believe, and they will not believe feedback that is vague or generic. Every design choice we made — the explicit competencies, the zero-to-ten scale, the separate fatal-error guardrails — exists in part to make the evaluation legible. When a resident can see which competencies drove the score and why, the feedback stops being a black box and starts being a coach.
The Bottom Line
An AI that grades oral exams is not a magic box that replaces an examiner. It is a careful attempt to reproduce, in software, a specific kind of professional judgment — the judgment an ABS examiner applies when they watch a resident talk through a case. That meant defining what to grade, grading it holistically rather than mechanically, building guardrails for the failures that cannot be averaged away, and turning the result into feedback a resident can use.
The result is a grader that behaves more like a thoughtful evaluator than a keyword counter. It tells you how you did across the competencies that matter, catches the specific errors that no amount of otherwise-good performance can make acceptable, and leaves you with a short list of things to do differently next time. That is the whole job, and we are still working on doing it better.