PracticeTherapy.ai

Can an AI coach judge therapy responses accurately?

Yes, with clear criteria and careful tuning. How the PracticeTherapy.ai coach judges an answer, how GEPA tuned it, and how it scored on the courses' example answers when the current coach went live.

Yes. When our current AI coach went live for our Common Factors (CF) and Motivational Interviewing (MI) courses, it was 97.0% accurate on the 570 example answers in the course books (which is the same set of answers we used to tune it). The PracticeTherapy.ai coach takes a transcript of your spoken answer and compares it with the criteria for the skill you are practicing. It can be done accurately if the criteria are clear and written down and the AI coach has been carefully tuned and tested. Realistically, we tuned the CF and MI courses aggressively using GEPA, a method for improving an AI model's written instructions.

What does the coach see when it judges an answer?

The important part here is getting the voice coach feedback just right. It would've been frustrating to criticize a correct response. The voice coach uses ElevenLabs voice agents for transcription and speech. The AI language model behind the feedback is OpenAI's gpt-5.6-luna, and it gets five things: the name of the skill, the line the client just said in the video, the skill's numbered criteria, one passing example answer and the transcript of what you said.

In skill practice there are specific criteria, and the question is really "did you meet the clinical criteria" so that's what the voice coach focuses on. An answer passes only when it meets every one of the skill's criteria. The coach is instructed to judge the meaning of your answer rather than the exact words you use, so a close paraphrase passes. (The example answer is a reference, not a template.)

The coach responds with two or three short sentences. If you pass, the coach tells you specifically what you did well. If you miss, the coach identifies one thing you did well, the one criterion you missed, what you might try instead, and invites you to try again. If you miss a second time, the coach also gives you the model answer. If you miss a third time, the coach suggests that you take a break and come back to the skill later.

Why judge against written criteria?

The backbone of deliberate practice is its criteria. The American Psychological Association's (APA) deliberate practice books give each skill a short list of criteria, and trainers give their feedback against them (Boswell and Constantino, 2022). One practice book calls the criteria "perhaps the single most important element" of an exercise, because they tell the trainee exactly what to do (Katz and colleagues, 2023). The same book keeps feedback short, a minute at most, and tied to the criteria (Katz and colleagues, 2023).

This is following the exact format of all Deliberate Practice books used to train mental health professionals, including those published by the APA. The exercise screen shows you the same criteria the coach checks.

What is GEPA?

GEPA is a published method for improving the written instructions (a "prompt") that guide an AI model (Agrawal and colleagues, 2025). GEPA doesn't retrain the model. It runs the model on a set of examples with known correct answers. Then it uses an AI model to reflect, in plain language, on why the model got answers wrong. That AI model then suggests changes to the instructions, which are tested, and the process repeats, building on the versions of the instructions that do best. (GEPA stands for "Genetic-Pareto," as different versions of the instructions compete against each other, and new versions are born from the best performers.)

To optimize the Common Factors and Motivational Interviewing coaches at PracticeTherapy.ai, we conducted about twenty measured rounds of tuning work with GEPA. In a single round on a single Common Factors skill, we compared 21 different versions of the instructions across 5,532 scored judgments.

How were the Common Factors and Motivational Interviewing coaches tuned?

How the coach's instructions were tuned

Labeled answers

  • Example answersEach exercise's model answer and an example of a common mistake
  • Real practice answersLabeled where the coach and a blind AI reviewer agreed; Justin decided the rest
  1. Run the coachThe live coach's instructions, tools and settings, on the tuning answers
  2. Read the mistakesAn AI reviewer explains each wrong call in plain words
  3. RewriteSmall changes aimed at the pattern, plus a placebo of the same length
  4. CompareKeep a change only if it beats the current version and the placebo, then repeat
  5. Final testThe 570 example answers from the course books, the same set used for tuning

We built a test bench, a program that sends any answer to the model that runs the coach (with the same instructions, tools and settings as the live coach). This allowed us to score thousands of answers without doing a single voice session.

Every exercise has a model answer and an example of a common mistake. Since these are designed to be right and wrong, respectively, they are labeled by design. We also labeled real practice answers from the site. To label an answer, we'd send it to the coach and a blind AI reviewer. In cases where the two agreed, we used that verdict as the label. Justin hand-labeled every answer where the two disagreed.

We split the answers into groups, based on the client line they are an answer to. No client line in the set we tuned on appears in the set we tested on.

In each round we run the coach on the tuning answers. An AI reviewer looks at all the wrong decisions (correct answers that the coach failed, or flawed answers that it passed) and notes why they happened. It then writes a series of small changes to the coach's instructions aimed at that pattern of failure. All versions of the coach are run side by side on the same set of answers (along with a placebo version containing a note of the same length that doesn't refer to any mistakes, because introducing any text at all shifts the results a little). A change is only adopted if it outperforms the current version by more than the placebo note shifted the results, without failing more correct answers.

The coach's instructions have three layers: instructions shared by all skills, a note for each course, and a note for a single skill. A fix goes in the narrowest layer that will hold it (the measured wins in these two courses were in the course and skill notes). There are skill notes for six skills: two in Common Factors (Empathic Conjecture and Exploring the Internal Impact of External Stressors) and four in Motivational Interviewing (Complex Reflections, Part 1; Complex Reflections, Part 2; Eliciting Change Talk; and Reflecting Change Talk). We then froze the instructions and scored the finished coach on the 570 example answers in the course books (the same set we used to tune it).

What did tuning teach about instructing an AI coach?

The clearest lesson we learned from tuning is to give the AI model an action rather than a verdict. Instructions that prompted the AI coach to take an action before making a decision worked; instructions that told it what to conclude didn't.

Three kinds of action worked. First, run a check before the verdict: the note attached to the skill Complex Reflections, Part 2, for instance, instructs the coach to "find the word or phrase in their reply that names how the client feels" before marking the exercise complete. Second, imagine a consequence: the note for the skill Eliciting Change Talk asks the coach to "imagine the client answering it in one honest sentence" and then to note which way the answer leans. Third, widen the definition of what counts: the skill note for Empathic Conjecture, for instance, lists the various forms a correct opening might take ("It sounds like you're feeling", "Sounds like you feel" or "I hear that you feel").

A few things that didn't work: instructing the model to be "careful" or "lenient" or "strict"; banning a conclusion (the model reached the banned conclusion); asking the model to point to the proof for a conclusion before making the conclusion (the model made up the proof); and restating a rule that the model had already been given.

We learned two other lessons from changing the coach's instructions. One: short notes are better than long notes (in the rounds of Motivational Interviewing, any rewrite that was longer than about 800 to 1,000 characters consistently performed worse than its own shortened version). Two: practice answers are never quoted in the instructions (we had one quoted example that caused the coach to reject correct answers that started with the same words).

How accurate was the coach when the current coach went live?

When we launched 5.6-Luna we had 97%+ accuracy and 90%+ sensitivity. Before the current coach went live, we did the deciding evaluation on the 570 example answers in our course books (the same set of answers that we used to tune the coach). For every exercise in the 19 skills in our Common Factors and Motivational Interviewing courses, this set of answers contains the model answer and an example of a common mistake. The coach made the right decision (pass or fail) for 97.0% of all 570 example answers.

To be more specific: the coach failed 94.0% of the example answers that were flawed (sensitivity) and passed 100% of the example answers that were correct (specificity), i.e., the coach never rejected a model answer. 100% of the example answers that the coach failed were flawed (precision). A single number that balances sensitivity and precision is the F1 score. For the example answers, the F1 score for the coach was 96.9%.

How is a change to the coach tested now?

A change to the Common Factors or Motivational Interviewing coach ships only if it reaches at least 97% accuracy on held-out answers (i.e., answers the new coach wasn't tuned on). Performance is always reported in rates (accuracy, sensitivity, specificity, precision, F1 score) rather than in counts. Whenever the new coach gets a held-out answer wrong, a blind panel of three AI judges (shown only the criteria, the client's line, and the response) reviews the answer to make sure it wasn't mislabeled. Every round of testing has the same goal: the coach should never reject a correct answer (and it shouldn't win by passing everything, because a coach that passes everything teaches nothing).

What does this mean when you practice?

That the automatic feedback you get when you practice on PracticeTherapy.ai will help you improve your clinical skills. For each call the coach makes, you can compare it to the same criteria that the coach uses to assess you. These criteria are listed on the screen for each exercise. (The coach judges only your words, not your tone of voice or body language, and What can't an AI coach hear? explains why that focus works.) The models that power the coach are described in Which AI models does PracticeTherapy.ai use, and why?.

Cover of Deliberate Practice in Common Factors

Common Factors Therapy

Nine relationship skills that every talk therapy shares, practiced out loud with an AI voice coach.

Start practicing now 0$/free

Sources

  • Agrawal, L. A., Tan, S., Soylu, D., Ziems, N., Khare, R., Opsahl-Ong, K., Singhvi, A., Shandilya, H., Ryan, M. J., Jiang, M., Potts, C., Sen, K., Dimakis, A. G., Stoica, I., Klein, D., Zaharia, M., & Khattab, O. (2025). GEPA: Reflective prompt evolution can outperform reinforcement learning. arXiv. https://arxiv.org/abs/2507.19457
  • Boswell, J. F., & Constantino, M. J. (2022). Deliberate practice in cognitive behavioral therapy. American Psychological Association. https://doi.org/10.1037/0000256-000 (cited for the deliberate practice introduction shared by the 17 books of the American Psychological Association's Essentials of Deliberate Practice series, series editors Tony Rousmaniere and Alexandre Vaz)
  • Katz, M., Christensen, M. J., Vaz, A., & Rousmaniere, T. (2023). Deliberate practice of TEAM-CBT. Springer. https://doi.org/10.1007/978-3-031-46019-7
  • PracticeTherapy.ai coach records: the launch evaluation of the gpt-5.6-luna coach, the tuning reports behind it and the coach's instructions (internal).