At JustAnswer, the world’s largest Q&A Expert marketplace, I led a project reviewing Expert’s conversations with customers using AI analysis tools.
We used OpenAI’s Whisper to analyse audio their conversations and ChatGPT4-o to process the data. This allowed us to:
- Optimise Expert phone calls in a structured, emotionally intelligent way
- Gain deeper insights into Expert-customer communication
- Improve our customer satisfaction score (CSAT)
Understanding the “conversational racetrack”
Professor Elizabeth Stokoe’s conversational racetrack metaphor provides a framework for audio analysis. This concept illustrates how conversations follow a structured path, similar to a racetrack, and can help us identify hurdles that lead to conflict and misunderstandings.
By using Whisper, we were able to analyse Expert phone calls with customers, focusing on these themes:
- Turn-taking: How effectively is the customer and Expert taking turns? Are transitions smooth or are there frequent interruptions?
- Pauses and silences: Identify moments of hesitation and silence. Are the pauses due to thinking, misunderstanding, or other reasons?
- Disruptions: Look for disruptions that cause misunderstandings or conflict. How are these hurdles navigated or resolved by the Expert?
- Positive outcomes: Highlight parts of the conversation where mutual understanding and satisfaction were achieved.
The conversational racetrack breaks down a conversation into four elements:
- Curves (unexpected challenges)
- Straightaways (smooth, direct exchanges),
- Checkpoints (crucial information points)
- Pit stops (pauses for clarification or detours).
They help us analyse how conversations flow and where potential misunderstandings or disruptions may occur.
Overseeing this experiment, I conducted a detailed review of two specific Expert-customer phone calls using Whisper, focusing on the key elements of the conversational racetrack. This allowed us to pinpoint the Expert’s strengths, weaknesses, and identify areas of improvement.
Whisper case study
This section presents an in-depth analysis of two Expert-customer phone calls on JustAnswer, focusing on how each Expert navigated the conversational racetrack.
Call 1:
Expert performance ratings fall on a scale of 1-5, where 5 is excellent:
- Expert: Dr. Hannah F**
- Customer: Not specified
- Topic: Ovarian remnant syndrome and billing issue
| Timestamp | Name | Element | Details |
|---|---|---|---|
| 00:00 | Dr. Hannah F | Start Line (Introduction) | “Hi this is the doctor from just answer.” |
| 00:30 | Customer | Pit Stop (Billing issue) | “Why do you charge me $48 and not the original agreed $5?” |
| 02:00 | Customer | Straightaway (Medical history) | “I was diagnosed while hospitalised.” |
| 05:00 | Customer | Curve (Diagnosis discrepancy) | “One minute I get in there and I diagnosis the next minute. The next doctor is like no.” |
| 10:00 | Customer | Checkpoint (Current symptoms) | “I get these itchy rashes for the past two years.” |
| 15:00 | Customer | Pit Stop (Doctor experience) | “This doctor from Russia which I should never trust the Russian Communist.” |
| 20:00 | Dr. Hannah F | Straightaway (Medical recommendation) | “You need to have your PCP (primary care physician) order as soon as possible a pelvic ultrasound.” |
| 25:00 | Dr. Hannah F | Finish line (Follow-ups and closing) | “Please follow up with the pelvic ultrasound and then try and get an experience either general GYN (Gynaecologist) or GYN oncologist.” |
Expert performance ratings:
- Active listening: 4
- Clarity of explanations: 4
- Adaptability to customer needs: 3
- Problem-solving approach: 4
- Emotional intelligence: 5
- Technical proficiency: 4
Customer engagement:
- Overall engagement: 4
- Significant shifts: Customer’s increasing distress and Dr. Hannah F’s calming reassurance.
Efficiency and resolution:
- Efficiency: 3
- Clarity of resolution/next steps: 4
Overall performance:
- Rating: 4
- Top strength: Emotional intelligence
- Main improvement area: Adaptability
Summary: Dr. Hannah F exhibited strong emotional intelligence and active listening skills, effectively addressing the customer’s medical concerns. However, there was a slight challenge in adapting to the unexpected billing issue, impacting overall adaptability.
Call 2:
Expert performance ratings fall on a scale of 1-5, where 5 is excellent:
- Expert: Dr. Ali**
- Customer: Not specified
- Topic: Growth on left thumb and difficulties with images
| Timestamp | Name | Element | Detail |
|---|---|---|---|
| 00:00 | Dr. Ali | Start Line (Introduction) | “Dr. Ali.” |
| 00:45 | Customer | Curve (Symptom Description) | “I have a growth that is growing down to my left thumb.” |
| 02:30 | Customer | Pit Stop (Urgent care experience) | “I went to an urgent care to have it cut out and they charge me $300.” |
| 07:00 | Customer | Straightaway (Current treatment) | “I ordered every kind of fungus thing. I’ve done proxide to anti-fungal to carousel.” |
| 12:00 | Dr. Ali | Checkpoint (Expert recommendation) | “It can be done by either of these specialities.” |
| 15:00 | Customer | Pit stop (Customer concerns) | “I don’t have health insurance.” |
| 18:00 | Dr. Ali | Finish line | “You have to track out some plan to get the biopsy.” |
| 20:00 | Dr. Ali | Finish line (Closing) | “I wish I could be of any help to you but I am not currently practicing in the US.” |
Expert performance ratings:
- Active listening: 4
- Clarity of explanations: 4
- Adaptability to customer needs: 3
- Problem-solving approach: 4
- Emotional intelligence: 5
- Technical proficiency: 4
Customer engagement:
- Overall engagement: 4
- Significant shifts: Customer’s frustration with image upload and Dr. Ali’s guidance through the process.
Efficiency and resolution:
- Efficiency: 3
- Clarity of resolution/next steps: 4
Overall performance:
- Rating: 4.0
- Top strength: Emotional intelligence
- Main improvement area: Adaptability
Summary: Dr. Ali provided clear and empathetic advice regarding the customer’s thumb growth, guiding through technical difficulties in image sharing. Despite strong emotional support, adaptability was challenged by the constraints of remote assistance.
**Names have been changed for privacy reasons.
Conclusion
By leveraging Whisper and ChatGPT4-o, we gained invaluable data points in Expert-customer dynamics, identifying pain points, and targeted improvements.
If this was operated at scale, this process could inform and improve tailored coaching materials for all Experts, including:
- Improved communication: Improve turn-taking, reducing interruptions, and responsiveness.
- Emotional intelligence: Help Experts to recognise and appropriately respond to customer emotions.
- Conflict resolution: Equip Experts with strategies to navigate and resolve disruptions smoothly.
Want to learn more about JustAnswer? Read about my prompt design work here. You can also review my UX and brand copy in my portfolio.
Daniel Spence Agnew 