Audio only fallback is a legitimate tool. Half my client calls do not need my face, and dropping video takes the requirement from megabits to kilobits. Nobody has ever objected.
Set a limit before you arrive. I decide I will talk to four people and I let that make the decision for me rather than reading the moment. Takes the judgement out of it.
Ask for the number the day before completion as well as at estimate stage. Mine moved by about 300 between the two and nobody would have told me if I had not asked.
Check your keyboard is not also doing this in the tap dance or combo code if you have any. Two features racing for the same key produces intermittent behaviour that looks exactly like a timing problem.