Listener-specific language switching needs a rule for audio already in flight. A participant can switch from Japanese to French while the Japanese synthesis is queued or playing, and a late completion can otherwise reintroduce the old language after the interface shows the new preference.
I would attach a preference generation and speaker-turn ID to each translated segment, then discard or finish older segments according to an explicit playback policy. Test a switch during synthesis and again during reconnection. That makes the recovery behavior observable and keeps shared translation work separate from each listener's current playback state.