The measurement log: results, failures and what we changed

01First sound

First sound: faster, with setbacks

In the first live test, the AI avatar was heard a median 3.8 seconds after the call, partly because speech recognition closed real voices later than the synthetic ones in the lab. Later it was 1.80 seconds. The curve was not straight: the next run went back up to 2.11 seconds, and in one case the first sound took 5.1 seconds. At one point the timing itself failed altogether.

The figure on our other pages, 1.5–2 s, sums up the runs at the laptop, counted from the last word of the call; the median is about two seconds. Usually a moderator and a guest took part, sometimes with an AI voice as guest. In the lab, with synthetic voices played back, first sound came at 1.7–1.8 s. Unless an entry below says “lab”, it comes from the laptop.

Slow calls

In one run the AI avatar was heard after 1.7 to 4.1 seconds, slowest when the question came before its name. The 5.1 seconds came about because speech recognition closed the second part of a question late.

Opening word

Once the AI avatar opened an answer with “Ja.” (yes) and went on with “Nein” (no); another time it said its opening word twice. In the run at 1.80 seconds the neutral opening came first, the content usually a good one to two seconds later. On the last call of one run the start of the answer was ready before its turn, so the opening was skipped.

Long silence

For a while, one early answer in every run took six to seven seconds to reach its content; in a hall, a long silence would have followed the neutral opening. After that, the silence was gone even on the first call.

Since the “Ja.” followed by “Nein”, the opening is neutral, and a repeated opening word is cut. The AI avatar now takes the second part of a question as soon as it sounds complete.

02Stop

Stop: after the stop came a new answer

STOP in the lab

STOP ended the audio after 334 ms; in an answer resumed after an interruption, after 374 ms.

STOP live

STOP silenced the AI avatar in under half a second, then it started over: a sentence with its name, said while it was talking, had called it again.

“Danke”

For the first time, a “Danke” (thank you) mid-answer stopped the AI avatar, but only after 4.8 seconds, as the microphone also caught its own voice. In the following run, a “Danke” with its name stopped it mid-sentence, though the thanks and the echo arrived together; it was silent about a second after that sentence ended.

After the restart despite STOP, we changed two things: nothing said before a stop calls it any more, and its own voice is meant to count as an echo. After the late “Danke”, it also began looking for a stop word in the latest words while speaking, even with its echo mixed in.

03Calls

Calls: a follow-up without a name went unanswered

Follow-up without a name

Worst at first was a follow-up without a name that got no answer. Soon after, the AI avatar answered follow-ups without its name, and at length when asked to explain. Most recently, first sound came 1.85 seconds after a call by name, 2.20 after a follow-up without.

The bare name

One question waited 18 seconds because speech recognition split the AI avatar’s name from the rest, and it took the bare name as referring to the sentence before.

Meant or not

Through an entire run the AI avatar never jumped in when not meant; three times, though, a question at the end of a long contribution was saved only by its name that followed. It also happened that it cut in on a tester who had only paused. It let a jab from the moderator pass, since nobody had called it.

“Du” question

A question put to it with the informal German “du” (you), on a new topic right after its answer, got silence from the AI avatar until its name came.

After the unanswered follow-up, conversation without a name was added. A bare name now counts as a call, with the question still to come. The AI avatar checks a long contribution as a whole, and if the speaker resumes after a pause, it withdraws the answer before it is heard. It is now meant to lean toward taking such a “du” question right after its answer as addressed to itself, but does not reliably recognize one: later it passed over one until its name came.

04Answers

Answers: a limit set too low

Relevance

In the first run each of the AI avatar’s answers referred to what had been said, and a call spread over three sentences got one answer.

Findings of an AI voice

The first time an AI voice was the guest, answers ran 17 to 22 seconds each, median first sound 3.7 seconds. The AI voice then named six weaknesses, which we checked against the records: no stop signal, replies to half sentences, deaf to topic changes, overlong answers, no closing word, one argument three times.

Length limit

Then several answers hit the length limit, and one tester said their question had been longer than the answer.

The fixed upper limit came after the AI voice’s findings. It is now higher.

05Interruptions

Interruptions: measured only in the lab

Interruption in the lab

After an interruption we triggered ourselves, the “Moment” line came after 1.3 s, counted without speech recognition; 3.6 seconds in, the avatar resumed at the start of the interrupted clause. This is measured only in the lab, not yet live. In the lab test we found and fixed two bugs: the end of the “Moment” line also ended the resumed answer, and a pause of about 0.9 seconds between its sentences counted as the end of the answer.

Name added late

The AI avatar mistook a name added late for an interruption and opened with the “Moment” line.

Not yet measured, not even in the lab, is the rule that followed: a name now counts as an interruption only if said while the AI avatar is already speaking.

06Open

What no test has shown yet

All runs listened through the laptop’s built-in microphone, which also heard the AI avatar itself. That way, its own echo has already set off a “Moment” line. No run has yet caught an intended interruption from the moderator; that is measured only in the lab.

All runs were in standard German. We have not tested English yet, and the characters do not speak it for now.

None of the runs at the laptop lasted long enough for the AI avatar to have to summarize earlier parts. We measured that only with prerecorded discussions.