Why Can’t Users Teach Siri about Its Mistakes?

posted in: September 2019 | 0
Photo of people holding up signs

Original photo by rawpixel from Pixabay


Why Can’t Users Teach Siri about Its Mistakes?

With the recent fuss about Apple contractors listening to some of our Siri conversations (see “Apple Suspends Siri’s “Response Grading” Eavesdropping,” 2 August 2019), I couldn’t help but wonder why we users can’t perform this task. If you’re anything like me, you already talk back to Siri when it makes mistakes or triggers unexpectedly.

Apple never said precisely what its contractors were doing, but TechCrunch’s Matthew Panzarino, who Apple favored with the exclusive explanation, described the “grading” process like this:

This takes snippets of audio, which are not connected to names or IDs of individuals, and has contractors listen to them to judge whether Siri is accurately hearing them—and whether Siri may have been invoked by mistake.

His description brings to mind Apple’s famous “1984” ad, and its endless rows of gray-clad workers, except they’re all wearing headphones, listening to Siri audio snippets and pressing one of three buttons for each: Correct, Incorrect, Inadvertent Invocation. It’s a dystopian image, and we can only hope the actual job is less soul-sucking.

Whether or not the scene I’m visualizing has any overlap with reality, I see no reason that we users can’t provide the feedback Apple needs to improve Siri’s accuracy. Here’s my proposal.

Opt-in, for Starters

Regardless of anything else, Apple would need to make this proposed Siri feedback program opt-in, by asking users if they want to enable Siri feedback during initial setup and via a toggle switch in Settings > Siri & Search.

That’s just polite, given that inadvertent invocations can record speech that the user never intended to be public. Apple has already said it will make the current human-driven grading program opt-in.

Feedback via Siri Shortcuts or Buttons

When enabled, the Siri feedback switch would create three Siri shortcuts with the default phrases of “Good job,” “That’s wrong,” and “I didn’t invoke you.” The Settings screen should provide usage instructions as well. You’d be able to give Siri feedback after any response, either via voice or with buttons for devices with screens.

Left: a mockup of the proposed Siri shortcuts Right: a mockup of the feedback buttons
Left: Mockup of the proposed Siri shortcuts. Right: Mockup of proposed feedback buttons.

As with other Siri shortcuts, you’d be free to change the phrase however you like—I imagine the inadvertent invocation would become “Shut up!” for many people and there would be more colorful replacements for “That’s wrong.”

Obviously, there should be no requirement that you provide feedback on any particular interaction with Siri, so the “Good job” response probably wouldn’t get that much use. After all, if Siri works properly, there’s little reason to offer praise—you’re not training a dog. I’d reserve such feedback for when Siri surprises me with an accurate response to a difficult query.

I suspect the other two corrections—for an incorrect response or an inadvertent trigger—would be far more commonly used. I already have trouble restraining myself from admonishing Siri when it misunderstands my commands, and it drives me nuts when Siri pipes up for no apparent reason whatsoever. OK, I admit it, I’d be among the people changing “I didn’t invoke you” to “Shut up!”

Plus, when you correct one of these Siri mistakes, Siri should apologize for messing up. A simple “I’m sorry” would go a long way toward reducing user annoyance.

Ideally, Apple would enhance Siri so it could keep listening for feedback and followup queries after it responds to you, allowing you to avoid prefacing the feedback phrase with “Hey Siri” or a button press. Google Home already does this with its Continued Conversation feature, which listens for 8 seconds after it responds, and Amazon’s Alexa listens for 5 seconds after responding with its Follow-Up Mode.

I don’t know if Apple could get the results it needs by just taking user reports at face value or if it would still need to send some percentage of reports to humans for additional verification.

Don’t Take That Tone with Me

So far, what I’ve described is largely mechanical, and well within the capabilities of both Apple and users. I’d happily start correcting Siri tomorrow if Apple added such a feature.

But I’d like to see Apple’s vaunted engineers go further and start recognizing tone of voice and emotional state. I assume that a machine-learning algorithm could be taught to distinguish between a user’s normal tone of voice and volume and when they speak more abruptly and loudly. I’m sure lots of us raise our voices in irritation when Siri interrupts a conversation or completely biffs what seems like a simple command.

“Hey Siri, play a Beatles song.”

“NO. Not Ann Peebles! The BEATLES!”

With such recognition—and assuming the Siri feedback switch is enabled—Apple could detect signals that Siri had messed up without any extra interaction from the user. The accuracy level would undoubtedly be lower because your irritation might have resulted from the cat having just brought in a dead mouse rather than anything Siri did, but it would still be better than nothing.

More Feedback for Less Cost

Matthew Panzarino said that Apple’s grading program likely evaluated fewer than 1% of Siri’s daily requests. In 2015, Apple noted Siri was handling 1 billion requests per week, which works out to 143 million requests per day. That number has undoubtedly skyrocketed since 2015, but let’s go with it.

How many people would Apple need to hire to process even 1 million requests per day—a round number in that “less than 1%” range? Assuming someone could process 3 requests per minute on average, and keep that up for a full 8-hour day, Apple would need at least 700 workers. So unless Apple is really processing far fewer requests than “less than 1%” implies, it seems likely that the company has been employing thousands of people to improve Siri’s accuracy.

Replace those people with millions of Siri users, and I’d suggest that Apple would get vastly more feedback that was potentially more accurate and would save boatloads of money. Besides, this is the sort of thing computers should do, not people. That job must eat your brain.