The Experience

Geoff:

The experience.

At the core of every intelligent system is a connection between the intelligence and the user. This connection is called the intelligent experience. The intelligent experience might automate an action. It might prompt the user. It might add information to a display. There are many options. This section introduces the goals of an intelligent experience and some options for balancing one.

An effective intelligent experience will present the intelligence to the users by choosing how they perceive it. Balancing the quality of the intelligence with the forcefulness of the experience. Achieves the system’s objectives by encouraging users to behave in productive ways and without getting upset. It will minimize any intelligence flaws by reducing the impact of small errors and helping users discover and correct any serious problems that do occur. And it will create data to grow the system by shaping interaction. So, they produce unbiased data that is clear, frequent and accurate enough to improve the system’s intelligence. There are number of factors that go into balancing intelligence experience to achieve these goals. The forcefulness of the experience refers to how bold it is when taking actions. Does it do small things, or big things? Are they easy to undo, or hard to undo when there’s a problem? The frequency of the experience can be controlled in a number of ways. You could have your intelligence interact with a user whenever the intelligence has any idea at all. Or, you could wait and interact when the intelligence is sure that it has a very good idea. Or, you could explicitly limit the interactions to something like one per hour, one per day. The value of success is how much benefit the user perceives from a positive outcome and how likely they are to notice that they got that benefit because of the entire intelligence in the first place. The cost of a mistake involves how much damage the action causes to the user, how likely the user is to notice the mistake, and how much of the mistake they can recover from. The quality of the intelligence refers to how many mistakes it makes. And I promise you, your intelligence makes mistakes. Just about everything built with machine learning is going to make mistakes. So, you need to be able to deal with that.

Now, let’s go through a simple example of balancing the quality of the intelligence in the forcefulness of the experience. This is just two of the factors that we discussed. And we can balance in other ways, but we’ll just give this example. The chart there compares to the quality and forcefulness in a not super technical way. Higher numbers mean better quality and more forcefulness. Lower numbers mean worse quality and less forcefulness. Don’t get too hang up on that. Let’s consider how this might work in a system designed to combat spam email. When an email arrives, the intelligence examines it and predicts if the message is spam or not. A forceful experience would be to delete every message that the intelligence thinks is spam. The intelligence thinks an email is spam? Poof! Gone. If the intelligence is super accurate, that’s great. Because users never need to think about spam. They never see it. They never waste any time dealing with it. But, when the intelligence makes a mistake, that’s a pretty costly mistake. The users’ important communications are deleted with no notifications and really no re-course. A less forceful experience would be to move the emails that the intelligence thinks that are spam to a junk folder. This isn’t nice because users have to read through their junk folders ever so often. But it’s a trade-off that allows less accurate intelligence that provides significant value to users. Less forceful still would be to put every message into the user’s inbox. No matter if it’s the intelligence thinks its spam or not, but to annotate the ones that the intelligence thinks might be spam with some hint for the user. Maybe by adding a note to the top of suspicious emails that say, “Reminder: Don’t give out your personal information in response to emails.” So, which is right? Well, it depends to the rest of the system. If the intelligence is perfect, then the most forceful experience is the best. But if not, you might need to balance the rest of the system to adapt. And one way is by having a less forceful or less frequent experience. Most systems will have some form of hybrid experience which behaves differently on easy parts of the problem than it does on hard parts of the problem. For example, by deleting messages that it’s certain are spam but informing on the messages that it’s not so sure about.