The Chatbot Dilemma

I’m in the process of gathering all my written pieces over the years onto my own website. However, one piece stands out as particularly poignant given how GPT has simplified the creation of chatbots compared to seven years ago. It’s quite an intriguing exercise to reflect on what’s evolved and what’s remained the same.

There’s a lot of discussion going on on the web about whether chatbots are really the next big thing or just a short-term fad. Since I’m the co-founder of Ada, a personal assistant for real estate, I’m obviously biased — but don’t be mistaken, the dilemma is a huge topic within our team.

Certainly, artificial intelligence will change the world — no question about it. But it’s still unclear whether the technology for building intelligent agents is ready for mass markets right now, and whether conversational interfaces are a good choice given the current features of our smart devices.

Last month, I wrote that developing a chatbot is a fairly cost-effective approach to building a prototype using the Wizard of Oz technique.

I stand by what I said. Serving first customers with a chatbot will provide every early-stage startup with a tremendous amount of learning, feedback and market insight — but it’s not the entire story.

As you go down the road, the technology will start costing you a lot more than it did when you employed human operators.

Why? AI development requires hiring great engineers and researchers, having broad knowledge about current state of the art, access to cutting-edge tools — and sometimes inventing new ones as you go.

They aren’t cheap anymore. (Not right now, at least.)

Should chatbots be just better, simpler apps?

Dan Grover, a product manager at WeChat, raised more doubts with his influential post, “Bots won’t replace apps. Better apps will replace apps.” He noticed that while the chatbot trend emerged in the West as a response to the popularity of WeChat and Line in Asia, both of these apps took a very different approach to the topic.

The fact is, WeChat doesn’t employ AI-powered chatbots. They use HTML5 menus and webview interfaces to let partners build simple apps within the chats and use the WeChat platform to streamline identity persistance, payments, offline storage, and data sharing. No intelligent agents involved — just a simple, yet powerful, app-within-an-app.

That’s a bit puzzling since Silicon Valley essentially used WeChat as a proof of concept — but it wasn’t even the same concept.

So, are we on a hype train?

If there’s no proof of concept yet, then the bot business is just the new Wild West for technology and products.

To restrain it, we need to lay some groundwork.

• Just as we have product metrics such as DAU and business metrics like LTV to keep our products and business models in check, we need new exhaustive metrics to determine whether building a bot is really a logical choice or just a caprice.
• Similarly, we need new UX frameworks to guide our decision making based on the metrics we’ll invent. These frameworks should help answer a single question: if a chatbot is indeed a good choice, howshould we build it?

These metrics will help us understand the business of making intelligent agents and personal assistants just as we understand building new SaaS businesses or creating new mobile apps today.

If we put things this way, 2016 is nothing more than a new 2006. Maybe time really is just a flat circle.

Enter cognitive ergonomics.

Let me give you an example of what I had in mind.

Last month, I read a great piece on cognitive ergonomics and, particularly, cognitive efficiency:

Cognitive Efficiency […] is a measure of how much cognitive effort is saved through the interface. For example, compare […] alternatives to getting calendar information for something 2 weeks away. In the first, a user has to exit a messaging platform, open a calendar, scroll to the right day, and find the event on the calendar. In the second example, a user can ask […], right from a messaging platform, a question about the day, or the event, thus saving several steps.— Rob May, Cognitive Ergonomics and Chatbots

Measuring cognitive efficiency is a great example of a user experience metric that we can use to create an extremely simple framework.

1. Given a task, create two prototypes that help a user solve the task: one bot-based and one (pretty standard) app-based.
2. Measure cognitive efficiency for each prototype.
3. Choose the prototype that solves the task better.

You can even mix the bot-based approach with the app-based approach — just like we do at Ada. We’re building a hybrid product that switches between an SMS chatbot and a mobile web app based on the task we need to solve. As we found out, there are three tasks at which bots excel in our case:

• Bots are great at onboarding. Bots are an excellent acquisition channel and since they don’t require the user to learn new UI, they help maximize retention after registration. The most notable example is perhaps the Slackbot.
• Bots facilitate discovery. An intelligent agent can help you a lot when you don’t really what what you’re looking for. For example, when users hire Ada to find a new apartment to rent, we can help them decide on their criteria based on local market information and real estate trends.
• Bots can use SMS notifications naturally. Text messages have a 98% open rate which results in tons of engagement.

There’s still a lot to be done, of course, and the article above is by no means a full solution — but I hope it answers some questions regarding chatbots, intelligent agents, and conversational interfaces.