After publishing What Every Marketer Should Know Before They Build or Buy a Chatbot, we got a lot of questions about how to choose an AI-chatbot vendor. I get it. There’s a lot that goes into building a great marketing chatbot – or, frankly, any sort of system that uses conversational intelligence. Even when your IT folks have time to build a chatbot internally, there’s often copious amounts of training and testing you need to do. Not to mention the ongoing maintenance.
So, if you are looking for an outside provider to help you customize your chatbot, here are some things to look for.
EASY-TO-USE AND ACCURATE
Sounds very pedestrian, doesn’t it? I mention this because many of the best-looking chatbots right now are dumber than a box of rocks. Marketers often don’t recognize it because they’re so enamored with the plug-and-play drag-and-drop, use-it-immediately vendor pitches and demos.
When you’re looking at usability, be sure to look at how your team can use/customize it. Get an idea of how frequent the vendor upgrades/updates are. Ask what enhancements are coming down the pike. Take a deep dive into how your visitors will use it. One of the easiest ways to spot weak/bad chatbots? Asking questions where there are no answers is a good technique, but an even better one is seeing how it behaves when it’s in a loop. (A loop is a pattern that keeps repeating.)
Remember to look closely at how it routes people. You’re looking for maximum efficiency here. The routing needs to be tight. Otherwise, you’ll risk upsetting your visitors to the point where they fail and bail. Check out how problems are escalated and how seamless the process is. Figure out how leads/inquiries get to Sales. Deep dive into how the data is collected and when. A lot of the uber cheap chatbots don’t keep bounce data, the greeting data, or conversations where the intro fields are not completely filled out. (Yikes!)
Remember, a big part of user-friendliness in the eyes of your visitors will be clear, concise answers and, more importantly, the accuracy of those answers. Some of the newest chatbots have such impressive personalization that marketers forget to consider whether the answer is right. You can tell a lot about a system by the percentage of error returns and the error handling.
Please review all this “easy-to-use” stuff from a handheld device on a 3G wireless connection. Yes, it’s a garbage experience even on the best day, but it’s also best for evaluation.
CREATIVE AND BRANDING CAPABILITIES
What kind of branding options does the vendor offer? Can you name your chatbot? Are there any limitations to changing the design? Do they allow you to have endless prepared statements? (Some of the platforms do limit these, strangely enough.) Are you allowed to change the language and/or creative (copy/design) based on pre-determined metrics (ex: the country from which the person is coming in?)
Can you add images and videos? What about things like PDFs and instruction manuals? Can you attach them, or does everything need to be a link? Are there built-in inquiry forms and exit surveys? Can you easily send transcripts of the chat via email? Does the system integrate with your live chat software? Do they offer triggered emails/SMS as part of the system, or is it a separate integration? What does their Knowledge Base look like? I recommend asking to see a real-life, in-use example of a Knowledge Base before you choose a vendor.
DYNAMIC TARGETING OPTIONS
Ask the vendor what options they have for targeting – this includes targeting within the chatbot AND targeting the chatbot (or not) on your site itself.
Determine how easy it is NOT to show the chatbot to certain audiences. (Specific channels, for example.) How quickly and accurately can you customize your content to specific devices? See how easy it is to offer different content for new visitors vs. repeat visitors; how quickly the language changes when the user is from a different country; what the messaging says if you know the person has items in their shopping cart or has been on a particular product page for longer than usual; and so on. This is where our eyes can get bigger than our stomachs quite rapidly, and it’s still a good process to go through. Precise targeting capabilities can help your users have a much better experience.
One of the most critical things you’ll want to find out is the performance hit of the chatbot. I prefer to ask this question to other customers who use the product. Many of them will be willing to share the direct impact of their bot on speed and performance.
COMPREHENSIVE ANALYTICS/REPORTING
For me, a lack of quality analytics can be a dealbreaker regarding marketing stuff. I WANT ALL THE DATA. With Artificial Intelligence, it’s essential to have easy, reliable insights and access to the raw numbers to back them up in case you want to dig deeper. When choosing an outside vendor, get copies of their standard reports. Completely anonymized reports won’t be helpful to you, so try to get as “real” (detailed) data as possible.
After you’ve reviewed their entire stack of reporting, figure out what you’re missing and get a quote on how much the Custom Reporting will be. Some of the best vendors in the space charge an arm, a leg, and a kidney to customize anything. I’ve learned the hard way that getting your sticker shock upfront is better.
INTEGRATIONS
Leveraging your incoming real-time data is critical, so be sure to get the right team members involved when looking at the available integrations the vendor offers. Ask whether they provide open APIs. Figure out how/where you want the data to be accessible – your CDP, email, SMS, social media, call center, third-party extensions/packages, ad programs, SEO, and so on. Then figure out how it can get from one place to another, how you can audit that flow and its ease and reliability, and whether there are any ramifications, especially performance-related.
This is also an excellent time for you to decide whether you want AI data to go direct to another vendor/package/solution or if you want to be the central repository and distributor for all your data. (This is often referred to as the Hub and Spoke model.) Data from one AI-enabled tool to another can result in a tornado-meets-volcano situation, so you’ll want to ensure you have the proper structure set up before it starts flowing.
TRAINING AND DISRUPTION PROTOCOLS
For maximum success, you’re going to want to train your model; test it exhaustively, and then be able to disrupt it down the road. There are three significant parts of model testing, usability testing, accuracy testing, and routing testing. (Routing includes escalation.)
One of the best ways to know whether you’ve got a good vendor on the hook is to ask them a few simple questions. How is your model trained, and can we impact that for our unique instance if we use your software? What’s the process for disrupting the model if we want/need to?
Be sure to get their documentation on training and disruption. They may refer to disruption in many different ways. What you want to know is how you can rejigger something quickly and efficiently without scorching the earth (aka doing a complete rebuild.) Being transparent about if/how you can disrupt things separates the wheat from the chaff.
DYNAMIC TESTING
What kinds of testing capabilities does the platform have? What are their typical testing recommendations? How are the results presented? With AI/ML, your testing programs move from old-school A/B splits to high-powered, dynamic testing, so you must know how the tests are executed and reported. Find out if the software/vendor allows you to filter by channel, audience, product/service, category, type of visitor (repeat/new), and such. This is an area where you need to dig.
THE LEGAL STUFF — BIAS, PRIVACY, SECURITY, AND COMPLIANCE
I’ve grouped this all together, but it should not diminish how important this category is. Many vendors in this space are not explicit about their Bias, Privacy, Security, and Compliance standards (including Accessibility) and, frankly, ongoing efforts. If nothing else, it’s important to have proper documentation for your records. (This is helpful for any AI-related projects.)
SUPPORT TEAM
With any AI marketing vendor, it’s important to talk to the team you’ll be working with daily. Some of the most well-known vendors in the space have incredible sales teams and some of the weakest reps/teams. With new projects, especially involving machine learning and intelligent automation, you need a solid team. (Also, ask about internal turnover in the support positions you’ll deal with – AI projects are complicated enough without a revolving door of contacts that you’ll need to train again about your business.)
Need more tips about Choosing an AI/ML vendor? We’ve got you! Read How to Choose an AI Marketing Vendor That Best Fits Your Company here.
Have more questions about choosing a chatbot that’s right for you & your marketing efforts? Tweet @amyafrica or write info@eightbyeight.com.
This article is Part 3 in the Chatbot series. Read Part 1 and Part 2 here.
Catalogers and other offline marketers have many advantages when it comes to E-Commerce. 
