- Client: BMW Jorssen
- Services Provided: Conversational Interfaces
A swift and secure booking process with in-chat ItsMe authentication, image processing and OCR
Can we use a chatbot to improve the customer experience of booking a test drive?
Booking a test drive seems like a simple process, but it’s not. Once the customer completes his request in the web form, he has to be contacted by a sales rep in order to pick the most appropriate time. Once this is done, a confirmation of his appointment has to be sent and a few days before the appointment, he has to be reminded to bring the necessary documents to his appointment.
The sales rep has to manage the planning, taking into account the available cars and licence plates and leaving enough time to clean the cars in between test drives. Once the customer arrives at his appointment, he has to scan all the documents (drivers licence, insurance card, …) and perform a damage appraisal prior to the test drive.
This is very time-consuming for the employees, and from a customer point of view there’s other, more efficient ways to manage the booking process.
Crafting the conversation
Before starting the development of the conversational interface, we had to think about the different conversational flows and define the necessary technical integrations to reach the goal of the project: create a swift and secure booking process. During this phase we defined the different user journeys and mapped the workflows of the dealership’s employees to these journeys. Based on our findings we created an architecture for the chatbot.
We chose for the Azure platform because it offers us all the tools we need to design, build, test and deploy intelligent conversational interfaces. The architecture is based on these main components:
- Microsoft Bot Framework
- Microsoft LUIS as NLP
- Microsoft Custom Vision for Image recognition
- Outlook 365 for the calendar and planning integration
- Translator service for Date & Time awareness of the chatbot in Dutch
- Adaptive cards to create a user friendly visualisation of the available cars
- Blob storage
- Jquery datatables
- Google Cloud Vision for OCR
- ItsMe for user authentication
- DotNetCore and the Entity Framework
“We would immediately opt for this new and innovative way of booking test-drives and appointments when renewing our current systems. The chatbot could allow our customers to book test-drives and other appointments in a more efficient and fluent way, while allowing our employees to focus on customer service rather than manually filling documents and creating appointments.” – Steve Topff, BMW Jorssen
The first step in the development process was the implementation of an authentication option for the user. “How will we identify the user as securely as possible?” was the question we had to answer here. We agreed on implementing ItsMe, a Belgian identification service that uses a mobile application to let users sign-in into different applications (mobile banking, tax collection, …). ItsMe was never implemented in a chatbot before so we were very excited to get the scoop!
User information our chatbot receives through the ItsMe API.
Through the ItsMe API, we receive a unique user-id, his first- and last name, gender, e-mail, phone number and address information. The service offers us a 100% guarantee about the identity of the person booking the test drive. We now have everything we need to know from the customer and move to the next step in the booking process.
“The chatbot could definetly be a part of a new customer-centric IT environment. Thanks to the chatbot, our customers could have a user-friendly and intuitive virtual assistant that can help them find a suitable car and available timeslot for a test-drive. The intelligence behind the chatbot can help us with automatic planning and makes sure the driver sends us the correct documents.” – Geert Peeraer, BMW Jorssen
During the booking process, the user needs to send 2 documents: a copy of his driving license, and a car registration document. When the user sends his documents, we use Microsoft Custom Vision to verify that the image that was sent contains the right document. We trained our model with a test-set we generated ourselves, containing several car registrations and driving licenses.
Here you can see how well our model is currently performing. The left number shows the precision, this is the number that tells us how certain our model is is that it allocated the correct tag to an image. The second value is the recall score, showing how much of the image Custom Vision should recognize he actually recognizes.
In the example above, the user has sent an image of a driving license. We can see in this response that Custom Vision is 0,9987 or 99,87% sure that the image that was sent is actually a driving license. In our project the confidence score needs to be 80% or more before we let the user go to the next step.
When the correct document is recognized, we apply OCR to extract the name and first name of the driver. We do this to cross-check with the information we received through the ItsMe API. You can see the response we get from the OCR below.
As you can see this response consist of a list of text annotations. This is a list of all the different areas with the text that is found in our image. The description contains the text and the bounding poly contains the x and y position of the corner points of the rectangle that resembles the area where the text is found.
When the chatbot asks the user for a date, we use LUIS to parse the response from the user to an actual date. This was pretty straight forward because LUIS comes with a prebuilt entity DateTimeV2. The only downside is that our project is in Dutch, and LUIS doesn’t come with this prebuilt entity in Dutch. To counter this we make use of a translator service to translate the user response to English before sending it to LUIS. The response we get from LUIS sometimes needs some parsing. For example with the sentence “Within 5 weeks Thursday” we get the following response:
You can see that we receive 4 different dates in this response. So after some parsing and thanks to the translator service, we can define the following end date from the Dutch conversation: 25/07/2019. Once we send this through the planning algorithm, the user will receive an answer from the bot containing all available time slots for that day.
What if a customer abandons the conversation? Or what if the user fails to authenticate with ItsMe but has all the other documents at hand? This is where lead generation and follow-up by the dealership will be crucial.
When for some reason the user wants or has to stop the booking process, the bot will, depending on where in the flow the user tries to stop, ask the user for more information. Details like his first and last name and his e-mail address are collected, so the customer can be followed-up. These details, together with his chosen car, date and time will then be sent to the garage to notify the sales rep that a person tried to book a test drive but failed or quit. Every person trying to book a test drive is a possible buyer so we need to follow-up on these conversations. The dealership can then contact this person and book the test drive by phone.
“The AI Engineers at Arinti did a great job in understanding our current processes and requirements and translating them into an innovative Minimum Viable Product in only a few weeks time.” – Nick Haverals, AMB Holding
Our project also included a back-end portal for the administrators, where they could find a calendar with upcoming test drives, manually register or adjust the booked test-drives, manage the test-drive-fleet and customize the confirmation e-mails and messages for customers.