ESSMArT way to manage customer requests

Abstract

Quality and market acceptance of software products is strongly influenced by responsiveness to customer requests. Once a customer request is received, a decision must be made whether to escalate it to the development team. Once escalated, the ticket must be formulated as a development task and assigned to a developer. To make the process more efficient and reduce the time between receiving and escalating the customer request, we aim to automate the complete customer request management process. We propose a holistic method called ESSMArT. The method performs text summarization, predicts ticket escalation, creates the ticket’s title and content, and ultimately assigns the ticket to an available developer. We began evaluating the method through an internal assessment of 4114 customer tickets from Brightsquid’s secure health care communication platform - Secure-Mail. Next, we conducted an external evaluation of the usefulness of the approach and concluded that: i) supervised learning based on context specific data performs best for extractive summarization; ii) Random Forest trained on a combination of conversation and extractive summarization works best for predicting escalation of tickets, with the highest precision (of 0.9) and recall (of 0.55). Through external evaluation, we furthermore found that ESSMArT provides suggestions that are 71% aligned with human ones. Applying the prototype implementation to 315 customer requests resulted in an average time reduction of 9.2 min per request. ESSMArT helps to make ticket management faster and with reduced effort for human experts. We conclude that ESSMArT not only expedites ticket management, but furthermore reduces human effort. ESSMArT can help Brightsquid to (i) minimize the impact of staff turnover and (ii) shorten the cycle from an issue being reported to a developer being assigned to fix it.

Appendices

Appendix 1 - Illustrative Example for Comparison of Processes

In this appendix we provide an anonymized Brightsquid customer support example to compare the traditional process of managing customer requests with the ESSMArT process.

1.1 Traditional Process of Managing Customer Requests at Brightsquid

Customers report issues to Brightsquid via telephone, chat or email, and issues are recorded in Zendesk. In this example, illustrated in Figure 13 below, the customer (Bob) contacts the CRM team, concerned about the management of secure messages in a team environment. The CRM agent (Alice) elaborates on the problem and provides a summary for the customer’s approval. Once Bob approves the description, Alice decides whether to escalate the ticket or resolve it on her own. If unable to determine a solution, Alice escalates the issue and informs Bob. The CRM manager (Carol), who is responsible for resolving or further escalating issues to the development project manager, reads through the ticket, expands upon the description as necessary, and escalates the issue to the project manager. The project manager (Erin) defines the ticket’s priority and assigns it to the most appropriate developer for resolution.

Fig. 13

Process in BS showing the conversational structure of a customer request at Bright- Squid. The underlined sentences have been chosen by our annotators to train machine learning based summarization techniques

1.2 Managing Customer Requests with ESSMArT

When Bob’s request is received, Alice elaborates the description further with the customer, as above. ESSMArT consequently summarizes the conversation as below:

Zendesk ticket (summary of the customer request)

John emailed me and wanted a copy of the a message note faxed to him. I sent John back a note, with the message note attached. He sends me back a thank you. Once I have deleted my note on Mr. Smith, everyone else on the Yellow Team still has that email. Unless I specifically tell everyone to delete it or unless the rest of the Yellow Team goes into the EMR and looks into John’s chart if it’s done, no one has any way of knowing the task has been dealt with. Is there a way to delete that task from everyone’s inbox?

ESSMArT then escalates the ticket to Carol, who receives a notification comprising the ESSMArT summary and escalation recommendation. If Carol agrees, ESSMArT creates a Jira development ticket by assigning a title using abstractive summarization (Check Figure 4 for a detailed example), assigning ticket priority, assigning a specific developer, and notifying Erin.

Jira ticket (development tickets)

Title: Delete everyone task in the EMR inbox Content: In the EMR system; a doctor had emailed a doctor and want a copy of a message note faxed to him. Staff member send back a note, with the message attached. The doctor sends back a thank you. Once doctor delete his note on patient, everyone else on the team still has that email. Unless the doctor specifically tell everyone to delete it or unless the rest of the team goes into the EMR and looks into the doctor’s chart to see if it’s done, no one has any way of knowing the task has been dealt with. is there a way to delete that task from everyone’s inbox? Priority: Major Assignee: Jane Doe

Once Erin agrees, the ESSMArT ticket is added to the Jira backlog. Screenshots of this example in ESSMArT were previously presented in Figure 9.

Appendix 2 - Confusion Matrices

In Tables 5, 7, and 8, we presented precision, recall, and F1-score of three state-of-the art classifiers (Naive Bayes, SVM, and Random Forest) to predict ticket escalation, prioritize the escalated tickets, respectively assign them to a developer. These are the results of ten times of running cross validation and we provided the averages. In addition to that, we provide the aggregated confusion matrix for the classifiers with the best performance and for all the three types of prediction. These confusion matrices are presented below.

Confusion matrix for ticket escalation

		Predicted
		Yes	No
Actual	Yes No	218 103	24 297

The False-Positives demonstrate that the classifier mistakenly considered a ticket is escalated while in fact it has not been escalated. The False-Negatives indicate the tickets that should have been escalated but were not detected by the classifier. The False-positives may add additional effort to the development team while the False- Negatives may result in customer’s dissatisfaction by not properly addressing their problem.

Confusion matrix for ticket assignment

*Aggregated across eight different classes

		Predicted
		Yes	No
Actual	Yes No	218 26	40 41

The False-Positives may result in more work for developers as the ticket may be assigned to a developer with not enough expertise. False-Negatives may result in more time and effort to fix the ticket as the developer with less expertise would handle the ticket.

Confusion matrix for ticket prioritization

*Aggregated across five different classes

		Predicted
		Yes	No
Actual	Yes No	133 44	49 101

The False-Positives and Negatives would delay fixing some of the important customers’ concerns prioritize lower or stuck behind lower priority tickets.

Appendix 3 - Survey Questions

The user study to evaluate ESSMArT was conducted in two main parts: First, using the prototype tool of ESSMArT using the data from Brightsquid for evaluation purpose, and second, asking questions to understand the perception of participants about the usability of ESSMArT in practice. In this appendix, we present the five questions raised to understand the perception of users about ESSMArT.

Figure 11 shows the results of this survey and the perception of CRM experts and project managers. The sample of questions and the screenshots of the used prototype were presented in Fig. 8 respectively Fig. 9.

• 1. How understandable did you find the results of ESSMArT?

  5- Very understandable 4- understandable 3- Somewhat understandable but slightly ambiguous 2- Somewhat understandable but mostly ambiguous 1- Not understandable

• 2. How likely you would use ESSMArT in practice?

  5- Definitely 4- Very likely 3- Maybe 2- not likely 1- Definitely not

• 3. To what extent you trust and rely on the ESSMArT results?

  5- Totally trust it 4- Trust it 3- Neutral 2- Not trust it 1- Not trust it at all

• 4. To what extent are you agree that ESSMArT reduces the time for deciding on a change request?

  5- Strongly agree 4- Agree 3- Neutral 2- Disagree 1- Strongly disagree

• 5. To what extent are you agree that ESSMArT reduces the time needed for CRM/PM tasks?

  5- Strongly agree 4- Agree 3- Neutral 2- Disagree 1- Strongly disagree