1 Introduction↩︎

Conversational agents have long been studied in the context of psychotherapy, going back to chatbots such as ELIZA [1] and PARRY [2]. Research in modeling such dialogue has largely sought to simulate a participant in the conversation.

In this paper, we argue for modeling dialogue observers instead of participants, and focus on psychotherapy. An observer could help an ongoing therapy session in several ways. First, by monitoring fidelity to therapy standards, a helper could guide both veteran and novice therapists towards better patient outcomes. Second, rather than generating therapist utterances, it could suggest the type of response that is appropriate. Third, it could alert a therapist about potentially important cues from a patient. Such assistance would be especially helpful in the increasingly prevalent online or text-based counseling services.¹

We ground our study in a style of therapy called Motivational Interviewing [3], [4], which is widely used for treating addiction-related problems. To help train therapists, and also to monitor therapy quality, utterances in sessions are annotated using a set of behavioral codes called Motivational Interviewing Skill Codes [5]. Table ¿tbl:tbl:misc? shows standard therapist and patient (, client) codes with examples. Recent NLP work [6]–[8], [9] inter alia has studied the problem of using MISC to assess completed sessions. Despite its usefulness, automated post hoc MISC labeling does not address the desiderata for ongoing sessions identified above; such models use information from utterances yet to be said. To provide real-time feedback to therapists, we define two complementary dialogue observers:

1. Categorization: Monitoring an ongoing session by predicting MISC labels for therapist and client utterances as they are made.

2. Forecasting: Given a dialogue history, forecasting the MISC label for the next utterance, thereby both alerting or guiding therapists.

Via these tasks, we envision a helper that offers assistance to a therapist in the form of MISC labels.

We study modeling challenges associated with these tasks related to:

We develop neural models that address these challenges in this domain.

Experiments show that our proposed models outperform baselines by a large margin. For the categorization task, our models even outperform previous session-informed approaches that use information from future utterances. For the more difficult forecasting task, we show that even without having access to an utterance, the dialogue history provides information about its MISC label. We also report the results of an ablation study that shows the impact of the various design choices.². In summary, in this paper, we

2 Background and Motivation↩︎

Motivational Interviewing (MI) is a style of psychotherapy that seeks to resolve a client’s ambivalence towards their problems, thereby motivating behavior change. Several meta-analyses and empirical studies have shown the high efficacy and success of MI in psychotherapy [10]–[12]. However, MI skills take practice to master and require ongoing coaching and feedback to sustain [13]. Given the emphasis on using specific types of linguistic behaviors in MI (, open questions and reflections), fine-grained behavioral coding plays an important role in MI theory and training.

Motivational Interviewing Skill Codes (MISC, table ¿tbl:tbl:misc?) is a framework for coding MI sessions. It facilitates evaluating therapy sessions via utterance-level labels that are akin to dialogue acts [14], [15], and are designed to examine therapist and client behavior in a therapy session.³

As Table ¿tbl:tbl:misc? shows, client labels mark utterances as discussing changing or sustaining problematic behavior (Ct and St, respectively) or being neutral (Fn). Therapist utterances are grouped into eight labels, some of which (Res, Rec) correlate with improved outcomes, while MI non-adherent (Min) utterances are to be avoided. MISC labeling was originally done by trained annotators performing multiple passes over a session recording or a transcript. Recent NLP work speeds up this process by automatically annotating a completed MI session [6]–[8].

Instead of providing feedback to a therapist after the completion of a session, can a dialogue observer provide online feedback? While past work has shown the helpfulness of post hoc evaluations of a session, prompt feedback would be more helpful, especially for MI non-adherent responses. Such feedback opens up the possibility of the dialogue observer influencing the therapy session. It could serve as an assistant that offers suggestions to a therapist (novice or veteran) about how to respond to a client utterance. Moreover, it could help alert the therapist to potentially important cues from the client (specifically, Ct or St).

3 Task Definitions↩︎

In this section, we will formally define the two NLP tasks corresponding to the vision in §2 using the conversation in table ¿tbl:tbl:example? as a running example.

Suppose we have an ongoing MI session with utterances \(u_1, u_2,\cdots, u_n\): together, the dialogue history \(H_n\). Each utterance \(u_i\) is associated with its speaker \(s_i\), either C (client) or T (therapist). Each utterance is also associated with the MISC label \(l_i\), which is the object of study. We will refer to the last utterance \(u_n\) as the anchor.

We will define two classification tasks over a fixed dialogue history with \(n\) elements — categorization and forecasting. As the conversation progresses, the history will be updated with a sliding window. Since the therapist and client codes share no overlap, we will design separate models for the two speakers, giving us four settings in all.

Task 1: Categorization. The goal of this task is to provide real-time feedback to a therapist during an ongoing MI session. In the running example, the therapist’s confrontational response in the third utterance is not MI adherent (Min); an observer should flag it as such to bring the therapist back on track. The client’s response, however, shows an inclination to change their behavior (Ct). Alerting a therapist (especially a novice) can help guide the conversation in a direction that encourages it.

In essence, we have the following real-time classification task: Given the dialogue history \(H_n\) which includes the speaker information, predict the MISC label \(l_n\) for the last utterance \(u_n\).

The key difference from previous work in predicting MISC labels is that we are restricting the input to the real-time setting. As a result, models can only use the dialogue history to predict the label, and in particular, we can not use models such as a conditional random field or a bi-directional LSTM that need both past and future inputs.

Task 2: Forecasting. A real-time therapy observer may be thought of as an expert therapist who guides a session with suggestions to the therapist. For example, after a client discloses their recent drug use relapse, a novice therapist may respond in a confrontational manner (which is not recommended, and hence coded Min). On the other hand, a seasoned therapist may respond with a complex reflection (Rec) such as “Sounds like you really wanted to give up and you’re unhappy about the relapse.” Such an expert may also anticipate important cues from the client. The forecasting task seeks to mimic the intent of such a seasoned therapist: Given a dialogue history \(H_n\) and the next speaker’s identity \(s_{n+1}\), predict the MISC code \(l_{n+1}\) of the yet unknown next utterance \(u_{n+1}\).

The MISC forecasting task is a previously unstudied problem. We argue that forecasting the type of the next utterance, rather than selecting or generating its text as has been the focus of several recent lines of work [16]–[18], allows the human in the loop (the therapist) the freedom to creatively participate in the conversation within the parameters defined by the seasoned observer, and perhaps even rejecting suggestions. Such an observer could be especially helpful for training therapists [19]. The forecasting task is also related to recent work on detecting antisocial comments in online conversations [20] whose goal is to provide an early warning for such events.

4 Models for MISC Prediction↩︎

Modeling the two tasks defined in §3 requires addressing four questions:

Many recent advances in neural networks can be seen as plug-and-play components. To facilitate the comparative study of models, we will describe components that address the above questions. In the rest of the paper, we will use boldfaced terms to denote vectors and matrices and small caps to denote component names.

4.1 Encoding Dialogue↩︎

Since both our tasks are classification tasks over a dialogue history, our goal is to convert the sequence of utterences into a single vector that serves as input to the final classifier.

We will use a hierarchical recurrent encoder [21], [22], [23] and others to encode dialogues, specifically a hierarchical gated recurrent unit (HGRU) with an utterance and a dialogue encoder. We use a bidirectional GRU over word embeddings to encode utterances. As is standard, we represent an utterance \(u_i\) by concatenating the final forward and reverse hidden states. We will refer to this utterance vector as \(\boldsymbol{v}_i\). Also, we will use the hidden states of each word as inputs to the attention components in §4.2. We will refer to such contextual word encoding of the \(j^{th}\) word as \(\boldsymbol{v}_{ij}\). The dialogue encoder is a unidirectional GRU that operates on a concatenation of utterance vectors \(\boldsymbol{v}_i\) and a trainable vector representing the speaker \(s_i\).⁴ The final state of the GRU aggregates the entire dialogue history into a vector \(\boldsymbol{H}_n\).

The HGRU skeleton can be optionally augmented with the word and dialogue attention described next. All the models we will study are two-layer MLPs over the vector \(\boldsymbol{H}_n\) that use a ReLU hidden layer and a softmax layer for the outputs.

4.2 Word-level Attention↩︎

Certain words in the utterance history are important to categorize or forecast MISC labels. The identification of these words may depend on the utterances in the dialogue. For example, to identify that an utterance is a simple reflection (Res) we may need to discover that the therapist is mirroring a recent client utterance; the example in table ¿tbl:tbl:misc? illustrates this. Word attention offers a natural mechanism for discovering such patterns.

We can unify a broad collection of attention mechanisms in NLP under a single high level architecture [24]. We seek to define attention over the word encodings \(\boldsymbol{v}_{ij}\) in the history (called queries), guided by the word encodings in the anchor \(\boldsymbol{v}_{nk}\) (called keys). The output is a sequence of attention-weighted vectors, one for each word in the \(i^{th}\) utterance. The \(j^{th}\) output vector \(\boldsymbol{a}_j\) is computed as a weighted sum of the keys: \[\boldsymbol{a}_{ij} = \sum_{k} \alpha^{k}_{j} \boldsymbol{v}_{nk} \label{eq:att95sum}\tag{1}\] The weighting factor \(\alpha^k_j\) is the attention weight between the \(j^{th}\) query and the \(k^{th}\) key, computed as \[\label{eq:att95weight} \alpha^{k}_{j} = \frac{\exp\left(f_{m}(\boldsymbol{v}_{nk}, \boldsymbol{v}_{ij})\right)}{\sum_{j^{\prime}} \exp\left(f_{m}(\boldsymbol{v}_{nk}, \boldsymbol{v}_{ij^\prime})\right)}\tag{2}\] Here, \(f_m\) is a match scoring function between the corresponding words, and different choices give us different attention mechanisms.

Finally, a combining function \(f_{c}\) combines the original word encoding \(\boldsymbol{v}_{ij}\) and the above attention-weighted word vector \(\boldsymbol{a}_{ij}\) into a new vector representation \(\boldsymbol{z}_{ij}\) as the final representation of the query word encoding: \[\boldsymbol{z}_{ij}= f_{c}(\boldsymbol{v}_{ij}, \boldsymbol{a}_{ij})\]

The attention module, identified by the choice of the functions \(f_m\) and \(f_c\), converts word encodings in each utterance \(\boldsymbol{v}_{ij}\) into attended word encodings \(\boldsymbol{z}_{ij}\). To use them in the HGRU skeleton, we will encode them a second time using a BiGRU to produce attention-enhanced utterance vectors. For brevity, we will refer to these vectors as \(\boldsymbol{v}_i\) for the utterance \(u_i\). If word attention is used, these attended vectors will be treated as word encodings.

To complete this discussion, we need to instantiate the two functions. We use two commonly used attention mechanisms: BiDAF [25] and gated matchLSTM [26]. For simplicity, we replace the sequence encoder in the latter with a BiGRU and refer to it as GMGRU. Table ¿tbl:tbl:word95att? shows the corresponding definitions of \(f_{c}\) and \(f_{m}\). We refer the reader to the original papers for further details. In subsequent sections, we will refer to the two attended versions of the HGRU as BiDAF\(^H\) and GMGRU\(^H\).

4.3 Utterance-level Attention↩︎

While we assume that the history of utterances is available for both our tasks, not every utterance is relevant to decide a MISC label. For categorization, the relevance of an utterance to the anchor may be important. For example, a complex reflection (Rec) may depend on the relationship of the current therapist utterance to one or more of the previous client utterances. For forecasting, since we do not have an utterance to label, several previous utterances may be relevant. For example, in the conversation in Table ¿tbl:tbl:example?, both \(u_2\) and \(u_4\) may be used to forecast a complex reflection.

To model such utterance-level attention, we will employ the multi-head, multi-hop attention mechanism used in Transformer networks [27]. As before, due to space constraints, we refer the reader to the original work for details. We will use the \((\boldsymbol{Q}, \boldsymbol{K}, \boldsymbol{V})\) notation from the original paper here. These matrices represent a query, key and value respectively. The multi-head attention is defined as: \[\label{eq:multihead95attention} {\small \text{Multihead}(\boldsymbol{Q},\boldsymbol{K},\boldsymbol{V}) = [\text{head}_{1};\cdots; \text{head}_{h}]\boldsymbol{W}^{O}}\tag{3}\] \[\text{head}_{i} = \text{softmax}\left(\frac{\boldsymbol{Q}\boldsymbol{W}^{Q}_{i}\left(\boldsymbol{K}\boldsymbol{W}^{K}_{i}\right)^T}{\sqrt{d_{k}}}\right)\boldsymbol{V}\boldsymbol{W}^{V}_{i}\] The \(\boldsymbol{W}_i\)’s refer to projection matrices for the three inputs, and the final \(\boldsymbol{W}^o\) projects the concatenated heads into a single vector.

The choices of the query, key and value defines the attention mechanism. In our work, we compare two variants: anchor-based attention, and self-attention. The anchor-based attention is defined by \(Q = [\boldsymbol{v}_{n}]\) and \(K=V=[\boldsymbol{v}_{1} \cdots \boldsymbol{v}_{n}]\). Self-attention is defined by setting all three matrices to \([\boldsymbol{v}_{1} \cdots \boldsymbol{v}_{n}]\). For both settings, we use four heads and stacking them for two hops, and refer to them as \(\textcolor{dkgray}{\mathrm{\small self_{42}}}\) and \(\textcolor{dkgray}{\mathrm{\small anchor_{42}}}\).

4.4 Addressing Label Imbalance↩︎

From Table ¿tbl:tbl:misc?, we see that both client and therapist labels are imbalanced. Moreover, rarer labels are more important in both tasks. For example, it is important to identify Ct and St utterances. For therapists, it is crucial to flag MI non-adherent (Min) utterances; seasoned therapists are trained to avoid them because they correlate negatively with patient improvements. If not explicitly addressed, the frequent but less useful labels can dominate predictions.

To address this, we extend the focal loss [28] to the multiclass case. For a label \(l\) with probability produced by a model \(p_t\), the loss is defined as \[\label{eq:focal} \text{FL}(p_{t}) = -\alpha_{t} {(1 -p_{t})}^{\gamma} \log(p_{t})\tag{4}\] In addition to using a label-specific balance weight \(\alpha_t\), the loss also includes a modulating factor \({(1-p_{t})}^{\gamma}\) to dynamically downweight well-classified examples with \(p_{t}\gg0.5\). Here, the \(\alpha_t\)’s and the \(\gamma\) are hyperparameters. We use FL as the default loss function for all our models.

5 Experiments↩︎

The original psychotherapy sessions were collected for both clinical trials and Motivational Interviewing dissemination studies including hospital settings [29], outpatient clinics [30], college alcohol interventions [31]–[34]. All sessions were annotated with the Motivational Interviewing Skills Codes (MISC) [35]. We use the train/test split of [6], [36] to give 243 training MI sessions and 110 testing sessions. We used 24 training sessions for development. As mentioned in §2, all our experiments are based on the MISC codes grouped by [7].

5.1 Preprocessing and Model Setup↩︎

An MI session contains about 500 utterances on average. We use a sliding window of size \(N=8\) utterances with padding for the initial ones. We assume that we always know the identity of the speaker for all utterances. Based on this, we split the sliding windows into a client and therapist windows to train separate models. We tokenized and lower-cased utterances using spaCy [37]. To embed words, we concatenated 300-dimensional Glove embeddings [38] with ELMo vectors [39]. The appendix details the model setup and hyperparameter choices.

5.2 Results↩︎

Best Models. Our goal is to discover the best client and therapist models for the two tasks. We identified the following best configurations using \(\text{F}_{1}\) score on the development set:

1. Categorization: For client, the best model does not need any word or utterance attention. For the therapist, it uses GMGRU\(^H\) for word attention and anchor\(_{42}\) for utterance attention. We refer to these models as \(\mathcal{C}_C\) and \(\mathcal{C}_T\) respectively

2. Forecasting: For both client and therapist, the best model uses no word attention, and uses self\(_{42}\) utterance attention. We refer to these models as \(\mathcal{F}_C\) and \(\mathcal{F}_T\) respectively.

Here, we show the performance of these models against various baselines. The appendix gives label-wise precision, recall and \(\text{F}_{1}\) scores.

Results on Categorization. Tables ¿tbl:tbl:main95rst95c95categorizing? and ¿tbl:tbl:main95rst95t95categorizing? show the performance of the \(\mathcal{C}_C\) and \(\mathcal{C}_T\) models and the baselines. For both therapist and client categorization, we compare the best models against the same set of baselines. The majority baseline illustrates the severity of the label imbalance problem. [7], \(\text{BiGRU}_{\text{generic}}\), [36] and [6] are the previous published baselines. The best results of previous published baselines are underlined. The last row \(\Delta\) in each table lists the changes of our best model from them. \(\text{BiGRU}_{\text{ELMo}}\), \(\text{CONCAT}^{C}\), \(\text{GMGRU}^{H}\) and \(\text{BiDAF}^{H}\) are new baselines we define below.

The first set of baselines (above the line) do not encode dialogue history and use only the current utterance encoded with a BiGRU. The work of [7] falls in this category, and uses a 100-dimensional domain-specific embedding with weighted cross-entropy loss. Previously, it was the best model in this class. We also re-implemented this model to use either ELMo or Glove vectors with focal loss.⁵

The second set of baselines (below the line) are models that use dialogue context. Both [36] and [6] use well-studied linguistic features and then tagging the current utterance with both past and future utterance with CRF and MEMM, respectively. To study the usefulness of the hierarchical encoder, we implemented a model that uses a bidirectional GRU over a long sequence of flattened utterance. We refer to this as \(\text{CONCAT}^{C}\). This model is representative of the work of [9], but was reimplemented to take advantage of ELMo.

For categorizing client codes, \(\text{BiGRU}_{\text{ELMo}}\) is a simple but robust baseline model. It outperforms the previous best no-context model by more than 2 points on macro \(\text{F}_{1}\). Using the dialogue history, the more sophisticated model \(\mathcal{C}_{C}\) further gets 1 point improvement. Especially important is its improvement on the infrequent, yet crucial labels Ct and St. It shows a drop in the \(\text{F}_{1}\) on the Fn label, which is essentially considered to be an unimportant, background class from the point of view of assessing patient progress. For therapist codes, as the highlighted numbers in Table ¿tbl:tbl:main95rst95t95categorizing? show, only incorporating GMGRU-based word-level attention, \(\text{GMGRU}^{H}\) has already outperformed many baselines, our proposed model \(\mathcal{F}_{T}\) which uses both GMGRU-based word-level attention and anchor-based multi-head multihop sentence-level attention can further achieve the best overall performance. Also, note that our models outperform approaches that take advantage of future utterances.

For both client and therapist codes, concatenating dialogue history with \(\text{CONCAT}^{C}\) always performs worse than the hierarchical method and even the simpler \(\text{BiGRU}_{\text{ELMo}}\).

Results on Forecasting. Since the forecasting task is new, there are no published baselines to compare against. Our baseline systems essentially differ in their representation of dialogue history. The model \(\text{CONCAT}^{F}\) uses the same architecture as the model \(\text{CONCAT}^{C}\) from the categorizing task. We also show comparisons to the simple HGRU model and the GMGRU\(^H\) model that uses a gated matchGRU for word attention.⁶

Tables ¿tbl:tbl:main95rst95forecast? (a,b) show our forecasting results for client and therapist respectively. For client codes, we also report the Ct and St performance on the development set because of their importance. For the therapist codes, we also report the recall@3 to show the performance of a suggestion system that displayed three labels instead of one. The results show that even without an utterance, the dialogue history conveys signal about the next MISC label. Indeed, the performance for some labels is even better than some categorization baseline systems. Surprisingly, word attention (\(\text{GMGRU}^{H}\)) in Table ¿tbl:tbl:main95rst95forecast? did not help in forecasting setting, and a model with the self\(_{42}\) utterance attention is sufficient. For the therapist labels, if we always predicted the three most frequent labels (Fa, Gi, and Res), the recall@3 is only 67.7, suggesting that our models are informative if used in this suggestion-mode.

6 Analysis and Ablations↩︎

This section reports error analysis and an ablation study of our models on the development set. The appendix shows a comparison of pretrained domain-specific ELMo/glove with generic ones and the impact of the focal loss compared to simple or weighted cross-entropy.

6.1 Label Confusion and Error Breakdown↩︎

Figure 1 shows the confusion matrix for the client categorization task. The confusion between Fn and Ct/St is largely caused by label imbalance. There are 414 Ct examples that are predicted as St and 391 examples vice versa. To further understand their confusion, we selected 100 of each for manual analysis. We found four broad categories of confusion, shown in Table ¿tbl:tbl:c95client95errors?.

The first category requires more complex reasoning than just surface form matching. For example, the phrase indicates that the client is very confident about changing behavior; the phrase indicates, in this context, that the client drinks or smokes after work. We also found that the another frequent source of error is incomplete information. In a face-to-face therapy session, people may use concise and effient verbal communication, with guestures and other body language conveying information without explaining details about, for example, coreference. With only textual context, it is difficult to infer the missing information. The third category of errors is introduced when speech is transcribed into text. The last category is about ambivalent speech. Discovering the real attitude towards behavior change behind such utterances could be difficult, even for an expert therapist.

Figures 1 and 2 show the label confusion matrices for the best categorization models. We will examine confusions that are not caused purely by a label being frequent. We observe a common confusion between the two reflection labels, Rec and Res. Compared to the confusion matrix from [7], we see that our models show much-decreased confusion here. There are two reason for this confusion persisting. First, the reflections may require a much longer information horizon. We found that by increasing the window size to 16, the overall reflection results improved. Second, we need to capture richer meaning beyond surface word overlap for Res. We found that complex reflections usually add meaning or emphasis to previous client statements using devices such as analogies, metaphors, or similes rather than simply restating them. Closed questions (Quc) and simple reflections (Res) are known to be a confusing set of labels. For example, an utterance like may be both. Giving information (Gi) is easily confused with many labels because they relate to providing information to clients, but with different attitudes. The MI adherent (Mia) and non-adherent (Min) labels may also provide information, but with supportive or critical attitude that may be difficult to disentangle, given the limited number of examples.

6.2 How Context and Attention Help?↩︎

We evaluated various ablations of our best models to see how changing various design choices changes performance. We focused on the context window size and impact of different word level and sentence level attention mechanisms. Tables ¿tbl:tbl:rst95cxt95client? and ¿tbl:tbl:rst95cxt95therapist? summarize our results.

History Size. Increasing the history window size generally helps. The biggest improvements are for categorizing therapist codes (Table ¿tbl:tbl:rst95cxt95therapist?), especially for the Res and Rec. However, increasing the window size beyond 8 does not help to categorize client codes (Table ¿tbl:tbl:rst95cxt95client?) or forecasting (in appendix).

Word-level Attention. Only the model \(\mathcal{C}_{T}\) uses word-level attention. As shown in Table ¿tbl:tbl:rst95cxt95therapist?, when we remove the word-level attention from it, the overall performance drops by 3.4 points, while performances of Res and Rec drop by 3.3 and 5 points respectively. Changing the attention to BiDAF decreases performance by about 2 points (still higher than the model without attention).

Sentence-level Attention. Removing sentence attention from the best models that have it decreases performance for the models \(\mathcal{C}_T\) and \(\mathcal{F}_T\) (in appendix). It makes little impact on the \(\mathcal{F}_C\), however. Table ¿tbl:tbl:rst95cxt95client? shows that neither attention helps categorizing clients codes.

6.3 Can We Suggest Empathetic Responses?↩︎

Our forecasting models are trained on regular MI sessions, according to the label distribution on Table ¿tbl:tbl:misc?, there are both MI adherent or non-adherent data. Hence, our models are trained to show how the therapist usually respond to a given statement.

To show whether our model can mimic good MI policies, we selected 35 MI sessions from our test set which were rated 5 or higher on a 7-point scale empathy or spirit. On these sessions, we still achieve a recall@3 of 76.9, suggesting that we can learn good MI policies by training on all therapy sessions. These results suggest that our models can help train new therapists who may be uncertain about how to respond to a client.

7 Conclusion↩︎

We addressed the question of providing real-time assistance to therapists and proposed the tasks of categorizing and forecasting MISC labels for an ongoing therapy session. By developing a modular family of neural networks for these tasks, we show that our models outperform several baselines by a large margin. Extensive analysis shows that our model can decrease the label confusion compared to previous work, especially for reflections and rare labels, but also highlights directions for future work.

Acknowledgments↩︎

The authors wish to thank the anonymous reviewers and members of the Utah NLP group for their valuable feedback. This research was supported by an NSF Cyberlearning grant (#1822877) and a GPU gift from NVIDIA Corporation.

8 Appendix↩︎

8.0.0.1 Different Clustering Strategies for MISC

The original MISC description of [5] included 28 labels (9 client, 19 therapist). Due to data scarcity and label confusion, some labels were merged into a coarser set. [36] retain 6 original labels Fa, Gi, Quc, Quo, Rec, Res, and merge remaining 13 rare labels into a single COU label, they merge all 9 client codes into a single CLI label. Instead, [6] merge only 8 of rare labels into a OTHER label and they cluster client codes according to the valence of changing, sustaining or being neutral on the addictive behavior[35]. Then [7] combine and improve above two clustering strategies by splitting the all 13 rare labels according to whether the code represents MI-adherent(Mia) and MI-nonadherent (Min) We show more details about the original labels in Mia and Min in Table ¿tbl:tbl:misc95mia95min?

8.0.0.2 Model Setup

We use 300-dimensional Glove embeddings pre-trained on 840B tokens from Common Crawl [38]. We do not update the embedding during training. Tokens not covered by Glove are using a randomly initialized UNK embedding. We also use character-level deep contextualized embedding ELMo 5.5B model by concatenating the corresponding ELMo word encoding after the word embedding vector. For speaker information, we randomly initialize them with 8 dimensional vectors and update them during training. We used a dropout rate of 0.3 for the embedding layers.

We trained all models using Adam [41] with learning rate chosen by cross validation between \([1e^{-4}, 5*1e^{-4}]\), gradient norms clipping from at \([1.0, 5.0]\), and minibatch sizes of 32 or 64. We use the same hidden size for both utterance encoder, dialogue encoder and other attention memory hidden size; it has been selected from \(\{64, 128, 256, 512\}\). We set a smaller dropout 0.2 for the final two fully connected layers. All the models are trained for 100 epochs with early-stoping based on macro \(\text{F}_{1}\) over development results.

8.0.0.3 Detailed Results of Our Main Models

In the main text, we only show the \(\text{F}_{1}\) score of each our proposed models. We summarize the performance of our best models for both categorzing and forecasting MISC codes in Table ¿tbl:tbl:main95rst? with precision, recall and \(F_{1}\) for each codes.

8.0.0.4 Domain Specific Glove and ELMo

We use the general psychotherapy corpus with 6.5M words (Alexander Street Press) to train the domain specific word embeddings \(\boldsymbol{Glove}_{psyc}\) with 50, 100, 300 dimension. Also, we trained ELMo with 1 highway connection and 256-dimensional output size to get \(\boldsymbol{ELMo}_{psyc}\). We found that ELMo 5.5B performs better than ELMo psyc in our experiments, and general Glove-300 is better than the \(\boldsymbol{Glove}_{psyc}\). Hence for main results of our models, we use \(\boldsymbol{ELMo}_{generic}\) by default. Please see more details in Table ¿tbl:tbl:rst95elmo?

8.0.0.5 Full Results for Ablation on Forecasting Tasks

In addition to the ablation table in the main paper for categorizing tasks, we reported more ablation details on forecasting task in Table ¿tbl:tbl:rst95cxt95anticipate?. Word-level attention shows no help for both client and therapist codes. While sentence-level attention helps more on therapist codes than on client codes. Multi-head self attention alsoachieves better performance than anchor-based attention in forecasting tasks.

8.0.0.6 Label Imbalance

We always use the same \(\alpha\) for all weighted focal loss. Besides considering the label frequency, we also consider the performance gap between previous reported \(\text{F}_{1}\). We choose to balance weights \(\alpha\) as {1.0,1.0,0.25} for Ct,St and Fn respectively, and {0.5, 1.0, 1.0, 1.0, 0.75, 0.75,1.0,1.0} for Fa, Res, Rec, Gi, Quc, Quo, Mia, Min. As shown in Table ¿tbl:tbl:loss?, we report our ablation studies on cross-entropy loss, weighted cross-entropy loss, and focal loss. Besides the fixed weights, focal loss offers flexible hyperparameters to weight examples in different tasks. Experiments shows that except for the model \(\mathcal{C}^{T}\), focal loss outperforms cross-entropy loss and weighted cross entropy.

3pt

References↩︎