1 Introduction↩︎

The widespread adoption of generative AI powered by Large Language Models (LLMs) has created vast repositories of conversation data [1]. These dialogues offer valuable insights into user behaviors and system performance. However, effectively analyzing and leveraging this accumulated conversational data remains an underexplored challenge in the field [2]–[4].

Unlike conventional information systems, large language model-based systems operate through open-ended interactions without predefined specifications [5], [6]. Users interact with them in diverse, unpredictable ways, creating unique challenges for conversational data analytics. Traditional approaches to extracting product insights struggle with these datasets: supervised learning techniques face prohibitive labeling costs [7], manual dialogue review becomes impractical at the scale of millions of conversations [8], [9], and conventional metrics fail to capture the complex evolution of user satisfaction across multiple turns [10].

Given these challenges, many product development teams have adopted an exploratory “Retrieve and Analyze” workflow to derive insights from their conversation data (see Appendix 6 for a detailed real-world case study). In this approach, retrieval quality fundamentally determines analysis effectiveness—if relevant conversations cannot be efficiently found, critical insights remain hidden despite analyst expertise. For example, when investigating satisfaction issues, product managers often use basic keyword searches like “unhappy” or “disappointed,” missing cases where dissatisfaction is expressed implicitly or across multiple turns. This retrieval gap creates significant blind spots in understanding user experiences and severely limits the value extracted from conversation datasets.

To address these limitations, we build upon the concept of Conversational Data Retrieval (CDR) [11]: the task of retrieving relevant conversations from large chat histories based on queries targeting conversation-specific content and context. As illustrated in Figure 1, CDR differs fundamentally from traditional document retrieval [12], [13] by addressing conversation-specific challenges: multi-turn exchanges, implicit meanings, and topic shifts.

Beyond analytical use cases, effective CDR enables applications such as AI memory systems [14], [15], and retrieval-augmented generation [16], [17]. However, current retrieval solutions were not designed with conversations in mind, limiting the potential of these applications.

Despite its value, CDR remains underexplored in research. This gap stems from several factors: the proprietary nature of industrial conversation datasets, privacy concerns limiting public data availability, and the lack of standardized evaluation metrics [18], [19]. These challenges have hindered research progress on methods specifically designed for CDR.

To address this gap, we introduce a comprehensive benchmark for CDR. Figure 2 illustrates our construction process, including data collection, query design, and validation. Our contributions include: (1) the first benchmark specifically targeting CDR, comprising 1.6k queries and 9.1k conversations; (2) evaluation of 16 commonly used embedding models revealing performance disparities between document and conversation retrieval; (3) a taxonomy of five essential analytical tasks exposing unique challenges in CDR; and (4) practical query templates developed with domain experts applicable to product improvement workflows. This benchmark facilitates structured development of conversation retrieval models, supporting applications and research in conversation analysis.

2 Related Works↩︎

Information retrieval (IR) has evolved from lexical matching [20] to neural approaches [21]. Recent advances in generative AI have linked retrieval with conversation, enabling conversational search [22], [23], agent memory systems [24], [25], and retrieval-based reasoning [26], [27]. However, these efforts focus on using retrieval to enhance conversations [28], [29], not on effectively retrieving conversational data itself.

The unique value of conversational data lies in its multifaceted nature. Human-Computer Interaction research has identified several dimensions critical for understanding these interactions—including user intentions, emotional responses, conversation flow patterns, and trust development [8], [9]. These elements often span multiple turns and contain implicit signals that traditional document retrieval approaches struggle to capture [30], [31]. Effective analysis requires methods to identify these complex patterns within conversations.

While industrial applications generate vast conversational data, privacy concerns and proprietary issues severely limit public access to these datasets [32]. Even available datasets often lack sufficient coverage of specific analytical dimensions needed for comprehensive evaluation [33], [34].

Synthetic conversational data offers a valuable solution to these constraints, as high-quality synthetic dialogues can match or exceed the performance of systems trained on real data [35]–[37]. This approach enables more controlled evaluation by systematically varying conversation attributes while maintaining natural dialogue properties.

However, existing IR benchmarks [38], [39] focus on documents, while dialogue datasets typically focus on generation tasks. This creates a significant gap between the analytical needs identified in HCI research and available evaluation frameworks. Our CDR benchmark addresses this gap by integrating multi-dimensional aspects of conversations with a comprehensive evaluation framework.

3 Designing the CDR Benchmark↩︎

3.0.0.1 Data Collection and Industrial Requirements

To establish a foundation for the CDR benchmark, we collected conversational data from 11 diverse open-source dialogue datasets including LMSYS Chat [40], WildChat [33], and DialogSum [41]. To ensure quality and remove duplicates, we applied filtering using the NeMo Curator framework [42], refining approximately 2.4 million conversations to 600k high-quality dialogue instances. The complete data sources and filtering method are detailed in Appendix 7.

To ensure industrial applicability, we sampled 1k conversations for analysis and gathered input from 20 experts in generative AI product development. From this combined research, we identified key information needs when examining conversational data and determined five core areas for product improvement, shown in Table 1. These areas reflect how conversational data differs from traditional document retrieval challenges.

3.0.0.2 Query Template Design and Generation

From the five core areas in Table 1, we created 130 query templates that capture the specific characteristics of each category. Each template included placeholder elements to cover diverse conversational scenarios.

For example, a template in the Emotion & Feedback category might be: “Find conversations where users express {emotion} after {system_action}” For the {emotion} placeholder, values included “frustration,” “disappointment,” and “satisfaction.”

We defined approximately 510 placeholder values across different categories. By combining these placeholders with our templates, we generated a total of 28k specific queries. Full details of templates and placeholders are provided in Appendix 8.

3.0.0.3 Query-Aligned Conversation Synthesis Method

Finding conversations that match our diverse queries presented two challenges: our corpus could not cover all specific query scenarios needed, and manually labeling thousands of conversations would be prohibitively time-consuming.

To address these limitations, we first retrieved top-5 candidate conversations for each query using the embedding model [43]. Twenty expert annotators with industry experience in conversational AI product development then manually reviewed these candidates. They classified each as related or unrelated based on whether it faithfully reflected the query intent. When no suitable match existed, we used reasoning-capable language models—Claude-3.7 [44], o1 [45], and o3-mini [46]—to create synthetically aligned conversations by adapting existing conversations from our corpus. These LLM-generated conversations were also validated by expert annotators to ensure both query fidelity and conversational naturalness.

Our conversation generation prompt (detailed in Appendix 9.1) instructed models to maintain each conversation structure and characteristics while incorporating elements needed for query alignment. This approach preserved the natural variation found in real conversations while ensuring examples contained features necessary for evaluation.

By combining pre-aligned conversations with synthetically aligned conversations, our method maintained domain diversity. Figure 3 shows balanced coverage across major categories like People and Society (17.00%) and Business and Industrial (15.79%), as classified by a fine-tuned classifier³.

3.0.0.4 Expanding Query-Conversation Relevance Mappings

To create a realistic retrieval benchmark, we expanded each query to match multiple relevant conversations through a three-step process. First, we trained a specialized reranker model using 300k conversations from our corpus. We generated positive and negative query examples with LLaMa 3.3 70B [47] using prompts in Appendix 9.2 and 9.3, and fine-tuned the GTE Reranker [48] (training detailed in Appendix 10). Second, we applied this reranker to identify candidate relevant conversations, selecting pairs with relevance scores above 0.9 and excluding overly general queries matching more than 50 conversations.

Third, we validated mappings through comprehensive human assessment. Expert annotators conducted full manual evaluation of approximately 4k query-conversation pairs across 200 queries. We applied binary relevance criteria with majority voting for reliability, conservatively removing non-consensus cases. We trained a ModernBERT-based [49] relevance classifier using these manually validated pairs, achieving 95.2% accuracy, as detailed in Appendix 11. For remaining queries, we applied the classifier to predict relevance for all pairs, then employed two-stage human verification. First, we prioritized uncertain cases where sigmoid scores fell below 0.9. Second, we identified boundary inconsistencies where irrelevant predictions appeared among relevant pairs, and vice versa. Human annotators verified both uncertain predictions and inconsistent boundaries, ensuring comprehensive coverage while efficiently allocating annotation effort to critical cases.

3.0.0.5 Benchmark Composition and Characteristics

Our comprehensive mapping pipeline provides an efficient method for constructing high-quality query-conversation pairs. This methodology offers a practical solution for industrial deployment where cost-effective data mapping is essential. However, as our goal is to establish a rigorous benchmark, we conducted additional validation to ensure maximum integrity. We employed four LLMs—GPT-4o [50], o3-mini [46], Claude 3.7 Sonnet [44], and Gemini 2.0 Pro [51]—with the prompt in Appendix 9.4 to cross-check all pairs. Cases where LLMs disagreed were flagged for expert review by annotators, who applied consistent binary relevance criteria with majority voting. Pairs without clear consensus were conservatively discarded. Through this multi-stage validation approach combining LLM scalability with human verification at each step, 97% of all query-conversation mappings passed assessment, ensuring the final benchmark meets the highest quality standards.

The final CDR benchmark consists of 1,583 queries and 9,146 conversations (Table 2). Conversations average 5.4 messages and 464 tokens⁴. Each query maps to 20.44 relevant conversations on average. Query distribution spans five core areas: Intent & Purpose (36.1%), Emotion & Feedback (20.1%), and three other categories, with detailed examples provided in Appendix 15.

max width=

4 Experiments and Analysis↩︎

4.1 Experimental Setup↩︎

We evaluated 16 widely used embedding models from open-source communities and commercial providers including OpenAI [43], Cohere [52], and Voyage AI [53]. Performance was assessed using NDCG@10, Recall@10, and Precision@10 at three retrieval granularities—session-level, turn-level, and sliding window (chunk size = 3). For detailed evaluation methodology, see Appendix 12.1.

4.2 Results and Analysis↩︎

Table [table:embedding-model-result] summarizes model performance on our CDR benchmark. Among commercial API models, Voyage-3-large [53] achieved the highest performance in both turn-based (NDCG@10: 0.5079) and session-based (NDCG@10: 0.5036) evaluation, while Text-embedding-3-large [43] led in sliding chunk settings (NDCG@10: 0.5130). Among open-source models, Stella_en_1.5B_v5 [13] demonstrated consistently high performance across all evaluation settings. Interestingly, some models showed significant performance variations across different evaluation settings. For instance, NV-Embed-v2 performed poorly in turn-based evaluation (NDCG@10: 0.3170) but achieved substantially improved performance in session-based evaluation (NDCG@10: 0.4592). Even top-performing models scored just above 0.5 in NDCG@10, highlighting the challenges of modeling conversational structure, context transitions, and implicit references.

4.3 Performance Across Task Categories↩︎

Figure 4 reveals performance variations across task types. All models score highest in ‘Emotion & Feedback’ and ‘Intent & Purpose’, but perform poorly in ‘Conversation Dynamics’ where even the best models score below 0.17. This suggests current models are good at understanding content and explicit statements but struggle with understanding how conversations develop and flow.

No model excels across all categories - even top-performing Voyage-3-large shows varied results. This suggests no dominant approach exists yet for CDR. Optimal architectures remain unexplored, particularly for conversation structure understanding, which is crucial for practical applications.

4.4 Analysis of Retrieval Failures↩︎

We identified three consistent failure patterns in current embedding approaches to CDR. Table 3 illustrates these challenges: Role Recognition Failure, Dynamic Progression Failure, and Semantic Contextual Misinterpretation. These failures occur across both turn-based and session-based analysis, revealing models’ inability to capture conversation dynamics at multiple levels. The consistent challenge is that models miss implicit meanings that emerge from conversational context—patterns where the actual roles, progressive changes, or situational context must be inferred from dialogue flow rather than explicit keyword matching.

These errors stem from a fundamental limitation: current models process conversations as collections of words and topics similar to documents, rather than as dynamic exchanges with temporal flow and implicit state changes. Standard embeddings capture vocabulary similarities but miss the contextual evolution and interactive nature of dialogue. This explains the poor performance in Conversation Dynamics across all models and signals that effective retrieval systems must be redesigned to capture the unique properties of human dialogue like turn-taking patterns and implicit state transitions.

5 Conclusion↩︎

Conversational Data Retrieval (CDR) benchmark establishes the first comprehensive framework for evaluating retrieval systems on conversation data. Experimental results show that even the highest-performing models have not reached satisfactory performance. Our benchmark exposes fundamental challenges unique to conversational data: understanding implicit states, tracking conversation flow, and interpreting contextual references. Our work provides standardized evaluation methodology and query templates for product improvement while establishing a foundation for conversation-specific retrieval techniques that better capture the multi-dimensional nature of human-AI interactions.

Limitations↩︎

Our benchmark is limited to English text-based conversations, which may constrain evaluation in multilingual or multimodal settings. This focus, while enabling controlled evaluation, could limit the broader applicability of our findings to diverse linguistic contexts and interaction modalities in global conversational AI applications.

Our benchmark evaluates embedding-based retrieval models, reflecting their widespread adoption in conversational memory systems where turn, session, and segment-level granularities are commonly employed. However, the lack of specialized retrieval models designed specifically for conversation represents a gap in the field that our benchmark could help address through future development of conversation-tailored representation architectures.

While our benchmark provides comprehensive evaluation of retrieval models with robust data quality validation through domain expert involvement, it does not extend to empirical studies of industrial problem-solving applications. Although our motivation stems from real-world challenges and our benchmark identifies optimal approaches under current conditions, further research is needed to validate the practical value of these findings in actual deployment scenarios and their impact on end-user satisfaction in conversational AI systems.

Ethical Considerations↩︎

In the development and application of the CDR benchmark, we carefully considered various ethical aspects. Since conversational data inherently contains user interactions and diverse linguistic expressions, we prioritized privacy protection throughout the data collection and processing stages. We utilized only publicly available open-source datasets and included specific guidelines in our conversation generation prompts to address any potentially remaining personal expressions or sensitive information (see Appendix 14). These guidelines included instructions to “appropriately redact or anonymize any PII in reference conversations,” “avoid generating conversations that could be misleading, harmful, or promote unethical behavior,” and “ensure that no personally identifiable information such as names, addresses, phone numbers, financial details, social security numbers, or other sensitive data is exposed or inferred.”

To ensure diversity and balance in the conversational data, we designed the benchmark dataset to encompass a wide range of domains without bias toward specific topics or areas. As shown in Figure 3, we included a balanced representation of conversations from broad domains such as society, business, technology, and science, thereby minimizing bias toward particular areas. Additionally, we explicitly incorporated a ‘Trust, Safety & Ethics’ category in the benchmark’s task areas to establish ethical conversational retrieval capabilities as an important evaluation criterion.

While conversational data retrieval technology can contribute to positive purposes such as improving service quality and user experience, it also carries potential risks of privacy infringement or misuse as inappropriate surveillance tools. We recognize this duality and hope that the CDR benchmark will serve as a tool to promote balance between ethical values and innovative technological advancement. Through this, we believe that the development of conversational AI systems can progress in a direction that respects users’ rights and dignity.

Acknowledgments↩︎

This work was supported by Coxwave, Artificial intelligence industrial convergence cluster development project funded by the Ministry of Science and ICT(MSIT, Korea) & Gwangju Metropolitan City, Institute for Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (RS-2019-II190075, Artificial Intelligence Graduate School Program(KAIST)) and the Korea government(MSIT) (No. RS-2024-00509279, Global AI Frontier Lab).

6 Industry Case Study of the “Retrieve and Analyze” Approach↩︎

The “Retrieve and Analyze” methodology mirrors how analysts and product managers naturally approach problem-solving in real business environments. When faced with user feedback or product issues, human analysts typically form initial hypotheses, gather relevant examples, analyze patterns, and progressively refine their understanding through iterative investigation. What has changed with large-scale conversation data is not this fundamental analytical process, but rather the need for computational assistance to efficiently navigate thousands or millions of conversations.

The following case study illustrates how this human-centered analytical approach, supported by conversation retrieval capabilities, works in practice. This example is adapted from an actual business scenario at a health and fitness application company, demonstrating both the natural analytical workflow and the positive impact of effective conversational data retrieval.

A customer experience team was investigating increased user dissatisfaction following a recent update. Traditionally, they relied on user ratings and manual reviews of customer complaints. However, after implementing chatbot support, these methods became inadequate - the chatbot interface lacked rating systems, and the chat volume overwhelmed manual inspection capabilities.

To address this challenge, the team implemented a conversation retrieval system, beginning with a broad query: “Find sessions where users express dissatisfaction.” Sample analysis revealed mentions of the points reward system, prompting them to deep dive with a more targeted search: “Find conversations where users express dissatisfaction with changes to the points reward system” to determine if this was a widespread issue rather than isolated incidents.

This refined approach confirmed their hypothesis, revealing reduced point accumulation rates as the primary driver of dissatisfaction, with users consistently comparing the new system unfavorably to the previous one. Through this methodical process of hypothesis formation and targeted validation, the team efficiently pinpointed the specific issue causing user frustration—a discovery that would have consumed significantly more time and resources using traditional review methods.

7 Dataset Sources and Filtering Method↩︎

7.1 Dataset Sources↩︎

We constructed an initial dataset comprising around 2.4 million conversations by aggregating 11 diverse open-source datasets. To ensure broad coverage of dialogue scenarios, our dataset includes both real-world and synthetic conversational data. An overview of the datasets is provided in Table 4.

7.2 Filtering Method↩︎

For data quality management, we employed a multi-stage filtering process using the NeMo Curator framework. We first applied exact and fuzzy deduplication to remove identical or near-identical conversations. Next, we conducted semantic deduplication by utilizing a model fine-tuned for semantic search⁵, effectively filtering out semantically redundant instances. For quality filtering, we employed a model fine-tuned for conversation quality assessment⁶, retaining only conversations labeled as high-quality among high, middle, and low categories. Finally, heuristic score filtering was applied to remove data with excessive punctuation, URLs, and repeated lines, paragraphs, or \(n\)-grams. Through this comprehensive filtering process, we obtained approximately 600,000 refined conversation data points from an initial set of 2.4 million dialogues.

8 Template and Placeholders↩︎

To ensure the CDR benchmark captures realistic analytical scenarios, we collaborated with domain experts in conversational AI product development to design comprehensive query templates and placeholder values. This expert-guided approach ensures that our benchmark reflects actual information needs encountered in industrial applications. All templates and placeholder values are available in our public repository.

8.1 Query Template↩︎

We provide representative examples of query templates used in the benchmark generation across five task categories (Emotion & Feedback, Intent & Purpose, Conversation Dynamics, Trust, Safety & Ethics, Linguistic Style & Expression). The examples of these query templates are presented in Table 5.

8.2 Placeholders and Example Values↩︎

We illustrate examples of placeholder values utilized within query templates in Table 6, showing potential variability across queries generated for the benchmark.

9 Prompts↩︎

This section provides the detailed prompts used throughout our CDR benchmark development process.

9.1 Conversation Generation Prompt↩︎

Figure 5 outlines the prompt for generating synthetic conversations that closely match specific queries, ensuring natural, multi-turn conversations that accurately reflect query intent while maintaining appropriate length and format.

9.2 Query Generation Prompt↩︎

Figure 6 shows the prompt used to generate a single synthetic search query from a set of conversations, designed to help LLMs identify key insights and patterns within conversation clusters while focusing on product management perspectives.

9.3 Query Augmentation Prompt↩︎

Figure 7 presents the prompt for augmenting the initial query by generating three hard negative examples and one alternative positive formulation, facilitating contrastive learning by creating semantically similar but functionally distinct queries.

9.4 Relevance Classification Prompt↩︎

Figure 8 shows the prompt used for assessing query-conversation relevance. This prompt was used by modern LLMs during our verification process to evaluate whether conversations were relevant to specific queries.

10 Reranker Training↩︎

To effectively map queries to relevant conversations, we trained a specialized reranker model using approximately 300k conversations from our filtered corpus. We used LLaMa 3.3 70B [47] to generate training data through a two-step process. First, we applied the synthetic query generation prompt (Figure 6) to create one relevant query per conversation that captured the core information needs represented in the dialogue. Then, using the query augmentation prompt (Figure 7), we generated three hard negative queries (semantically similar but intentionally irrelevant) and one additional positive query (different wording but preserving intent) for each conversation.

This approach yielded approximately 1.5 million query-conversation pairs with a 2:3 positive-to-negative ratio. We fine-tuned the GTE-Multilingual-Reranker model [48] using a binary cross-entropy loss function with hard negatives, a learning rate of 2e-5 with linear warmup and decay, and maximum sequence length of 8192 tokens to accommodate longer conversations. The model was trained for 3 epochs on a single NVIDIA H100 GPU. The reranker achieved an average precision of 96.22% on our validation set after the final epoch. We applied this model with a threshold score of 0.9 to identify candidate relevant conversations across our corpus for the final benchmark construction.

11 Classifier Training↩︎

We trained a specialized binary relevance classifier to validate the reliability of the mapped query-conversation relationships. This classifier was designed to distinguish relevant and irrelevant query-conversation pairs in alignment with human judgments. For training, we utilized approximately 3K relevance pairs obtained through human assessment and an additional 20K synthetic relevance pairs generated through the procedure described in 10. This resulted in a training dataset of approximately 23K pairs with a balanced distribution of relevant and irrelevant examples.

Fine-tuning was performed on the Modern-BERT-base model[49] using a learning rate of 2e-5 with linear warm-up and decay scheduling, batch size of 128, and maximum sequence length of 8192 tokens. The model achieved an average precision of 95.2% on the validation set. We applied this classifier, which was trained on human-verified data, to verify and filter the remaining query-conversation mappings. This ensured that only high-confidence pairs were retained in the benchmark and that relevance standards remained consistent and reliable throughout the process.

12 Evaluation Details↩︎

12.1 Evaluation Setup↩︎

To ensure fair and consistent comparison across all evaluated models, we applied unified evaluation protocols. Each model was tested using its original embedding dimension and maximum sequence length as specified in the official documentation. For prompt-based embedding models, we utilized the prompts without any modifications. All conversational data used in the experiments was preprocessed in a uniform manner, ensuring format consistency across all models and minimizing performance variations arising from preprocessing discrepancies.

We define three evaluation settings that differ in the granularity of the retrieval unit:

Turn-based Evaluation: Each conversation turn is treated as an independent unit. For a given query, the model retrieves the most similar individual turn from the corpus. The conversation containing the retrieved turn is considered the final match.

Sliding Chunk Evaluation (k=3): Conversations are segmented into overlapping chunks of three consecutive turns. Given a query, the model retrieves the most similar chunk from all chunks in the corpus. The conversation containing the retrieved chunk is selected as the final match.

Session-based Evaluation: The entire conversation serves as the retrieval unit. For a given query, the model directly retrieves the most similar conversation session from the corpus.

12.2 Efficiency Evaluation↩︎

We also measured practical runtime metrics to evaluate real-world usability:

• Ingestion Time: The total time required to embed the entire test corpus of 9,146 conversations. This process includes tokenization, model forwarding, and storing the embeddings in memory.

• Inference Time: The combined time required to: (1) embed all 1,583 queries, (2) retrieve the corresponding conversations using these embeddings, and (3) compute the final rankings. This represents the end-to-end query processing time.

All experiments were conducted with a batch size fixed at 4 for both ingestion and inference measurements. The reported times represent the total elapsed time for processing the entire dataset.

12.3 Hardware Specifications↩︎

All evaluations were conducted under the same setup, and the hardware specifications are summarized in Table 7.

13 Additional Experimental Results↩︎

13.1 Performance by Additional Metrics↩︎

To provide a comprehensive evaluation beyond the primary results reported in Table [table:embedding-model-result], we present an extensive analysis of model performance across multiple evaluation metrics. We evaluate all models under three distinct retrieval configurations: Turn-based, Sliding Chunk (k=3), and Session-based approaches. For each configuration, we report performance across five key metrics: Accuracy (ACC), Precision (P), Recall (R), Normalized Discounted Cumulative Gain (NDCG), and Mean Reciprocal Rank (MRR), evaluated at cutoff thresholds of 1, 5, 10, and 20. The detailed results are systematically presented in Tables [table:turn-based-model-additional-result], [table:sliding-chunk-model-additional-result], and [table:session-based-model-additional-result], respectively. This multi-faceted evaluation framework enables a thorough assessment of model efficacy across varying retrieval granularities and ranking depths, providing deeper insights into the comparative strengths and limitations of each approach.

max width=

max width=

max width=

13.2 Performance per Tasks↩︎

Table [table:embedding-model-result-per-task] presents the detailed NDCG@10, Recall@10, and Precision@10 performance of all evaluated embedding models across the five task categories in our benchmark. As illustrated in Figure 4, performance varies significantly between task types, with most models showing strengths in content-oriented categories like ‘Emotion & Feedback’ and ‘Intent & Purpose’ while struggling with interaction-focused categories, particularly ‘Conversation Dynamics’.

The table highlights the lack of a universally dominant approach for conversational data retrieval tasks. Even top-performing models like Voyage-3-large demonstrate inconsistent performance across different categories. Notably, ‘Conversation Dynamics’ remains challenging for all models, with the highest scores barely reaching 0.17, indicating a substantial opportunity for architectural improvements specifically designed to capture conversation flow and structure.

max width=

14 Dataset License and Disclaimer↩︎

In this work, we utilize multiple publicly available open-source dialogue datasets to construct our initial data pool. The LMSYS-Chat-1M dataset is distributed under a custom LMSYS-Chat-1M License Agreement and is non-redistributable. The WildChat-1M-Full dataset is licensed under ODC-BY 1.0 (Open Data Commons Attribution). The Bitext Customer Support dataset is released under the CDLA-Sharing 1.0 license. The Schema-Guided Dialogue (SGD) dataset is provided under a CC BY-SA 4.0 license. The DialogSum and DailyDialog datasets are released under CC BY-NC-SA 4.0, while OpenDialKG and Bot-Adversarial Dialogue (BAD) are licensed under CC BY-NC 4.0, with the latter restricted to research use only. The MultiWOZ 2.2 and SmolTalk datasets are licensed under Apache 2.0. The MobileConvRec dataset is released under a CC BY 4.0 license. All datasets were used strictly for research purposes in compliance with their respective licenses. No personally identifiable information (PII) was included, and all data was either anonymized by the original providers or manually verified during the curation process. We do not claim ownership of these datasets and respectfully acknowledge the contributions of their original authors and publishers.

15 Query-Conversation Pair Examples by Task Category↩︎

We provide representative examples of query-conversation pairs for each of the five task categories in the CDR benchmark: Intent & Purpose, Linguistic Style & Expression, Trust, Safety & Ethics, Emotion & Feedback, and Conversation Dynamics. Examples for each category are shown in Tables 8–12.

References↩︎