[orcid=0009-0002-8434-9754 ]

[orcid=0009-0006-4278-3666 ]

[orcid=0000-0001-8518-917X ]

Personal Knowledge Graph Construction , Large Language Models , Triple Extraction , Conversational Recommendation Systems , Personalized AI , Decentralized Personalization

1 Introduction↩︎

Personalized Knowledge Graphs (PKGs) have emerged as a powerful paradigm for representing user-specific preferences, contextual signals, and behavioral patterns in a structured and machine-interpretable form. Unlike global knowledge graphs that encode only widely “notable” facts, PKGs capture fine-grained, transient, and often non-canonical personal information essential for personalization, user modeling, and adaptive AI systems. Foundational work by Balog and Kenter [1] formally articulated the promise and challenges of PKGs, while subsequent surveys [2], [3] have underscored the need for scalable construction pipelines and interoperable representations. Complementary advances in KG-enhanced recommendation [4] and conversational preference elicitation [5] further demonstrate that explicitly structured relational signals consistently improve personalization quality.

Building on this trajectory, we investigate a dialogue-driven PKG construction approach that leverages open-weight large language models (LLMs) to extract user–item relational triples, validate their semantic fidelity, and assess their downstream utility in recommendation settings. In doing so, this work advances creating, managing, and exploiting personalized, dynamically generated KGs that remain interpretable, interoperable, and semantically grounded.

Although pre-trained LLMs excel at open-domain knowledge extraction, their capacity to construct PKGs, particularly from conversational interactions, remains underexplored. Traditional recommender systems that incorporate structured user–item relationships via graph-based neural models have demonstrated clear gains in accuracy and interpretability [6], [7]. However, such methods typically depend on centrally logged, explicit interaction traces, which are often unavailable due to privacy concerns or infrastructural constraints. Privacy considerations may restrict the collection or sharing of detailed behavioral data, while infrastructural limitations may prevent platforms from maintaining persistent user identifiers, centralized logging pipelines, or cross-service data aggregation. This motivates an alternative paradigm: extracting user preferences directly from natural language dialogue, enabling decentralized PKG construction without persistent tracking or centralized logging.

Entities (users, movies) are assigned Internationalized Resource Identifiers (IRIs), and user–item relations such as likes, seen, and suggested are captured through a small, domain-specific ontology. This yields RDF-compliant graphs that inherit the benefits of formal semantics, global interoperability, and compatibility with downstream reasoning frameworks. We therefore define our workflow as a Personal Knowledge Graph Construction (PKGC) pipeline that transforms conversational data into consistently structured, semantically annotated knowledge graphs. To evaluate the feasibility of this approach, we study the ability of modern LLMs to extract accurate and semantically grounded preference triples from natural multi-turn dialogue in a movie recommendation setting. We use the Recommendation Dialogues (ReDial) dataset [8], [9], which contains human-to-human conversations paired with structured annotations describing users’ stated or implied preferences. This dataset enables a principled comparison between human-annotated ground truth and LLM-extracted PKG structures. Our contributions are as follows:

1. A prompt-based PKG construction pipeline that converts conversational recommendation dialogues into structured PKGs of (User, Relation, Item) triples, using an explicit semantic schema with IRIs and an RDF-compliant relation vocabulary.

2. A systematic evaluation of open-weight LLMs (Qwen3 and Qwen2.5 variants, in multiple sizes and prompting regimes) against the ReDial dataset’s structured annotations, providing the first large-scale assessment of conversational PKG extractability.

3. An analysis of the downstream utility of extracted PKGs, demonstrating how extraction fidelity influences recommendation performance and showing that LLM-derived PKGs can serve as a privacy-preserving and interpretable foundation for decentralized personalized services.

1.1 Motivation↩︎

A central motivation for PKGs is their role in enabling personalized recommendations while supporting mechanisms that promote viewpoint diversity, as demonstrated in frameworks such as KG Inversion [10]. These approaches assume the availability of a reliable method for constructing PKGs directly from users’ organic interactions, including natural language dialogues. This paper provides that missing foundation by evaluating how effectively LLMs can extract structured preference triples needed for PKG construction.

The core of our approach is to use PKGs as a structured, intermediate representation of user preferences, created specifically to be consumed by an LLM. While an LLM could, in theory, process raw conversational history, such an approach is inefficient for on-device applications. Conversations are often meandering and contain irrelevant information that would needlessly consume the LLM’s limited context window. A PKG, by contrast, distills these dialogues into a concise set of salient facts.

One might consider using an LLM to generate a textual summary of the conversation instead. However, this approach presents significant challenges as user history grows. Combining summaries from multiple conversations to create a unified user profile without introducing redundant information would require a sophisticated and computationally expensive LLM-based merging process. In contrast, avoiding redundant information in a PKG is an efficient, algorithmic task; redundant facts (e.g., the same preference expressed in two different conversations) can be trivially ignored or deduplicated. This efficiency is paramount for systems designed to run on user devices.

Furthermore, the structured nature of PKGs makes them highly adaptable and queryable. Retrieving a specific subset of preferences (i.e., all movies a user has seen but dislikes) is a simple graph traversal operation. Achieving the same result from a conversation summary would again require another LLM call to parse the text and extract the relevant information. This adaptability is precisely what enables complex, targeted interventions for promoting recommendation diversity, a task that would be far more complicated and less reliable if performed on unstructured text.

As personalization needs evolve, dynamic PKG adaptation becomes increasingly important. Advances in knowledge editing and continual learning [11]–[13] offer mechanisms to insert or update specific facts in PKGs before they are presented to an LLM, altering functionality without retraining. These tools could enable PKGs to not only be constructed from conversations but also updated to reflect changing user preferences, a capability missing from static graph models and current dialogue-based extraction pipelines.

Therefore, the central motivation for this research is to evaluate the feasibility of using modern open-weight LLMs for the foundational task of triple extraction. By benchmarking the ability of LLMs to bridge the gap between unstructured dialogue and structured PKGs, we are taking the essential first step toward unlocking a more efficient, adaptable, and privacy-preserving paradigm for personalized recommendations. Additionally, having a downstream recommendation task for our extracted PKGs gives us a domain in which to evaluate the efficacy of the extracted PKGs.

1.2 Dataset Considerations↩︎

To evaluate our approach, we use the ReDial dataset [8], [9], a widely adopted benchmark for conversational recommendation research. The ReDial dataset consists of 11,348 dialogues between over a thousand users centered on movie recommendations, mirroring the conversational interactions our proposed system is designed to process. A key advantage of ReDial is its inclusion of structured metadata that details user preferences.

These annotations provide a structured reference that allows us to systematically evaluate the correctness of triples extracted by the LLM-based pipeline.

Another important attribute of ReDial is the conversational diversity it captures. The dataset encompasses a diverse range of conversational styles from over a thousand unique users, featuring varying levels of formality. This diversity ensures the LLMs are benchmarked against realistic, informal language rather than perfectly structured text, a scenario unlikely in real-world recommendation-seeking conversations. In this context, ReDial serves as a high-quality testbed, offering both scale and annotation fidelity to rigorously evaluate the performance of LLM-driven PKG extraction pipelines.

The extraction pipeline itself is not specific to ReDial and can be applied to other conversational datasets or user–assistant interaction logs. In practice, the method only requires dialogues containing identifiable user–item mentions and a schema to map extracted relations into structured triples.

2 Related Work↩︎

2.0.0.1 Open-Domain LLM-Based Knowledge Base Construction (KBC).

LLMs have recently been explored as general-purpose engines for KBC, where early work shows that models such as GPT-3 and T5 can be prompted to produce entity–relation–entity triples from raw text without supervised training [14], [15]. More recent prompt- and retrieval-augmented methods [16] refine this capability, improving extraction fidelity while also revealing persistent issues around consistency, hallucination, and coverage. EDC [17] proposes a schema-flexible pipeline: Extract, Define, Canonicalize, demonstrating strong performance on open-domain datasets. Multi-agent approaches such as CooperKGC [18] reduce hallucination by cross-validating extractions among LLM agents. Our work is aligned with this literature but differs in scope: rather than constructing general-purpose factual KGs, we specialize the extraction process to detect user preference relations in conversational settings, incorporate domain-specific normalization for movie entities, and ground all extracted triples in a PKG structure.

2.0.0.2 Dialogue Relation Extraction (DRE).

This research direction aims to recover relational knowledge from multi-turn conversations. Empirical studies [19] show that frontier LLMs such as ChatGPT and LLaMA-70B perform strongly on DRE benchmarks, benefiting from their ability to reason over long-range dependencies and incomplete context. This suggests LLMs may be well-suited for personalized knowledge mining from conversational data. However, these prior works evaluate extraction against synthetic or task-specific benchmarks, whereas our study conducts a systematic evaluation using human-annotated conversational recommendation data, providing a more realistic and grounded assessment.

2.0.0.3 Preference Extraction in Recommender Dialogues.

LLMs have been used both to simulate recommendation dialogues for data augmentation [20] and to extract user preferences from conversation [21], [22]. Benchmarks such as LLM-REDIAL [20] show that off-the-shelf LLMs struggle with next-item prediction in zero- and few-shot settings, with performance improving only when fine-tuned or supplemented with user histories, highlighting the importance of persistent user modeling. Prior work has explored implicit preference detection [21] and hybrid LLM–retrieval pipelines [22], but these systems focus primarily on optimizing downstream recommendation. In contrast, our work centers explicitly on the extraction task itself, quantifying how accurately LLMs can recover user-specific relational triples that align with human annotations.

2.0.0.4 KG-Enhanced Recommendation Systems.

KGs are widely used to enrich recommender systems with structured relational context. Prior studies [23] show that integrating LLM-derived signals into KG-based architectures improves personalization, while industrial systems [24] demonstrate that validating inferred graph edges enhances robustness under sparse data conditions. These works leverage KGs for downstream reasoning and recommendation, whereas our primary contribution lies upstream: we generate the KG itself from conversational interaction and quantify the fidelity of the extracted relations before they are used in recommendation tasks. This distinction positions our method as complementary to KG-grounded recommenders and LLM-based scoring models.

2.0.0.5 PKGs and User-Centric Semantic Modeling.

PKGs have been explored as a means of modeling personal, context-rich information. Work in personal health informatics has shown how they support individualized reasoning [25], PKG construction from IoT and behavioral data [26], and personalized diet recommendations grounded in clinical semantics [27]. Recent efforts integrate PKGs with causal reasoning and LLMs to generate adaptive, interpretable health insights [28], and apply KG-based personalization techniques in nutrition recommendation [29], and patient modeling [30]. Our work extends this PKG-centered perspective to the domain of conversational recommendation: instead of deriving PKGs from sensors or structured logs, we study how reliably LLMs can extract personal relational knowledge directly from natural multi-turn dialogue, providing a semantic foundation for decentralized, privacy-preserving personalization.

3 Conversational Triple Extraction Pipeline↩︎

Our methodology for the triple extraction pipeline is illustrated in 1. The pipeline takes a multi-turn dialogue between a user and a conversational agent as input and produces structured preference triples of the form \((\texttt{User}, \textit{relation}, \texttt{Item})\). These triples are subsequently normalized, aligned to external knowledge graph identifiers, and incorporated into a PKG.

For experimental evaluation, we instantiate this pipeline using conversations from the ReDial dataset (1.2). ReDial contains dialogues between an initiator and a respondent discussing movie recommendations. In our setup, the initiator is treated as the user whose preferences are modeled, while the respondent plays the role of the conversational assistant.

By default, the movies are represented by IDs in the conversations, but we replaced them with their titles instead. Each conversation is annotated with metadata features. These features describe the initiator’s relationship with each movie mentioned, and we use them to derive relational triples of the form \((\texttt{User}, \textit{relation}, \texttt{Movie})\). The features and the relations they map to are defined in 1.

To facilitate comparison between extracted triples and ground-truth annotations, we construct an augmented version of the dataset in which the metadata-derived relations are replaced by triples produced by the extraction pipeline. This augmentation step is used solely for evaluation and downstream experimentation; it is not required for PKG construction itself. By maintaining the same data format as the original annotations, the augmented dataset enables a controlled comparison between PKGs built from LLM-extracted relations and those derived from the original human annotations. This allows us to measure how extraction quality influences the performance of downstream recommendation models.

2 details the PKG construction process.

3.1 PKG Schema and Semantic Modeling↩︎

We build our PKG using open and interoperable methodologies to ensure that the extracted user-item signals can plug cleanly into larger ecosystems like Wikidata. All the extracted concepts and relations are grounded in RDF and structured through an explicit preference ontology that captures common interaction signals between users and recommended items.

The ontology defines a small set of user–item relations (e.g., preference, interaction history, and recommendation events) that can be consistently represented across conversational domains. While the experimental evaluation in this work focuses on movie recommendation dialogues, the schema itself is not tied to any specific dataset or domain. Instead, it provides a general structure for representing conversationally extracted preferences in a PKG.

3.1.0.1 Entity Representation with IRIs:

Each graph node is assigned a stable, dereferenceable IRI for long-term reuse with external KGs. Users are represented using dataset-provided identifiers, which typically correspond to anonymized participant IDs or session identifiers in conversational datasets. These identifiers uniquely distinguish individual users while preserving privacy, and are incorporated into the PKG namespace using the pattern pkg:User_id. For example, a user with identifier 42 would be represented as pkg:User_42. Entities referenced in conversations are linked to globally recognized identifiers when possible.

As an example, the ReDial dataset provides worker IDs, which we use in the User namespace as pkg:User_workerId for user identity. We also align the extracted movie titles with the corresponding QIDs from the movies’ official Wikidata IRIs. For each extracted triple, the referenced movie name is matched against a curated Wikidata index, and the correct QID is inserted into the PKG to enable global resolution and semantic reuse. Using a conversation that indicated a user likes the movie “The Godfather” as an example, the extracted triple would be as follows (where wd is the Wikidata and pkg is the User Preference Ontology we defined as illustrated in 4): (pkg:User_42, pkg:likes, wd:Q47703). Using the canonical Wikidata identifiers increases interoperability while preserving traceability between extracted triples and the global entity graph.

3.1.0.2 Ontology Design:

Our User Movie Preference Ontology (4) defines two core entity types:

• pkg:User — the source of preference and interaction signals

• pkg:Movie — each anchored to a unique Wikidata entity

These are linked by five object properties that capture taste, experience, and conversational recommendation flow:

• pkg:likes, pkg:dislikes — explicit sentiment

• pkg:hasSeen, pkg:hasNotSeen — watch status, even when implied

• pkg:wasSuggested — tracks which movie was recommended to the user

All properties use strict domain/range definitions, ensuring only valid user-to-movie edges are created. This type-safe design prevents malformed triples from entering the graph during extraction or recommendation inference.

3.1.0.3 Feature Mapping:

The predicates in the PKG ontology correspond to common user–item interaction signals observed in conversational recommendation settings, such as expressed preference, prior experience, or recommendation events. For evaluation purposes, these relations can be mapped to structured annotations available in a given dataset. In our experiments, we instantiate this mapping using the metadata provided in the ReDial dataset (1), enabling a direct comparison between ground-truth annotations and the triples extracted by the LLM while preserving the intended semantic meaning when serializing the relations into RDF.

3.2 Query and Usage Patterns↩︎

Once constructed, PKGs can then be queried using SPARQL and used for downstream tasks. Because the PKG is represented in RDF and linked to external knowledge graphs (e.g., Wikidata), it can support flexible subgraph retrieval and reasoning for personalized services. One common usage pattern is context retrieval for recommendation systems. A user’s PKG may contain preferences across multiple domains (e.g., movies, food, books). For a given task, however, only a relevant subgraph may be needed. For instance, when generating movie recommendations, the system can query only the subset of triples describing the user’s interactions with movies, ignoring unrelated preference signals. This enables recommendation models to operate on focused semantic context rather than the entire graph.

The PKG can also evolve dynamically as users continue interacting with conversational systems. New dialogues with an LLM can be processed by the extraction pipeline to generate additional preference triples, incrementally expanding the user’s PKG and improving personalization over time. As an example, the SPARQL query in [lst:query] retrieves movies that were suggested to a user but have not yet been seen by that user.

SELECT ?movie WHERE {
    BIND(pkg:User_42 AS ?user)
    ?user pkg:suggested ?movie . 
    FILTER NOT EXISTS { ?user pkg:seen ?movie . } 
}

4 Experimental Design↩︎

We evaluate our triple extraction pipeline using a benchmark conversational recommendation dataset. Each dialogue is processed by the extraction model to generate structured \((\texttt{User}, \textit{relation}, \texttt{Item})\) triples. These extracted triples are then incorporated into an augmented version of the original dataset, replacing the reference interaction annotations while preserving the underlying conversational content.

This setup enables a controlled comparison between knowledge graphs constructed from LLM-extracted triples and those derived from the original annotations. The two variants are evaluated using standard information extraction metrics as well as their impact on downstream recommendation performance.

4.1 Evaluation Metrics↩︎

To evaluate the quality of the extracted triples, we compare the set of triples generated by the LLM against the reference triples derived from dataset annotations. Extraction performance is measured using standard information extraction metrics based on the counts of:

• True Positives (TP): Extracted triples that match a ground-truth triple.

• False Positives (FP): Extracted triples not found in the ground-truth set.

• False Negatives (FN): Ground-truth triples not extracted by the model.

A triple is considered a match if the subject, predicate, and object all correspond to the same entities and relations after normalization and entity linking.

From these counts, we calculate \(precision = \frac{TP}{TP + FP}\), \(recall= \frac{TP}{TP + FN}\), and \(F1= 2 \cdot \frac{precision \cdot recall}{precision + recall}\).

Precision reflects the correctness of the model’s predictions, while recall captures their completeness. The F1-score summarizes overall extraction performance by balancing these two aspects. Metrics are computed over the set of triples extracted from each dialogue and aggregated across the dataset.

4.2 Model and Prompting Setup↩︎

To evaluate the robustness of the proposed extraction pipeline across model architectures and scales, we conduct experiments using models from the Qwen3 [31] and Gemma-3 Instruct [32] families. Using our prompt template (available in our repository¹), we perform ablation experiments across zero to ten-shot prompting settings. Few-shot examples follow our shot template² and are prepended to the target conversation during inference. To study the effect of model scale, we evaluate several representative sizes from each family. Because the two families do not provide identical parameter counts, we select the closest available sizes to enable a comparable evaluation across scales. The specific model variants used in our experiments are shown in 5.

Although the Qwen3 models are base language models and the Gemma-3 models are instruction-tuned, the objective of this study is not to perform a head-to-head benchmark between model families. Rather, we use these models to examine how extraction performance varies with model scale, alignment, and prompting strategy within the proposed PKG construction pipeline.

4.3 Downstream Task Evaluation↩︎

To assess the practical utility of the extracted triples, we evaluate their impact on a downstream recommendation task. We run triple extraction on the full ReDial dataset using the models described above and obtain a set of extracted triples per run. Using these triples, we augment the original dataset to obtain the same number of augmented datasets for evaluation. To evaluate the impact of extracted triples on recommendation performance, we compare models trained on the augmented ReDial dataset against models evaluated on the original ground-truth ReDial triples.

Specifically, we evaluate the effectiveness of the downstream task on the FedTREK-LM recommendation system [33], [34]. This recommendation system uses a local LLM in a federated system to generate recommendations using PKGs for context, so it fits nicely with our local, LLM-based, triple-extraction pipeline. This framework constructs PKGs from the ReDial dataset and trains on the conversations to learn good recommendations, followed by evaluating itself on a separate test set from the dataset. To evaluate our augmented dataset in this framework, we construct the FedTREK-LM dataset with our augmented dataset as the base, but then we still use the original ground-truth test dataset for evaluation. This way, we can compare the results with the original ground-truth training dataset against the results with our augmented dataset constructed from our extracted triples. For our primary comparison metric, we will look at the F1-scores of the recommendation models. This experiment, therefore, measures whether the triples extracted by the LLM are sufficiently accurate and semantically meaningful to support downstream personalization tasks.

5 Results↩︎

In this section, we primarily analyze the impact of model size and family on these metrics with a brief look into the effect of k-shot prompting. A comprehensive breakdown of the triple extraction performance across all tested models, families, and prompting configurations is provided in 2. This table details the precision, recall, and F1-scores for individual relation categories as well as the macro-averaged overall metrics, providing a granular view of the extraction fidelity. Our best performing model for extraction was Gemma-12B Instruct (as shown in 5), with the Gemma-3 Instruct family outperforming Qwen3. All models showed an increase in performance until a certain shot setting, with a stark drop at the 10-shot setting. Performance generally scaled with the number of shots until a significant decline at the 10-shot mark. This suggests a threshold where the models begin to overfit to the provided examples, prioritizing the prompt’s fixed patterns over the unique conversational structures in the test data. As our testing jumped from 5 to 10 shots, an optimal inflection point likely exists within that interval before performance degrades.

2pt

5.0.0.1 Which relations tell the strongest signal?

In 6, the clearest patterns emerged when the data was tied directly to user taste:

• liked has the highest F1-score across the board, making it the most reliable source for recommendations. It captures enthusiasm and critique well, even when negative cases are included.

• seen is solid but softer. The model can detect it, but watch status is often implied, and implication is harder for LLMs to pin down.

• suggested is the trickiest. Tracing who introduced which movie sometimes requires long-range conversational memory. Unsurprisingly, smaller models struggle the most here, with particularly low F1-scores. However, this has less impact on the final recommendation goal than liked.

Although a granular analysis of individual triple mistmatches was not performed, a systematic analysis of the relation-specific metrics suggests that the observed recall gaps are rooted in the complexity of conversation that relations appear in. The significantly higher F1 scores for the “liked” relation across all models indicate that preference sentiment is typically expressed through explicit, high signal tokens that were easily captured during extraction. In contrast, relations with lower recall reflect the inherent ambiguity of conversational context. Additionally, the performance disparity can be attributed to the fact that the Gemma models were instruction-tuned, which may play a role in ambiguity resolution.

5.1 Downstream Task Results↩︎

As outlined in Section 4.3, we evaluate the utility of our extracted triples by training the FedTREK-LM recommendation framework on our augmented datasets and comparing the results against models trained on the original ground-truth ReDial triples. For our baseline augmented datasets, we focus specifically on the 5-shot extractions. As detailed in 2, the 5-shot setting consistently yielded the peak, or near-peak, overall F1-scores during upstream extraction across all evaluated Qwen3 and Gemma-3 model sizes, making it the most stable and representative configuration before the severe overfitting and performance degradation observed at 10 shots. To circumvent VRAM scaling constraints, we restrict the downstream recommendation models to the Qwen3 family.

7 (left) illustrates the recommendation performance across different downstream model sizes. While transitioning from ground-truth annotations to LLM-extracted datasets naturally results in an F1-score degradation, this drop is surprisingly low relative to the upstream triple extraction scores for the smallest model. For example, 2 shows that the Qwen3-0.6B model achieved an overall extraction F1-score of only 0.3577 under the 5-shot setting; however, it maintained over 75% of its ground-truth F1-score baseline (0.1571 vs. 0.2087) in the downstream recommendation task. This resilience can likely be attributed to an implicit filtering effect: successfully extracted triples tend to originate from structurally clear conversations that the downstream LLM can inherently parse more effectively, naturally filtering out noisy or meandering datapoints. However, a stark degradation and general inverse scaling trend emerges with the larger models. While the original ground-truth data shows downstream performance scaling positively with model size, peaking at an F1-score of 0.2652 for Qwen3-8B, the augmented datasets cause the larger models to falter significantly. Despite a marginal improvement over the 4B model, the 8B model’s downstream F1-score plummets to 0.1475, failing to surpass even the 0.6B model’s augmented performance. This collapse directly correlates with their upstream extraction failures; as shown in 2, the 4B and 8B models struggled with the upstream extraction task, yielding poor overall F1-scores of 0.1771 and 0.1680, primarily driven by extremely low recall. The severely diminished quality of their extracted PKGs forms a sparse, weak contextual foundation that entirely bottlenecks their naturally higher downstream recommendation capabilities.

7 (right) details the impact of prompt scaling and extraction source on downstream utility, specifically utilizing the Qwen3-0.6B downstream model. Looking at the Qwen3 extractions, downstream performance steadily improves as the shot count increases, peaking at 3 shots with an F1-score of 0.159, before experiencing a slight dip at 5 shots and a severe degradation at 10 shots. Cross-referencing this with 2 reveals that while 5-shot extraction had a higher overall F1-score, the 3-shot extraction maintained a higher overall precision (0.8359 vs. 0.8183). This performance trade-off suggests a high downstream sensitivity to extraction precision within the same model family, indicating that wrong extractions (false positives) actively harm the recommendation task more than marginal increases in correct extractions help.

To further investigate the impact of extraction quality, we trained the Qwen3-0.6B downstream model using triples extracted by the Gemma-3 models under the optimal 5-shot setting. Intriguingly, we observe an inverse scaling relationship: the extractions from the smallest model, Gemma-1B, yielded the highest downstream F1-score (0.184), significantly outperforming the extractions from the 4B and 12B models. By analyzing the relation-specific metrics in 2, we can attribute this to the 1B model’s more holistic and balanced extraction profile. While the 12B model achieved higher overall precision, Gemma-1B maintained a significantly higher recall for the critical "Liked" category (0.7077 vs. 0.6451) and demonstrated much greater consistency in capturing the conversational flow, achieving the highest F1-score for the difficult "Suggested" relation (0.5438). This indicates that for a smaller downstream model like Qwen3-0.6B, a consistently populated, diverse knowledge graph, one that reliably captures explicit preferences alongside conversational context, provides a richer and more effective foundation for learning than a sparse, high-precision graph. Crucially, the fact that optimal downstream utility is achieved using triples extracted by a 1-billion parameter model validates our overarching goal of enabling decentralized, privacy-preserving PKG construction. Because extracting robust preference signals does not strictly require massive, computationally expensive models, this pipeline proves highly viable for efficient, on-device applications.

6 Conclusion↩︎

We present an evaluation of open-weight LLMs for extracting structured user preferences from recommendation dialogues, with the goal of populating PKGs directly from conversational input. While our upstream experiments showed that the Gemma-3-12B Instruct model achieved the highest overall extraction F1-score, our downstream recommendation analysis revealed a more nuanced reality: smaller, highly balanced models like Gemma-3-1B provide a more effective foundation for downstream learning, validating the viability of efficient, on-device PKG construction. Furthermore, highly capable larger models like Qwen3-8B demonstrated an ability to overcome sparse extractions through superior reasoning capacity. While this work employs standard machine learning evaluation metrics, its core contribution is to the Semantic Web community: we validate a reproducible pipeline for building PKGs from dialogue and analyze how extraction fidelity shapes downstream recommendation workflows.

To address the scalability and generalizability of this approach, we note that while this study utilized the ReDial dataset and its corresponding metadata features as a ground-truth framework for rigorous evaluation, our core extraction methodology is fundamentally dataset- and schema-agnostic. The prompt-based extraction relies on natural language understanding of multi-turn dialogues, meaning it can theoretically be applied to any conversational corpus. The specific ontology mapping used in this work was designed to align with ReDial’s structure for benchmarking purposes; however, the pipeline can seamlessly be adapted to target different downstream ontologies or domain schemas without requiring manual metadata mapping in real-world deployments.

This work demonstrates both the feasibility and current limitations of using LLMs to bootstrap PKGs for personalized recommendation systems. In contrast to traditional approaches that rely on explicit, centralized logging or extensive user instrumentation, our method offers a scalable alternative that derives preferences from natural dialogue. These findings open up several promising directions for future work, including better prompt engineering, lightweight fine-tuning, and structured reasoning over extracted triples.

As personalization needs evolve, dynamic PKG adaptation becomes increasingly important. Advances in knowledge editing and continual learning offer mechanisms to insert or update specific facts in PKGs before they are presented to an LLM, altering functionality without retraining. These tools could enable PKGs to not only be constructed from conversations but also updated to reflect changing user preferences, a capability missing from static graph models and current dialogue-based extraction pipelines.

A central challenge in many KG-based personalized service applications such as recommenders is the cold-start problem; obtaining high-quality, explicitly stated structured user preferences is labor-intensive and often infeasible at scale. By contrast, this paper shows that LLMs can effectively extract preference triples from both human-human and human-chatbot conversations, offering a low-friction way to initiate and refine PKGs over time. Such PKGs can also support retrieval-augmented generation (RAG) pipelines, enabling more transparent, controllable, and personalized responses from LLM-based systems. In conclusion, we believe our framework provides a valuable diagnostic lens on the current capabilities of LLMs in extracting user-specific knowledge, a critical step toward adaptive, conversational AI grounded in structured personalization.

Supplementary Material Statement:↩︎

All research artifacts, including source code, dataset construction scripts, and result generation pipelines, are available in our GitHub repository. All external datasets and software dependencies used in this work are documented and linked in the repository’s README.
https://github.com/brains-group/LLMTripleExtractor.

GenAI Usage Disclosure:↩︎

GenAI was used to help generate some parts of the code as well as the paper content. The authors retain full responsibility for the accuracy, originality, and overall content of the code and the paper.

References↩︎