1 Introduction↩︎

With the rapid expansion of academic journals and the increasing specialization of research fields, researchers face growing challenges in selecting the most appropriate journal for their work [1]. This process has become increasingly complex, as authors need to consider multiple factors, including journal scope, impact factor, readership, and publication speed [2]. Submitting to an unsuitable journal not only reduces the chances of acceptance, but may also delay the dissemination of research findings. Consequently, journal selection remains an important and challenging task, particularly for early-career researchers who may be less familiar with journal scopes as well as experienced scholars entering new fields [3]. In this context, effective journal recommendation systems are of growing interest. By automatically matching manuscripts with relevant journals, such systems may improve the efficiency and success rate of the submission process. They can also facilitate the diffusion of scientific knowledge and support the career development of researchers [4].

To address the growing complexity of journal selection, various journal recommendation systems have been developed. These systems evolve from basic text similarity tools to advanced deep learning models. Early systems such as eTBLAST [5] and JANE [6] rely on TF-IDF-based similarity measures and recommend journals based on aggregated matching scores. Although effective for capturing lexical similarity, these methods are limited to surface-level text information. Later studies introduce machine learning and hybrid approaches. For example, [7] propose a content-based recommender system for computer science publications using feature selection and softmax regression. In the biomedical domain, [8] develop Pubmender, a deep learning framework based on convolutional neural networks (CNNs) trained on biomedical article abstracts. More recently, transformer-based models have attracted increasing attention. For instance, [4] employ SBERT embeddings to capture semantic similarity between manuscripts and journal scopes, achieving improvements over classical baselines. Similarly, [9] explore transformer models trained on large-scale Web of Science data to identify latent patterns in journal selection. A broader review of scholarly recommendation systems is provided in [3].

Journal recommendation is a specialized branch of recommender systems, which are widely adopted in areas such as e-commerce [10], entertainment [11], social media [12], and academic publishing [3]. Major publishers such as IEEE, Springer, and Elsevier develop journal suggestion tools to assist authors during the manuscript submission process. In academic research, publication recommendation systems have been extensively studied in computer science [7], biomedical research [8], and computational linguistics [13]. Despite these advances, most existing journal recommendation systems remain fundamentally similarity-driven. They typically rely on predefined feature representations and fixed scoring functions. Consequently, these methods may have difficulty capturing the complex relationship between manuscript content and journal scope, particularly for interdisciplinary research.

Recent progress in large language models (LLMs) provides a potential alternative by allowing manuscript information and journal descriptions to be evaluated jointly in natural language, rather than through similarity scores alone [14]. LLMs are particularly suitable for text-intensive tasks because they are designed to process, integrate, and summarize information expressed in natural language. In recent years, LLMs have been increasingly applied to scientific contexts, including literature review assistance [15], citation recommendation [16], peer review support [17], and the analysis of scholarly documents [18]. These studies suggest that LLMs can effectively handle domain-specific terminology and complex scientific narratives when provided with appropriate contextual information. Journal recommendation naturally belongs to this class of tasks, since it mainly relies on textual descriptions of manuscripts and journals rather than explicit decision rules [19]. Motivated by these observations, this study explores the use of LLMs for journal recommendation through a prompt-based framework. We evaluate the recommendation performance of LLMs and further examine whether additional bibliometric context can improve recommendation accuracy.

To this end, this study investigates an LLM-powered framework for journal recommendation. Our analysis is based on a dataset collected from the Web of Science, comprising 23,609 research articles published between 2021 and 2025 from 49 journals. These journals also constitute the candidate set for recommendation and mainly cover statistics and related fields. For each article, we collect structured bibliographic information and textual content, including title, abstract, keywords, and reference list. In addition, we collect the corresponding Aims & Scope statements from official journal sources to characterize the thematic focus of each candidate journal. Based on this dataset, we develop a unified LLM-powered prompting framework that formulates journal recommendation as a structured decision-making task in natural language. The overall workflow of the proposed method is illustrated in Figure 1. Using this framework, we conduct a series of experiments to evaluate journal recommendation performance. These experiments include comparisons under different data settings (with and without reference list), accuracy analysis at different cutoff levels (Top-3, Top-5, and Top-10), stability analysis based on Top-5 overlap counts and Jaccard similarity, and additional investigations into potential information leakage and model reasoning behavior. Through these experiments, we aim to evaluate the effectiveness of LLM-based journal recommendation, examine the role of citation-related reference information in improving recommendation performance, and provide further insights into the relationship between model reasoning and recommendation outcomes.

The main contributions of this study are threefold. First, we provide an empirical evaluation of LLMs for journal recommendation based on a large-scale dataset of statistical publications. Specifically, we formulate journal recommendation as a natural language decision-making task, allowing the model to evaluate the alignment between manuscript content and journal scope directly from textual descriptions. Second, we propose a unified LLM-powered prompting framework that integrates textual content with reference-based information. In particular, we incorporate reference lists as additional contextual signals to enrich the representation of academic background information. This framework also enables comparisons under different information settings. Third, we conduct a series of experiments on recommendation accuracy, temporal stability, and reasoning behavior. The results show that incorporating the reference-based contextual information improves recommendation performance under most settings. We further observe from the reasoning analysis that accurate recommendations are often accompanied by clear and informative reasoning, whereas ambiguous reasoning may correspond to less precise ranking results. These findings provide useful insights for the design of AI-assisted journal recommendation systems. Our recommendation system is now publicly available at https://github.com/LLM-Journal-Rec/journal-recommendation-agent.

The remainder of this paper is organized as follows. Section 2 reviews the related literature. In Section 3, we introduce our dataset and provide a descriptive analysis. The LLM-powered journal recommendation framework is introduced in Section 4, including the design of prompts from multiple perspectives. Section 5 reports the main experimental results, including overall performance and the impact of incorporating citation information. The reliability, risks, and the reasoning behavior of the LLM-powered journal recommendation framework are examined in Section 6.

2 Literature Review↩︎

Journal recommendation is an important task in scholarly publishing. With the development of related technologies, a wide range of methods has been proposed to improve recommendation accuracy. This section reviews existing studies from several perspectives, including traditional recommendation methods, machine learning and deep learning approaches, and more recent frameworks based on Transformer architectures and LLMs.

2.1 Traditional Journal Recommendation Approaches↩︎

Early studies primarily adopt information retrieval-based approaches, including content similarity [20], topic modeling-based similarity [21], semantic structure similarity [22], and hybrid feature representations integrating multiple content aspects [23]. The first two approaches rely mainly on textual information, measuring similarity through word-level overlap or latent topic co-occurrence [24], [25]. In contrast, semantic structure-based methods explicitly incorporate domain knowledge to quantify semantic relatedness between papers. [26], for example, proposed a document-level semantic similarity framework that represents academic papers through structured topic events. Building upon this semantic perspective, several studies further explored hybrid representations by integrating multiple content features into richer paper embeddings for recommendation tasks. Along this direction, [27] combined co-citation and content similarity features to generate recommendations by leveraging heterogeneous information sources.

2.2 Machine Learning and Deep Learning Approaches↩︎

Subsequent studies introduced machine learning and deep learning methods to further improve recommendation performance. At the text representation stage, manuscripts are encoded into feature vectors using techniques ranging from traditional weighted representations to distributed embeddings such as Word2Vec and Doc2Vec [28]. Classical machine learning models, including gradient boosting trees [29] and support vector machines [30], mainly learn decision boundaries for category prediction based on vectorized features. [28] employ Doc2Vec to embed titles, abstracts, and keywords, and subsequently train XGBoost to map these embeddings to journal categories. With the increasing complexity of recommendation scenarios, neural network–based models, including CNNs [31], RNNs [32], and GNNs [33], have demonstrated strong effectiveness in practical applications. Compared with traditional machine learning approaches, neural network–based models are more capable of capturing complex nonlinear patterns and hierarchical structures. [8] develop a deep learning–based recommender using CNNs to extract high-level representations from article abstracts. In a related direction, [34] adopt an attentive RNN model with heterogeneous knowledge embeddings for citation recommendation. More recently, [35] propose a GNN-based venue recommender that models relationships among papers, authors, and venues through graph structures.

2.3 Recommender Systems Based on Transformers and LLMs↩︎

In recent years, Transformer architectures have advanced natural language processing by capturing fine-grained and context-sensitive relationships among tokens. Transformer-based models have become increasingly important in journal recommendation tasks, including BERT [36], [37] and its variants such as RoBERTa [38], SciBERT [39], and DistilBERT [40]. Among these approaches, [41] propose Sentence-BERT, which encodes text into high-dimensional vectors that can be efficiently compared using cosine similarity. In addition, [42] combine BERT-based contextual embeddings with topic modeling to derive topic-level representations.

Following the emergence of Transformer techniques, LLMs have become a new breakthrough in recommender systems. Recent studies mainly fall into two broad paradigms. The first treats LLMs as auxiliary components within existing recommendation frameworks [43]. Rather than replacing traditional methods, these approaches leverage the representational and reasoning capabilities of LLMs to enhance collaborative filtering and related recommendation models [44], [45]. The second paradigm places LLMs at the core of the recommendation system, focusing on designing specialized frameworks or reasoning mechanisms to strengthen recommendation performance. For example, [46] propose an instruction-tuning framework that aligns LLMs with recommendation tasks, substantially improving recommendation capability and cross-domain generalization. Another line of research investigates whether LLMs are inherently capable of performing recommendation tasks. [47] construct a benchmark with diverse and realistic user queries to systematically evaluate LLMs as personalized recommendation assistants. At the user level, [48] evaluate LLMs as personalized movie recommenders through an online user study and found that LLMs exhibit strong explainability. In conversational settings, [49] examine user experiences with LLM-based recommender systems through a diary study, highlighting their potential to support more personalized and exploratory recommendation interactions. However, the application of LLMs to academic venue recommendation remains largely underexplored, leaving open questions regarding how their semantic understanding and reasoning capabilities can be effectively utilized for this task.

3 Data and Descriptive Analysis↩︎

In our primary study, we utilize a large-scale dataset derived from the Web of Science. The dataset contains 23,609 research articles published between 2021 and 2025 across 49 journals covering statistical theory (e.g., Annals of Statistics), applied statistics (e.g., Annals of Applied Statistics), computational statistics (e.g., Statistics and Computing), and econometrics (e.g., Journal of Econometrics). These journals serve as candidates for our recommendation system. A complete list of the candidate journals is provided in Table 1.

To further investigate whether citation information can improve recommendation performance, we additionally construct a citation dataset consisting of 2,683 articles published in eight core statistical journals selected from the 49 candidate journals. An overview of the key statistics for both datasets is shown in Table 2. For each article, we collect five types of information, including title, abstract, keywords, reference titles, and reference journals, where the latter two are available only in the citation dataset. An illustrative example of the input features is presented in Table 3. In addition, for each candidate journal, we collect the corresponding Aims & Scope statement from the official journal website. These statements summarize the major research topics and application domains covered by each journal. A representative example is provided in Table 4. To ensure consistency with the current Aims & Scope of the candidate journals, both datasets are restricted to recently published articles.

To examine whether article metadata can facilitate LLM-based journal recommendation, we conduct a descriptive analysis of author-provided keywords. Specifically, we investigate the relationship between keyword frequencies and journal characteristics, including JCR categories and individual journals. Given the large number of articles in our dataset, we focus on the top-20 most frequent author-provided keywords. The corresponding JCR categories are obtained from the Web of Science based on the publishing journals. When a journal belongs to multiple JCR categories, only its primary category is retained for analysis. We first investigate the relationship between the frequencies of the top-20 keywords and JCR categories. As shown in Figure 2, three main observations emerge. First, statistics-related categories exhibit a broad and dominant presence across nearly all keywords, highlighting their foundational role in methodological research. Second, biomedical categories display a more concentrated pattern, with relatively high frequencies for keywords related to causal inference, survival analysis, and missing data, reflecting the methodological specificity of biomedical applications. Third, computer science-related categories show higher frequencies for machine learning, clustering, and classification, emphasizing a stronger focus on algorithmic and scalable data-driven approaches.

We next examine the relationship between keyword frequency and individual journals while retaining the same set of top-20 keywords. To ensure representativeness, we select 10 journals based on impact factor and diversity in research scope. As shown in Figure 3, two main observations are discussed below. First, the selected keywords are distributed unevenly across journals. For example, keywords related to causal inference are more concentrated in journals such as AoAS and JASA, whereas keywords associated with Markov Chain Monte Carlo appear more frequently in journals such as JCGS and SCmp. Second, journals with a stronger theoretical orientation (e.g., AoS and JASA) exhibit higher frequencies of keywords related to causal inference, variable selection, and Bayesian inference, while journals with a stronger computational focus (e.g., JCGS) are more strongly associated with clustering, high-dimensional data, and algorithmic modeling. Across both analyses, keyword distributions exhibit clear heterogeneity across JCR categories and individual journals, suggesting that article features contain informative signals for distinguishing research focus and supporting journal recommendation.

4 Prompt Design and Implementation↩︎

The prompt plays a central role in the recommendation process, as its design largely determines whether the model can effectively understand the task and utilize the input information. To adapt the journal recommendation task to LLMs, we design a prompt framework consisting of three instructional parts: main task specification, input description, and inference with output formatting. Within this framework, the model is required to select and rank the most suitable journals from a predefined candidate set based on the article information provided. In addition, the practical implementation of the proposed recommendation system is demonstrated at the end of this section.

4.0.0.1 Part 1: Main Task Description

The prompt begins by defining the model as an expert in statistics, econometrics, and data mining, as shown in Figure 4. The main task then instructs the model to recommend the 10 most suitable journals for a given article based solely on the information provided in the input, ranked from most to least suitable, and to provide a structured explanation for each recommendation. In our prompt design, the task is framed as evaluating the compatibility between the article content and the scope of each candidate journal. To achieve this, the prompt specifies three criteria for matching articles to journals: the methodological focus of the study, the scientific or application domain, and the intended academic audience as reflected in the terminology and keywords used in the article.

4.0.0.2 Part 2: Input Description.

The prompt incorporates two categories of input features, as shown in Figure 5. The first consists of journal and article features, including candidate journal scope descriptions that summarize the research focus and target audience of each journal, together with title, abstract, and keywords. The second category introduces citation features extracted from the reference list, including reference titles and reference journals, which provide signals about the research community to which the article belongs. However, these citation features are used only in the second stage analysis to examine the contribution of citation information and are not required from users in practical applications.

4.0.0.3 Part 3: Inference and Output Format.

The prompt specifies the constraints and output requirements to ensure controlled model behavior, as shown in Figure 6. To prevent information leakage, the model is restricted to using only the title, abstract, keywords, reference titles, and reference journals of the article, together with a predefined list of candidate journals. The output consists of two parts. In the primary setting, the model returns exactly 10 recommended journals from the candidate list, ranked from most to least suitable, using the exact journal names and no additional text. In the reasoning setting, the model generates structured explanations for each recommendation, covering four aspects: overall match, methodological alignment, application relevance, and contribution type. The reasoning outputs are used to examine whether the model recommendations reflect the content of the input article, with a detailed analysis provided in Section 6.3.

Based on the proposed prompt framework, we implement the recommendation system as a practical agent using Python 3.9 and the DeepSeek API¹. Figure 7 illustrates the workflow of the implemented system. Users are first required to provide an API key together with the corresponding API base URL to access the LLM service. The agent then takes article features, including the title, abstract, and keywords, as input and returns a ranked list of recommended journals. Note that reference information is not required from users in practical applications, since it is only incorporated in the second experiment (Section 5.2) to evaluate the contribution of citation information. The system further supports both single paper and batch recommendation.

5 Results and Enhancements↩︎

In this section, we first report the journal recommendation performance of the LLMs across five publication years. Specifically, we report the Top-3, Top-5, and Top-10 accuracies, and Mean Reciprocal Rank (MRR) for each year as well as the overall results. Top-\(K\) accuracy is widely used in information retrieval and recommendation tasks to evaluate whether the relevant item appears within the first \(K\) returned results [50]. In addition, MRR further evaluates the ranking position of the target journal by assigning higher scores when the relevant journal appears closer to the top of the recommendation list [51]. We then examine the role of citation structure by comparing these accuracy metrics across eight representative journals, highlighting how incorporating citation information affects the recommendation behavior of LLMs.

5.1 Overall Performance↩︎

In this section, we report the journal recommendation performance of DeepSeek-V3 on the base dataset described in Table 2, including the Top-3, Top-5, and Top-10 accuracies, MRR, for each publication year from 2021 to 2025, along with the average sample size and runtime. As shown in Table 6, the yearly sample size remains relatively stable over the five-year period, with an average of 4,721 articles per year. In terms of recommendation accuracy, the model achieves 40.23%, 53.67%, and 70.05% for the Top-3, Top-5, and Top-10 metrics, respectively, indicating strong overall performance across all evaluation criteria. The clear increase from Top-3 to Top-10 further suggests that DeepSeek-V3 is often able to include the target journal within a broader candidate set, even when it does not rank it among the very top positions. This is practically meaningful for journal recommendation, since authors typically consider multiple candidate journals rather than relying on a single recommendation. Therefore, the model can effectively narrow down the journal search space and provide a useful shortlist for manuscript submission. When examined across publication years, all three accuracy metrics exhibit a similar pattern, with relatively stronger performance in earlier years followed by a gradual decline in more recent years. Taking the Top-3 metric as an example, the accuracy increases slightly from 2021 to 2022, reaching a peak of 43.03%. However, from 2023 onward, the accuracy gradually declines, decreasing to approximately 37% by 2025. This trend may reflect the increasing breadth and diversity of statistical research, which leads to greater semantic overlap across journals and consequently makes journal recommendation more challenging.

While Top-\(K\) accuracy measures whether the target journal appears within a given recommendation range, it does not directly show how high the target journal is ranked within the list. To complement this perspective, MRR is further used to evaluate whether the target journal is ranked near the top of the recommendation list. Overall, DeepSeek-V3 achieves an average MRR of 0.344, indicating that the target journal is typically ranked within the top few positions of the recommendation list, approximately around the third position. Specifically, three observations can be drawn from the results. First, the MRR values are relatively high in the earlier years, reaching 0.367 in 2021 and 0.368 in 2022. This result is consistent with the higher Top-3 accuracies in these two years, suggesting that the model not only identifies the target journal more often but also tends to place it in higher ranking positions. Second, MRR gradually declines from 0.348 in 2023 to 0.311 in 2025. Compared with its peak value of 0.368 in 2022, MRR decreases by approximately 15.5% by 2025. This pattern is consistent with the decline in Top-3, Top-5, and Top-10 accuracies, indicating that recent articles are more difficult to rank accurately. A possible reason is that recent statistical research covers broader and more overlapping topics, making it harder to distinguish among closely related journals. Third, although the MRR decreases over time, it remains above 0.300 in all years. This suggests that DeepSeek-V3 still maintains a reasonable ability to prioritize relevant journals, even when the recommendation task becomes more challenging. Therefore, the MRR results support the overall conclusion that DeepSeek-V3 can provide useful journal rankings rather than only broad candidate lists. Overall, these results show that DeepSeek-V3 performs well in generating useful journal recommendations, especially in constructing a relevant candidate list, while its gradual decline in recent years also suggests room for improvement in fine-grained ranking among semantically similar journals.

5.2 Effect of Citation Structure↩︎

In scholarly communication, citations link related studies and provide informative signals regarding the research area and methodological framework of an article. Therefore, examining the role of citation information can help evaluate whether citation-based signals improve journal recommendation performance in academic publishing scenarios. In Section 5.1, the journal recommendation task is based on article-level information, including the title, abstract, and keywords as indicators of its research scope. In this section, we extend the primary experiment by incorporating citation information to investigate whether it can further improve journal recommendation performance. The experiment is conducted on the citation dataset introduced in Section 3, which consists of articles published in eight core statistical journals, i.e., AoAS, AoS, Bka, JASA, JBES, JCGS, JRSSB, and SCmp. More details can be found in Table 1. Specifically, we augment the input by incorporating the reference titles and reference journals of cited works, while keeping the task objective unchanged. The dataset statistics and input features are summarized in Table 2 and Table 3, respectively. The study is further based on a journal citation network constructed in [52].

3pt

To analyze the results, we compare the recommendation performance of DeepSeek-V3 with and without reference information. The detailed results are presented in Table 7, and three key observations are discussed below. First, most journals exhibit improved recommendation accuracy after incorporating reference information. For example, journals such as JBES and JRSSB show consistent gains across the Top-3, Top-5, and Top-10 metrics. In particular, SCmp exhibits the largest improvement, with its Top-3 accuracy increasing by 23.84%. Second, such improvements are not universal. In particular, AoAS and AoS show noticeable declines after incorporating reference information. For AoAS, which focuses on applied statistics, published papers often span diverse application domains. Their reference lists tend to cover a wide range of research areas, making it harder for the model to identify clear disciplinary signals from the reference information. For AoS, a theory-oriented journal, articles typically exhibit a clear and focused methodological profile that can already be effectively captured by the title, abstract, and keywords alone. However, their reference lists may involve a broader body of theoretical literature, introducing additional noise into the input and thereby reducing recommendation precision. Third, some broad-scope journals exhibit overall improvements while still experiencing minor declines in specific metrics. Journals such as JASA and JCGS generally benefit from the inclusion of reference information, although slight decreases are observed for certain metrics, such as Top-10 accuracy for JASA and Top-3 accuracy for JCGS. This may be attributable to their broad scope, where the reference lists introduce mixed disciplinary signals that reduce precision under specific evaluation metrics.

6 Stability, Leakage Risk, and Interpretability↩︎

In this section, we first evaluate the stability of the LLM by computing the Top-5 overlap count and Jaccard similarity across five repeated runs. Second, we examine the potential issue of information leakage and discuss the corresponding mitigation strategies. Finally, we present a representative case study to investigate whether the model can generate structured and interpretable reasoning for journal recommendation.

6.1 The Stability of LLMs↩︎

Stability is an important consideration in LLM-based recommendation systems, as model outputs may vary across repeated runs even when the inputs remain identical. To assess the stability of our approach, we randomly sample 500 papers per year from the base dataset (Table 2), resulting in a total of 2,500 papers. The recommendation process is then repeated five times using the same prompts and inputs. To quantify stability, we adopt two metrics based on the Top-5 recommendation lists. The first is the overlap count, defined as the number of shared journals between two Top-5 recommendation sets from a pair of runs. The second is the Jaccard similarity, computed as the ratio of the intersection size to the union size of two Top-5 sets. Both metrics are averaged across all papers and all pairs of runs, and the results are reported in Table 8.

The results show that the Top-5 recommendations are relatively consistent across runs, with approximately four journals overlapping on average between any pair of runs and a Jaccard similarity of around 84%. Moreover, both metrics exhibit very limited variation across different pairs of runs. The overlap count ranges only from 4.47 to 4.49, while the Jaccard similarity varies between 83.90% and 84.69%. Such small fluctuations indicate that the recommendation results are highly reproducible under repeated executions with identical inputs and prompts. Although the recommendation lists are not identical across runs, the differences are small and remain within an acceptable range. The consistently high overlap and Jaccard values suggest that the variability mainly affects a small number of journals rather than causing substantial changes to the overall recommendation set. These findings indicate a moderate to high degree of agreement among recommendation sets, showing that DeepSeek-V3 can provide reliable journal suggestions across repeated runs in practical manuscript submission scenarios.

6.2 Potential Information Leakage↩︎

A natural concern in applying LLMs to journal recommendation is the risk of information leakage. Contemporary LLMs may be pretrained on large-scale corpora that include historical publication records and journal–article associations. As a result, LLMs may contain implicit knowledge about where certain types of articles are typically published, a phenomenon widely discussed in recent studies on LLM pretraining and evaluation [53], [54]. If left unconstrained, such knowledge may bias journal recommendation results toward historical publication patterns. We emphasize that our study does not aim to replicate real-world submission outcomes or editorial decisions based on historical publishing data. Instead, we investigate whether LLMs can perform journal recommendation under controlled prompt constraints. The evaluation relies solely on contemporaneous textual information and focuses on the semantic compatibility between articles and candidate journals. These considerations highlight the importance of controlling potential information leakage in prompt design for journal recommendation tasks.

To control information leakage, the prompt design adopts a series of structural constraints. First, the prompt restricts the decision process to information explicitly provided in the input text. Each prompt corresponds to a single article and includes only the title, abstract, keywords, reference title, and reference journal, together with predefined journal scope descriptions. This design reduces the probability of information leakage at the source [55]. Second, the journal recommendation task is formulated independently of the real-world publication context. Candidate journals are represented as semantic profiles defined by their scope descriptions, and the recommendation is formulated as a compatibility assessment between the article content and journal profiles. This design restricts the association between the model and real-world entities [56]. Third, the output is strictly constrained in both content and format. The model is required to return only a ranked list of candidate journals using the exact journal names. By prohibiting free-form explanations, this design reduces the opportunity for the model to expose memorized training content, historical publication patterns, or latent journal knowledge during generation. As a result, the risk of information leakage during the recommendation process can be mitigated [57]. This constraint is used in the main recommendation experiments, whereas the reasoning analysis in Section 6.3 is conducted separately for interpretability purposes. Finally, a shielding mechanism is introduced at the prompt level. The prompt prohibits the use of prior knowledge, journal reputation or impact, historical publishing patterns, and external search, ensuring that model outputs are derived exclusively from the controlled textual inputs. This design enhances the stability of the model in following information-usage constraints under complex instructions [58].

Under these constraints, journal recommendations rely on patterns in research topics, methodological focus, and scientific or application domains reflected in the provided text. As a result, potential information leakage effects are mitigated at the prompt level. We acknowledge that fully eliminating all forms of latent knowledge related to publication would require retraining or temporally constrained pretraining of LLMs. Such model-level interventions are beyond the scope of this study. The present design therefore focuses on prompt-level controls that constrain accessible information without modifying model parameters. Accordingly, the results should be interpreted as evidence of the extent to which LLMs can approximate journal recommendation judgments based on controlled textual inputs. The goal is not to reproduce editorial decisions or historical publication outcomes. Instead, the analysis evaluates whether consistent journal recommendation can be achieved when the process is restricted to controlled textual inputs available at submission time.

6.3 The Interpretability of LLMs↩︎

Existing studies show that LLMs are capable of generating natural language explanations for their recommendations [59]. In this section, we further investigate how the LLM produces its journal recommendations by prompting it to generate reasoning outputs. Specifically, based on the Top-10 journal recommendations obtained in Section 5.1, we prompt the model to first summarize the basic information of the target article and then generate corresponding reasoning for each recommended journal, as illustrated in the prompt design shown in Figure 6. Due to space limitations, only the reasoning output for one recommended journal is presented. An illustrative example of the generated reasoning output is shown in Figure 8. In this example, the Top-1 recommendation matches the true publisher, namely Stata Journal, indicating a correct recommendation. The reasoning output provides insight into how this recommendation is reached. In the Basic Information section, the model identifies the article as an applied, software-oriented methodological contribution, which serves as the primary signal for recommendation. From the Overall Match and Methodological Alignment sections, the model further identifies the core contribution of the article as a Stata-based decomposition tool, which directly aligns with the scope of Stata Journal. This example suggests that the model is able to produce consistent and interpretable reasoning that captures the relevance between the input article and the recommended journal. More broadly, such structured reasoning outputs provide researchers with a clearer basis for evaluating the recommendations, thereby improving the interpretability of the system and helping researchers better identify suitable target journals in practice.

7 Discussion↩︎

In summary, the primary goal of this study is to examine whether LLMs can perform journal recommendation using only submission-stage information. This setting is challenging because the model does not rely on historical submission records, user behavior, citation networks, or task-specific supervised training. Instead, the recommendation is based only on the semantic alignment between manuscript information and journal scope descriptions. The empirical results indicate that DeepSeek-V3 achieves Top-3, Top-5, and Top-10 accuracy of 40.23%, 53.67%, and 70.05%, respectively, which is broadly competitive with existing methods summarized in Table 9. For instance, compared with TF-IDF (Top-3: 42%, Top-5: 50%, Top-10: 58%), DeepSeek-V3 achieves comparable performance without requiring any task-specific training or index construction. This finding suggests that LLMs can capture useful semantic relationships between manuscripts and candidate journals in a zero-shot recommendation setting. Additional experiments further validate several properties of the proposed approach, including the benefit of incorporating reference information, the stability of model outputs, and the ability to produce interpretable reasoning for each recommendation. These results suggest that LLMs provide a viable approach for journal recommendation when only submission-stage information is available.

4pt

Several limitations of the present study should be acknowledged. First, the candidate pool of 49 journals is substantially smaller than those used in prior studies, as summarized in Table 9, and it remains unclear whether the results generalize to larger candidate sets. As the number of candidate journals increases, the semantic overlap among journals may become more complex, making fine-grained ranking more difficult. Second, the current analysis focuses on a single model, DeepSeek-V3, while recommendation performance may vary across LLMs with different architectures and training strategies. Third, the proposed framework generates recommendations based solely on static textual information available at submission time, without considering other factors that may influence journal selection in practice, such as author preferences, prior submission behavior, review speed, open-access policy, and journal impact. Therefore, the findings should be viewed as evidence that LLMs can support journal recommendation through semantic matching, rather than as a complete representation of all factors involved in journal selection.

These limitations point to several directions for future work. Evaluating the framework on larger and more diverse candidate pools would help assess its scalability and generalizability. In addition, systematic comparisons across different LLMs could provide further insight into how model architectures and training strategies affect recommendation performance. Future work may also incorporate richer journal-level information, such as publication history, and review-related metadata, to better capture the practical considerations involved in journal selection. Finally, future work may explore modeling journal selection as a sequential decision-making process in order to better reflect real-world submission and resubmission dynamics.

8 REFERENCES↩︎

References↩︎