1 Introduction↩︎

Research ideation is one of the most ambitious proposed uses of LLMs. Some controlled human studies have shown that LLM-generated ideas can match or approach those from human experts in terms of judged novelty and feasibility, demonstrating the potential of LLMs as ideation tools [1]–[3]. More recently, AI-scientist systems have already begun adapting LLMs to generate research ideas, execute experiments, and write paper drafts, bringing this capability into scientific process [4]–[10]. Despite this progress, a basic empirical question remains unanswered: What kinds of research ideas do LLMs tend to produce, and how far are current LLM-generated ideas from human researchers?

Most existing evaluations of LLM ideation judge ideas individually, using criteria such as novelty, feasibility, impact, or preference [1], [2], [11], [12]. In this work, we take a complementary distributional view. We use research taste to refer to the kinds of problems, gaps, and contributions that a source tends to produce across many comparable literature-grounded ideation contexts. Under this view, research taste concerns not only whether an individual idea is reasonable, but also what kinds of gap framings and contribution strategies repeatedly appear when the same source is asked to generate ideas under comparable constraints.

This distributional view matters because a single idea may appear novel, feasible, and coherent, while the broader set of ideas from the same source may still reflect a narrow range of research taste. Research communities generate many kinds of contributions: some papers discover a failure mode, some relax an assumption, some build a measurement instrument, some introduce a formal explanation, and others construct a system or artifact. An LLM that generates reasonable ideas one at a time may therefore still be behaviorally narrow if its outputs repeatedly identify the same kinds of gaps, use the same methodological paradigms, or rely on the same contribution templates [13]–[15]. Such concentration would affect how LLMs are used for brainstorming, literature exploration, and automated research agents, even when many individual outputs appear coherent.

We study this question through a constrained literature-grounded ideation task. Each example consists of a small set of closely related prior papers, represented by their titles and abstracts. The target output is a new research idea, separated into a motivation and a method. This differs from open-ended ideation such as write an idea about topic X. Grounding each generation in a small related-work context makes the task comparable across human and LLM outputs. The human idea is the idea realized in the real paper, while the LLM idea is generated from the reconstructed local context that seemingly reasonably preceded that paper. This setting also reduces the chance that differences are driven only by broad topic choice or by an LLM’s preferred generic paper format.

To build the evaluation framework, we collect real papers in machine learning and natural science domains. For each paper, we use a strong LLM-assisted extraction pipeline to identify the paper’s core human idea and to reverse-engineer several closely related prior works from which that idea can be understood. We retrieve these prior works and prompt evaluated LLMs to produce a new motivation and method from the same prior-work context. This yields paired human and LLM idea corpora over the same inputs.

We compare these corpora at the distributional level through a two-dimensional view of research taste. One dimension characterizes how a proposal frames the underlying research opportunity, ranging from identifying missing explanations or overlooked failures to exposing structural disconnects or limitations in existing understanding. The other captures the style of intellectual contribution through which the opportunity is developed into a research idea, including analytical, constructive, integrative, and exploratory forms of proposed methods. We introduce a taxonomy along these two axes, constructed by human experts through a review of research guidance from NSF, NIH, AHRQ and DARPA, and then iteratively refined using a held-out set of papers to ensure applicability across both machine learning and natural science domains. We apply the taxonomy at scale using an LLM annotator validated against independent human judgments.

We find that LLM-generated ideas occupy a substantially narrower region of the research-taste taxonomy than human ideas. This narrowing is most visible in an ideation pattern centered on connection, where model ideas more often frame the motivation as a need to link previously literatures, methods, or evidence streams, and more often develop the method by integrating, reconciling, or unifying existing approaches. In our evaluation, only 12.1% of human ideas motivated by the pattern of connection, and only 5.1% use synthesis or unification as the central method paradigm. By contrast, across the nine main evaluated LLMs, the corresponding rates range from 47.1% to 64.2% and from 22.5% to 38.7%, respectively. Human ideas also exhibit consistently higher normalized entropy on both taxonomy axes. This pattern remains stable across model families and scientific domains, indicating that current LLM ideation is disproportionately concentrated around integrative and synthesis-oriented types, while human research ideas span a substantially broader range of opportunity patterns and methodological paradigms.

2 Related Work↩︎

2.0.0.1 LLMs for Research Ideation.

Recent work has explored LLMs for scientific ideation, including generating, refining, and evaluating research hypotheses and directions. Early studies show that directly prompted LLMs can produce ideas perceived as highly novel, though often less feasible or well-grounded than human proposals [1]. Building on this, subsequent work has explored iterative refinement, retrieval-augmented generation, and search-based ideation pipelines [2], [16], [17]. Other approaches ground generation in external scientific signals such as retrieved literature, knowledge graphs, or emerging research trends [18]–[20]. Multi-agent collaboration has also become a common paradigm for improving idea diversity and critique through simulated scientific discussion [21], [22]. Benchmarks are proposed to evaluate dimensions including novelty, feasibility, and impact of generated ideas [23]–[25]. Our work differs from this line of research by studying whether the overall distribution of LLM-generated ideas resembles the distribution of ideas realized in human-written scientific papers.

2.0.0.2 Gaps between Human and LLM-Generated Content.

Even when LLM outputs appear fluent and useful, research shows that they can differ systematically from human outputs. Early detection work found that neural generations contain statistical artifacts in token ranks, sampling behavior, and likelihood geometry that distinguish them from human text [26]–[28]; broader comparison corpora and distributional metrics similarly show measurable gaps between ChatGPT or neural text and human expert writing [29], [30]. These gaps become more consequential when LLM outputs are used as substitutes for human populations or human judgments. In social simulation, LLMs can reproduce some aggregate patterns while still exhibiting distortions or demographic misalignment relative to real human responses [31], [32]. In evaluation and review settings, LLM judges vary substantially across tasks and require validation against human annotations, while LLM-generated paper reviews over-focus on technical validity and under-attend to novelty compared with expert reviewers [33], [34]. Our work follows this perspective of comparison between human and LLMs, shifting the target to the distribution of research ideas.

3 Evaluation Framework for Ideation↩︎

We propose an idea-level evaluation framework to systematically analyze what kinds of research opportunities and contribution strategies LLMs emphasize or overlook during scientific ideation. We first define a literature-grounded ideation task and construct paired human and LLM idea corpora accordingly. We represent each idea through its motivation and method, annotate these ideas with a research-taste taxonomy, and analyze.

3.1 Literature-Grounded Ideation Task↩︎

Each instance contains a set of related prior works \(X_i=\{(t_{i1},a_{i1}),\ldots,(t_{ik},a_{ik})\}\), with \(t\) denoting the title and \(a\) the abstract. The target output is a research idea \(y_i=(m_i,s_i)\), where \(m_i\) refers to the motivation and \(s_i\) stands for the proposed method. Guided by the provided literature context, the prompt directs models to identify research gaps across papers and generate a coherent research idea. This task is constrained. In an open-ended ideation setting, differences between human and model outputs can be confounded by topic selection, prior knowledge, and generic paper-writing templates, making the comparison difficult to interpret [1], [23]. By anchoring both human and model ideas to the same set of related prior works, we focus the analysis on how each source identifies research gaps and constructs contributions from a shared local context.

3.2 Idea Corpus↩︎

3.2.0.1 Human Idea.

We build our evaluation data from published research papers drawn from two sources: machine learning conference proceedings from ICLR, ICML and NeurIPS published between 2023 and 2026 and Nature Communications released from 2023 to 2025, covering 71 major scientific disciplines such as physics, chemistry, and biology. For each paper, the work itself represents the human endpoint, which is the research idea originally devised by authors for publication. We extract this idea into a structured representation using an LLM-assisted pipeline. The extraction prompt asks for the innovation, departure from prior work, and key insight, then rewrites the result into a proposal-style motivation and method. We then reverse-engineer 4 to 8 highly relevant prior studies based on the extracted idea and the paper’s related-work section. For each related work, we retrieve the title and abstract. The final input contains only the prior-work titles and abstracts as main idea summaries. The mixed corpus contains 11,683 valid human ideas in total. Full prompts and additional data details are provided in 6.

3.2.0.2 LLM Idea.

For each input, we prompt LLMs to generate a new research idea in the same structured format with two components. The motivation synthesizes research gaps across the provided papers, and the method outlines a concrete high-level approach. We evaluate a set of mainstream LLM families including Claude, Gemini, GPT, DeepSeek and Qwen.

3.3 Research-Taste Taxonomy↩︎

We label each idea with a two-axis research-taste taxonomy matching an idea’s structure of motivation and method. The opportunity pattern axis corresponds to the motivation: it asks what kind of gap makes the proposal worth pursuing, such as a contradiction, missing explanation, scope mismatch, evidence gap, disconnected literature, failure risk, or resource bottleneck. The method paradigm axis corresponds to the method: it asks what high-level contribution strategy turns that gap into a paper, such as synthesis, scope extension, robustification, formal derivation, empirical mapping, artifact construction, or optimization.

We designed this taxonomy to support interpretable distributional comparison. We first reviewed guidance from NSF, NIH, AHRQ, and DARPA on research proposals and problem formulation. From these sources, we extracted recurring elements organized around two broad dimensions: the opportunity that motivates a study and the method by which the study addresses it. This initial taxonomy contained 11 opportunity elements and 9 method elements. We refined these candidate elements using a held-out set of 150 papers. For each paper, we allowed up to two closest labels on each axis and included an other option for cases not covered by the initial taxonomy. We then examined coverage and label distributions, merged near-duplicate labels, split labels that conflated motivation and method, removed labels tied to a specific domain or technical substrate, and renamed categories using terms applicable across ML and natural-science papers. The final taxonomy contains seven opportunity patterns and seven method paradigms as shown in 2. Our final taxonomy meets three requirements: categories correspond to recurring gap framings and contribution strategies, generalizable across different research domains, and were checked during human validation to avoid systematic category collapse. This ensures frequency discrepancies stem from distinct research tendencies, not field differences or wording variations. We provide details in 7.

3.4 Automated Annotation↩︎

We use an LLM as an automated annotator to label every human and LLM idea [35]–[37]. The annotator receives the taxonomy, the prior-work titles for context, and the proposal motivation and method. It returns a primary and secondary label for each axis, confidence scores, and three diagnostic scores: surface stitching, bottleneck specificity, and boilerplate. We use primary labels for distributional comparisons and diagnostic scores for mechanism analyses.

Before applying the annotator at scale, we validated it on the same held-out set of 150 papers used for taxonomy calibration. Two authors examined ideas from this set and audited three outputs from each annotation pass including the opportunity-pattern label, the method-paradigm label, and the diagnostic-score profile. As a label-level reliability measure, we compute Cohen’s \(\kappa\) [38] between the GPT-5.4-mini¹ annotation and each author’s judgment, then average over the two human-LLM pairs. The resulting \(\kappa\) are 0.84, 0.81, and 0.93, respectively. We also inspect disagreements and confusion matrices to ensure that errors concentrate in semantically adjacent labels rather than reflecting systematic category collapse. Given this high agreement, we use GPT-5.4-mini as the automatic annotator for evaluation.

4 Experiments↩︎

Our experiments ask whether LLMs generate the same kinds of research ideas as human researchers when both are conditioned on the same local literature context. We first compare human and model label distributions on the two taxonomy axes. We then use the diagnostic scores to examine whether the same gap appears as lower specificity or more template-like proposals. Finally, we study how reasoning changes the distribution and use mechanism analyses to probe why the gap appears.

4.1 Setup↩︎

4.1.0.1 Data.

We evaluate on the data introduced in 3.2. The full evaluation set contains 11,683 ground-truth human ideas from diverse fields mixed together, one per source paper, matched to generations from each evaluated model. We present overall results, while detailed analyses and results for each individual domain are provided in 10.1.

4.1.0.2 Models.

The main comparison includes nine model settings: Claude-Sonnet-4.6, Gemini-3.1-Pro, GPT-OSS-20B, GPT-OSS-120B, GPT-5.4-mini, Qwen3-8B, Qwen3-32B, DeepSeek-V4-Flash, and DeepSeek-V4-Pro [39]–[44]. For the reasoning ablation, we additionally evaluate Qwen3-8B and DeepSeek-V4-Flash with thinking mode. Each model is prompted with the same input and asked to produce a structured idea.

4.1.0.3 Metrics.

For each source and taxonomy axis, we estimate the empirical distribution of primary labels. We use three distributional measures to characterize how model-generated ideas differ from human ideas. Total Variation Distance (TVD) measures the amount of label mass that would need to be reallocated for a model distribution to match the corresponding human distribution. Jensen-Shannon Divergence (JSD) [45] provides a bounded and symmetric measure of divergence between the model and human distributions. And normalized entropy, which measures how concentrated a source is over taxonomy labels. We report TVD as, \[\mathrm{TVD}(\hat{P},\hat{Q})=\frac{1}{2}\sum_{c\in A}|\hat{P}(c)-\hat{Q}(c)|,\] where \(\hat{P}\) and \(\hat{Q}\) denote empirical label distributions over the label set \(A\), and \(c\) indexes individual labels. We report JSD using base-2 logarithms, \[\mathrm{JSD}(\hat{P},\hat{Q}) =\frac{1}{2}\mathrm{KL}(\hat{P}\Vert M) +\frac{1}{2}\mathrm{KL}(\hat{Q}\Vert M),\] where \(M=(\hat{P}+\hat{Q})/2\). Normalized entropy over the labels \[H_{\mathrm{norm}}(\hat{P}) = -\frac{1}{\log_2 |A|}\sum_{c\in A}\hat{P}(c)\log_2 \hat{P}(c).\] Lower normalized entropy indicates greater concentration over a smaller set of research moves [29], [46].

4.2 Main Distributional Gap↩︎

3 and 1 show a consistent distributional gap across tested model families. Human ideas have high normalized entropy on both taxonomy axes, above 0.92. Model distributions are generally more concentrated than the human reference distribution, especially on the opportunity axis: opportunity entropy ranges from 0.550 to 0.758, while method-paradigm entropy ranges from 0.723 to 0.879. Even the closest model on the opportunity axis, Gemini-3.1-Pro, has TVD 0.348, meaning that over a third of the distributional mass would need to move to match human outputs. On the method axis, Claude-Sonnet-4.6 is closest, but its TVD remains 0.211.

The largest gap is a shift toward bridge-and-synthesis ideas. Only 12.1% of human opportunities are labeled as fragmentation or bridge opportunities, compared with 47.1 to 64.2% for the main LLMs. The same pattern appears on the method axis: explicit synthesis or unification accounts for 5.1% of human ideas but 22.5 to 38.7% of LLM ideas. This does not mean that synthesis is never a valid research contribution. Rather, it shows that when models are given nearby papers, they frequently turn ideation into a generic move of connecting or combining prior work, while human papers distribute more mass to explanation, measurement, risk, scope, artifacts, and optimization-style contributions. The mechanism analyses in 4.5 ask whether this gap reflects many diverse forms of synthesis or a narrower template for constructing proposals. We also test alternative prompts, and this preference for organizing new ideas through bridging and synthesis does not appear sensitive to the specific wording of the generation prompt. Details in 10.3.

4.2.0.1 Full-paper context ablation.

To better approximate richer ideation settings, we also evaluate a full-paper context condition in which models can draw on the original related-work papers rather than only abstract summaries. In this setting, for each related work paper, LLMs first read the full text and produce their own summary; these model-generated summaries are then used in place of abstracts as the input context for idea generation. We run this ablation on a subset sampling 500 papers from each of the two domains and evaluate Qwen3-8B and DeepSeek-V4-Flash. ¿tbl:tab:full95context95main? shows that richer context does not move either model closer to the human reference distribution. For both models, TVD and JSD increase on both taxonomy axes under full-paper summaries, while entropy decreases or remains nearly unchanged. The full label distributions in 10.2 show the same qualitative pattern: even when models are given a richer representation of the related literature, their generated ideas continue to exhibit the same preference for organizing new proposals through bridging and synthesis.

6pt

max width=

4.3 Diagnostic Scores↩︎

We next analyze the three diagnostic scores defined in 3.4 and reported in 2: surface stitching, bottleneck specificity, and boilerplate. These annotator-assigned scores characterize the specificity and template-likeness of each proposal. Surface stitching measures whether the idea is a superficial combination of prior work, bottleneck specificity measures whether it identifies a precise mechanism or limiting factor, and boilerplate measures generic phrasing. In 2, most model outputs receive lower specificity and higher boilerplate scores than human ideas, with especially strong degradation for the Qwen models, which also have the highest surface-stitching scores and flags. Claude-Sonnet-4.6 is an exception on these diagnostic dimensions: it has slightly higher bottleneck specificity and lower boilerplate than the human baseline, while still remaining distributionally shifted in 1. Taken as supporting evidence, these annotator-based diagnostic scores suggest that the research-taste gap is not only a low-level quality issue; even polished and specific model outputs can concentrate on a narrower set of opportunity and method patterns. Surface stitching is a stricter local judgment, while bridge and synthesis labels describe the broader research type. Together, however, they support the same interpretation: LLM ideation often converges on polished combinations of prior work, and for many models those combinations are also less specific. This interpretation is consistent with the archetype analysis in 4.5, where model-heavy clusters have lower bottleneck specificity and higher boilerplate than human-heavy clusters built around replacing or decoupling a local mechanism [26], [27].

4.4 Does Extended Reasoning Help?↩︎

Reasoning is considered a paradigm that can enhance the downstream capabilities of models [47]–[49]. However, in our ideation tasks, thinking mode moves the output distribution farther from the human reference for both model settings we test as shown in 3. For Qwen3-8B, enabling thinking increases bridge opportunities from 49.7% to 71.1% and explicit synthesis from 38.7% to 52.2%. Opportunity entropy drops from 0.658 to 0.481, and TVD from humans increases from 0.382 to 0.590. This adverse effect persists across both weaker and stronger models we tested. The same direction appears for DeepSeek-V4-Flash: bridge opportunities rise from 52.2% to 59.1%, synthesis rises from 22.5% to 30.7%, and both opportunity and method TVD increase. Thinking therefore appears to sharpen the model’s preferred ideation template instead of broaden the distribution toward human taste, and further reduces the diversity of generated ideas.

4.5 Mechanism Analyses↩︎

The taxonomy tells us where model and human distributions differ, but it does not fully explain why LLMs repeatedly choose bridge-and-synthesis moves. We therefore run follow-up analyses on human ideas and two representative model sources, Qwen3-8B and DeepSeek-V4-Flash. 4 suggests that the gap is not simply that models do more synthesis. Instead, model ideas concentrate around a recognizable recipe: select a high-frequency technical concept cluster and apply a safe synthesis operation such as integrating, unifying, combining, or adapting it with another nearby concept.

4.5.0.1 Archetype Clustering.

We use GPT-5.4-mini to rewrite each proposal into a one-sentence archetype that abstracts away domain-specific details while preserving the high-level idea. We then cluster these valid archetypes using TF-IDF and MiniBatchKMeans with \(k=30\) [50], and normalize the main verb in each archetype into an operation family. The strongest model-heavy operation is integrate: it appears 7,994 times in model outputs (34.2%) but only 275 times in human ideas (2.35%), giving a model-vs-human log-odds of 3.07. Other overrepresented operations are also synthesis-like, including unify (8.2% vs. 1.9%, log-odds 1.52), design (1.5% vs. 0.3%, 1.50), merge (1.37), and adapt (1.36).

The human-heavy side looks different. Human ideas more often use local intervention operations such as replace, decouple, and formalize. Specifically, replace makes up 9.13% of human operations versus just 0.92% for models, and decouple reaches 2.33% among humans and only 0.21% for models. Two human-majority clusters are largely missing from model outputs: replace (83.3% human) and decouple (85.4%). Relative to model-heavy clusters such as integrate / model / learning, they achieve higher bottleneck specificity (2.61, 2.70) and lower boilerplate (0.60, 0.51). Beyond reduced synthesis, human ideas typically revise targeted assumptions, modules and mechanisms in local research settings.

4.5.0.2 Representation Mechanism.

We map proposals, motivations, methods, prior works, archetypes and extracted concept phrases into a shared representation space with 2560-dimensional embeddings with Qwen3-Embedding-4B. These representations show strong consensus between model outputs from the same source paper. For a given input paper, the cosine similarity between ideas from Qwen3-8B and DeepSeek-V4-Flash reaches 0.8316, while the scores stand at 0.7242 for human–Qwen pairs and 0.7829 for human–Deepseek pairs. The model pair exceeds the average human–model similarity, suggesting that distinct model families converge to similar generation patterns even when given the same context and paper-specific prior knowledge as humans.

We quantify how diffusely each proposal is positioned relative to its prior-work set. Given a proposal embedding \(p\) and prior-work embeddings \(\{w_i\}_{i=1}^K\), we compute cosine similarities \(s_i=p^\top w_i\), convert them into a similarity distribution, and compute its normalized entropy \(H\). Higher \(H\) means that the proposal is comparably close to several prior works, rather than being dominated by a single reference. We also compute \(B = p^\top c + H - (s_{(1)} - s_{(2)})\), where \(c\) is the normalized centroid of the prior-work embeddings, and \(s_{(1)}\) and \(s_{(2)}\) are the largest and second-largest similarities \(B\) increases when a proposal is close to the overall prior-work centroid, has high entropy over priors, and its top two related-work similarities are not sharply separated. Human proposals have higher values on both quantities (\(H=0.7215\), \(B=1.4662\)) than model proposals, with Qwen3-8B at \(H=0.6494\) and \(B=1.3345\), and DeepSeek-V4-Flash at \(H=0.6745\) and \(B=1.4237\). This suggests that model proposals are less broadly positioned within the available prior-work set.

4.5.0.3 Concept Enrichment.

To connect archetype operations with their semantic content, we rank the representation-analysis concept clusters by model-vs-human log-odds, keeping clusters with at least 30 occurrences. We report both core archetype concepts and all concepts, which add proposal-level TF-IDF terms. Since the pool contains one human and two model sources, we interpret enrichment by log-odds instead of raw majority.

The strongest model-enriched core clusters are reusable technical motifs: multi-omics integration has 317 occurrences across 180 records and 95.0% model share (log-odds 1.96), followed by diffusion policy (93.1%, 1.61), multimodal generation (91.2%, 1.34), in-context learning (90.2%, 1.23), test-time adaptation / adaptive optimization (89.5%, 1.19), quantization, multi-agent / LLM-agent concepts, and multimodal reasoning (86.6–88.1%). The all-concept view keeps multi-omics and diffusion policy at the top and adds physical constraints, climate models, adversarial robustness, implicit regularization, and cross-modal representations. Many representative phrases already contain integrate, combine, or unify, suggesting that these concepts serve as slots for the synthesis template.

Human-enriched clusters are more local: trajectories and tracking trajectories (63.5% human, log-odds -1.50), ligands and molecular interactions, tokenization and token importance, equivariance / inverse problems / Hamiltonian structure, entropy / mutual information, routing and prototypes, and verification concepts (44.8 to 53.2%, -0.75 to -1.09). In the all-concept view, function vectors / internal representations, policy evaluation, denoising, and geometric concepts remain relatively human-enriched. These clusters name variables, constraints, and mechanisms to intervene on, together with the operation analysis, they suggest that LLMs start from high-frequency technical concepts and wrap them in safe integration moves, while human papers more often modify, separate, or formalize a narrower local mechanism.

The mechanism analyses suggest that the gap between LLMs and human is not random, nor simply a preference for legitimate interdisciplinary work. LLM ideas repeatedly instantiate an archetype-level recipe: choose a salient technical concept cluster, then integrate or unify it with another nearby object. Human ideas are more likely to make a narrower intervention, such as replacing a brittle component, decoupling two confounded mechanisms, or formalizing a local structure. This provides one plausible account of why LLM ideas can seem reasonable while still occupying a more concentrated region of research taste.

5 Conclusion and Discussion↩︎

We introduced an evaluation framework for comparing human and LLM research ideas under shared literature-grounded inputs. By extracting human ideas from real papers, prompting LLMs on reconstructed related-work contexts, and labeling ideas with a two-axis research-taste taxonomy, we find that LLMs occupy a much narrower region of research taste than humans. The central pattern is an overproduction of bridge-like opportunities and synthesis-oriented method paradigms, supported by diagnostic and mechanism analyses showing that many model outputs are less specific, more boilerplate-like, and organized around repeated integrate archetypes. LLM ideation should be evaluated as a distributional alignment problem. A useful ideation system should also diversify how it identifies problems, constructs contributions, and departs from familiar synthesis templates. This gives future ideation systems a target: preserve the fluency and scale of LLM generation while shifting proposals toward more specific, mechanism-aware, and less template-bound research patterns.

Limitations↩︎

Our corpus is broad but still STEM-centered, so research-taste distributions may differ in social science, humanities, clinical research, or engineering design. Our task reconstructs a local literature context from related works, whereas real researchers draw on tacit expertise, failed attempts, collaborations, reviewer feedback, and long-term research agendas. Although our taxonomy and LLM annotation pipeline are human-validated, they compress nuanced ideas into discrete labels and diagnostic scores, and some proposals naturally mix multiple research moves. Finally, we evaluate a finite set of models, prompts, and one-shot settings; interactive agents, domain-specific systems, retrieval-heavy pipelines, or prompt interventions may reduce some of the observed gaps. Our results should therefore be read as evidence about current LLM ideation under a controlled literature-grounded setting, not as a claim about all possible AI-assisted research workflows.

Acknowledgments↩︎

This work was supported in part by the U.S. National Science Foundation under award No. 2541654.

6 Dataset Details↩︎

Each data point starts from a real human paper and contains its extracted idea, represented as a motivation and method, together with a reconstructed local literature context. The literature context consists of proximal prior works, each represented by title and abstract. We collect papers from ICLR, ICML, NeurIPS, and Nature Communications, then merge the human source, all model generations, and all annotations by paper ID. Rows with missing or invalid model outputs or labels are removed. The final matched evaluation set contains 11,683 papers: 5,994 ML papers and 5,689 Nature Communications papers as shown in 4.

7 Taxonomy Design↩︎

The taxonomy was designed to compare research taste rather than topic, field, or technical substrate. Before coding examples, we reviewed proposal-writing and research-gap guidance from DARPA’s Heilmeier Catechism,² NIH application guidance,³ the NSF PAPPG project-description instructions,⁴ and AHRQ guidance on identifying research gaps.⁵ These documents shaped the distinction between identifying why current knowledge is insufficient and explaining what new contribution will address that insufficiency. We then iteratively refined labels on sampled human and LLM ideas, removing categories tied to domain or method family.

7.0.0.1 Opportunity Pattern.

• Puzzle / Contradiction: a paradox, tradeoff, surprising failure, or conflicting evidence.

• Explanation Gap: a missing causal, mechanistic, theoretical, or explanatory account.

• Scope Mismatch: unrealistic assumptions, narrow regimes, unclear transferability, or boundary conditions.

• Evidence Gap: missing ways to observe, measure, benchmark, audit, diagnose, validate, compare, or accumulate evidence.

• Bridge Opportunity: disconnected literatures, theories, evidence streams, communities, or methods that could be connected.

• Failure / Risk Gap: brittleness, unreliability, bias, uncertainty, safety/privacy/security risk, or reproducibility concerns.

• Resource Bottleneck: cost, compute, time, data access, sample scarcity, experimental burden, deployment friction, usability, or scalability constraints.

7.0.0.2 Method Paradigm.

• Synthesis / Unification: bridges, integrates, reconciles, or unifies separate literatures, theories, evidence streams, mechanisms, or methods.

• Relax / Extend Scope: makes prior work function under weaker assumptions, broader scope, new regimes, noisier conditions, or more realistic settings.

• Robustification: reduces failures, brittleness, risk, uncertainty, bias, unreliability, or trustworthiness problems.

• Formal Derivation: introduces a formal model, theorem, bound, objective, derivation, proof, ontology, taxonomy, conceptual distinction, or explanatory formulation.

• Empirical Mapping: builds or applies systematic measurement, benchmarks, diagnostics, datasets, empirical maps, comparative studies, or pattern analyses.

• Artifact / System: builds a concrete artifact, software system, platform, device, material, prototype, or deployment workflow as the central contribution.

• Optimization / Search: uses optimization, search, screening, tuning, active/adaptive design, scaling, resource allocation, or efficiency strategies to discover or improve a solution.

8 Experimental Details and Configurations↩︎

8.0.0.1 Artifacts and intended use.

This work uses public scholarly artifacts: paper metadata, titles, abstracts, extracted prior-work contexts, model-generated ideas, taxonomy labels, and annotation outputs. Source papers are drawn from ML conference proceedings and Nature Communications; abstracts are retrieved from public scholarly metadata providers when available. We use these artifacts only for research evaluation. Released dataset will preserve source attribution.

8.0.0.2 Privacy and content.

The corpus does not contain private user-generated text, recruited participant records, or human-subject data. It consists of public scholarly text and derived research-idea summaries.

8.0.0.3 Model and compute setup.

Each evaluated model generates one idea per input context. The main proposal-generation prompt uses the detailed profile in 6. Local open-weight runs use temperature \(0.6\), top-\(p=0.95\), top-\(k=20\), maximum 2,048 new tokens, and one output per input; the GPT API run uses temperature \(1.0\) and a JSON-schema output constraint. Thinking-mode runs differ only in the model-side thinking flag or reasoning setting. API models are run through provider APIs; local model and embedding jobs are run on cluster GPUs. The embedding analysis uses Qwen3-Embedding-4B with 2,560-dimensional embeddings, maximum length 512, batch size 12, bfloat16, and last-token pooling on CUDA.

8.0.0.4 Analysis parameters.

No hyperparameter search is performed for the reported distributional metrics. Archetype clustering uses TF-IDF with lowercasing, English stop words, 1–2 grams, \(\mathrm{min\_df}=2\), \(\mathrm{max\_df}=0.85\), sublinear TF, and MiniBatchKMeans with \(k=30\), batch size 512, seed 13, and n_init=auto. Concept extraction uses TF-IDF over proposal text with 1–3 grams and custom stop words, while concept clustering uses MiniBatchKMeans over Qwen3 embeddings with batch size 512 and seed 13. Linear probes use 5-fold stratified cross-validation with balanced logistic regression, \(C=1.0\), liblinear, and fixed random seed 13.

8.0.0.5 Human audit instructions.

Two authors audited a stratified sample of 150 annotations. For each item, they were shown the prior-work titles, motivation, method, LLM-assigned labels, diagnostic scores, and the taxonomy definitions in 7. They independently judged whether the primary Opportunity Pattern, primary Method Paradigm, and diagnostic-score profile were acceptable under the codebook, then disagreements were inspected through confusion matrices. No external annotators were recruited or paid.

8.0.0.6 AI-assistant use.

LLMs are part of the research pipeline: they assist with idea extraction, related-work reconstruction, proposal generation, annotation, and archetype rewriting. The manuscript reports these uses in the method and prompt appendices. We use LLMs for writing polishing and grammar checking.

9 Prompts for Idea Generation and Annotation↩︎

We show prompt templates used in the pipeline. Bracketed fields are filled separately for each paper.

10 Additional Distributional Analyses↩︎

10.1 Domain-Specific Results↩︎

We present the main distributional comparison separately for the Machine Learning corpus and the Nature Communications corpus in 6 and 7.

10.2 Full-Paper Context Ablation↩︎

The main experiments represent each proximal prior work with its title and abstract. To test whether the bridge-and-synthesis pattern is an artifact of this compressed context, we run a full-paper context ablation on 1,000 inputs: 500 sampled from the Machine Learning corpus and 500 sampled from the Nature Communications corpus. For each related-work paper, the model first reads the full text and produces a compact summary with detailed motivation, method and insight. We then use these model-generated full-paper summaries in place of abstracts as the context for idea generation, keeping the downstream annotation pipeline unchanged.

¿tbl:tab:full95context95ablation? compares the original abstract-context condition with the full-paper-summary condition on the same subset for Qwen3-8B and DeepSeek-V4-Flash. The richer context does not remove the bridge-heavy opportunity pattern. For Qwen3-8B, bridge opportunities increase from 456 to 551 cases; for DeepSeek-V4-Flash, they increase from 489 to 521 cases. The listed method-paradigm categories also do not show a compensating shift toward formal derivation, empirical mapping, artifact construction, or optimization.

7pt

max width=0.92

10.3 Prompt Ablation↩︎

The main generation prompt asks models to analyze a set of prior papers, identify research gaps and opportunities, and propose one coherent idea. This wording may itself encourage models to connect papers into a synthesis-style proposal. We therefore run a prompt ablation that relaxes this constraint. We use more neutral terms like generate and describe instead of expressions that might affect the paradigm of the model ideation.

¿tbl:tab:prompt95ablation? reports the resulting count changes on the full matched evaluation set. The ablation changes the distribution, but it does not eliminate the main qualitative pattern. For Qwen3-8B, bridge opportunities decrease from 5,807 to 5,247 cases, while failure / risk gaps increase from 987 to 1,238 cases; nevertheless, bridge opportunities remain the largest listed opportunity category. For DeepSeek-V4-Flash, bridge opportunities increase from 6,094 to 6,368 cases under the relaxed prompt. On the method axis, the relaxed prompt moves some mass toward artifact / system and optimization / search categories, especially for Qwen3-8B, but the shifts are modest relative to the overall corpus size. These results suggest that prompt wording affects the exact label mix, while the tendency to organize generated ideas around bridge-like opportunities remains stable across prompt variants.

7pt

max width=0.92

References↩︎