1 Introduction↩︎

In a live coding set, the performer types "neon city at 2am" and the ambient drone keeps playing. Five seconds later, the new configuration resolves—bright, electronic, fast—and the sound crossfades into it. The performer nudges brightness down two steps. The city gets darker. They switch rhythm to "heartbeat". The pulse slows. None of this required stopping the music, waiting for a GPU, or understanding DSP.

Neural text-to-audio systems such as MusicGen [1], MusicLM [2], and Stable Audio [3] produce high-quality audio from text. However, such systems typically return generated audio rather than interpretable synthesizer parameters; CTAG explicitly frames this as a tweakability problem for neural audio systems [4]. Azimi and Zareei [5] fine-tuned Stable Audio Open for live improvisation, demonstrating that even with GPU acceleration, generation latency and unpredictable outputs remain challenges.

Our SDK addresses this tension by generating interpretable parameter configurations for a procedural synthesizer rather than raw audio. Its live generator architecture decouples instruction resolution from audio emission—the current configuration keeps sounding while the next one resolves in the background. This brings LLM-powered text-to-music into live performance without audible interruption.

For the performer, the key affordance is persistent steerability: a prompt establishes a scene, but the musical work happens through named parameter edits—darkening the spectrum, switching rhythm, widening the space—while the stream continues. Text becomes a playable control surface, not a one-shot trigger.

The system is not a neural audio generator: language models resolve prompts into symbolic synthesizer configurations, and a deterministic procedural engine renders the sound—less timbrally general than neural text-to-audio, but inspectable, steerable, and stable during performance.

We contribute: (1) a live generator architecture that hides resolution latency behind continuous audio, enabling the SDK to function as an instrument runtime rather than a generation endpoint; (2) a 34-field configuration schema whose categorical labels mean most valid combinations sound musically coherent—a mapping-design contribution in the sense of Hunt and Kirk [6] and Magnusson [7]; and (3) an open-source release.¹ A companion ACM SIGGRAPH 2026 talk addresses the broader system architecture and benchmark framing; the present paper focuses on the instrument contribution: the live generator as performance runtime, the configuration schema as mapping design, and the prompt–listen–steer workflow.

2 Related Work↩︎

2.0.0.1 Text-to-audio instruments.

Azimi and Zareei [5] fine-tuned Stable Audio Open for improvisation, generating raw audio fragments. Shepardson et al. [8] used text notation to control neural vocal synthesis in Tungnáá. Both systems synthesize neural audio directly, with differing compute requirements and limited post-generation editability. Our SDK generates structured parameters instead—trading timbral fidelity for sub-second latency, direct editability, and reproducible output.

2.0.0.2 Text-to-synthesizer mapping.

CTAG [4] maps text to a modular synthesizer with 78 parameters via iterative optimization, but its search runs at generation time. SynthScribe [9] supports synthesizer sound retrieval, creation, and modification through multimodal text/audio tools. Our system front-loads LLM generation into dataset construction and retrieves complete multi-layer configurations at runtime. Latent-space instruments like Stacco [10] and Latent Mappings [11] navigate continuous neural spaces through physical gesture; our SDK navigates a discrete parameter space through language.

2.0.0.3 Programming frameworks.

ChAI [12] and ChuMP [13] added interactive AI tools and modular package management to ChucK.

3 The Instrument↩︎

3.1 Design Goals↩︎

Immediacy (G1): sub-second text-to-sound. Controllability (G2): all parameters exposed as named fields with human-readable labels. Reproducibility (G3): a given configuration and seed yield identical output. Learnability (G4): a single-line entry point that scales to full programmatic control. Continuity (G5): audio never stops during configuration resolution.

3.2 The Live Generator↩︎

The SDK’s core contribution is the live generator—an async streaming interface where the instrument is always sounding. A performer (or game engine, installation, or MIDI controller) writes a Python generator that yields instructions: text prompts, absolute configurations, or relative parameter adjustments. The SDK continuously emits audio chunks from the current configuration. When a new instruction arrives, it resolves in the background—calling an LLM, querying the retrieval index, or applying a parameter update—while audio emission continues uninterrupted. Once resolved, the SDK crossfades into the new configuration.

The sleep duration controls how long each configuration plays, and new instructions can be yielded at any time—including while a previous resolution is still pending. Figures 2 and 3 show the timing relationship between the generator, resolver, and audio output for both backends. The SDK queues instructions and resolves them in order. Rapid yielding can accumulate pending instructions with slow backends; with the fast backend, rapid yields produce transitions too quick to sound musical. In practice, performers quickly learn to pace yields to the backend’s resolution speed.

The SDK supports three backends. The fast mode (default) uses embedding retrieval—sub-second after warm-up, fully reliable, no network needed. The external mode routes to any LLM via the developer’s API key, with seamless playback despite network latency; failed calls fall back to fast mode. A local expressive mode runs a 270M fine-tuned model on-device (experimental). A web-based UI demonstrates the SDK as a playable instrument: it displays each configuration’s co-generated title and color palette alongside the parameter editor, with background particle effects responsive to playback state—creating an audiovisual performance surface.

3.3 Configuration Schema as Mapping Design↩︎

The schema contains 34 fields in five groups: global parameters (8: tempo, root, mode, brightness, space, density, motion, attack), six orchestration layers (bass, pad, melody, rhythm, texture, accent—each selecting a style from a curated set), spatial/texture (5: stereo, depth, echo, human, grain), melody generation (10), and harmony (5). All fields use categorical labels, designed for both human interpretability and reliable LLM generation—an LLM can produce "bright" far more reliably than a meaningful float like 0.73.

Eight of these fields use ordered labels that support relative stepping: a performer can issue Step(+1) or Step(-1) to move along the label sequence, clamping at boundaries. The remaining fields use style selectors, booleans, or bounded values accepting absolute assignments only. Each resolved configuration also includes co-generated metadata—a descriptive title, three color palettes, and a brief justification of the parameter choices.

The schema was iteratively refined through an AI-human design loop. Categorical labels make each level perceptually distinct and enable reliable LLM generation. Constrained style sets reduce discordant combinations. The design goal: most valid configurations should sound musically coherent. The schema is biased toward coherence to support exploratory play—a decision in Magnusson’s sense [7], not a limitation.

4 Retrieval-Based Configuration↩︎

Given a coherent parameter space, the remaining challenge is mapping text to configurations quickly enough for live use. The default fast backend uses a retrieval map of LLM-generated configurations curated offline. We distilled $$10,500 unique scene descriptions from the Common Pile [14], an openly licensed text corpus, prompted Gemini 3 Flash Preview to generate five candidate configurations per scene as valid schema instances, scored each rendered audio by measuring text–audio semantic similarity using LAION-CLAP [15], a contrastive language–audio model trained on diverse audio–text pairs, and selected the best, embedded each scene with a sentence transformer [16], and exported the map. At runtime, the nearest neighbor is returned—approximately one second on CPU.

5 Design Observations↩︎

The author used the instrument across multiple composition sessions. The live generator excels at gradual scenic evolution: yielding "quiet midnight hush", waiting, then stepping brightness down and switching rhythm to "heartbeat" produces a coherent ambient environment that evolves naturally. This two-level control—text for macro-level scene changes, parameter stepping for micro-level refinement—emerged as the instrument’s natural performance idiom. When using an external LLM backend, the 5–8 s wait for a new scene to resolve is noticeable but not disruptive—the current soundscape keeps playing, and the crossfade when it arrives feels like a natural transition rather than a technical artifact. The instrument struggles with sharp stylistic pivots and cannot match timbral specificity for prompts like “upright bass.”

Five lay participants each listened to four text-generated soundscapes paired with author-steered variations in randomized order, without being told which was which, and were asked to describe what they heard and whether changes felt intentional. All five independently described output as sounding musically intentional rather than random, offering informal support for the schema design goal. Limitations identified: no individual layer volume control, restricted stylistic range for culturally specific music, and occasional audio artifacts in dense configurations. These comments are treated as informal design feedback rather than evidence of general audience response: the sample was small, participants were lay listeners, and no statistical claims are made from these remarks.

A technical benchmark on 200 held-out prompts showed embedding retrieval scoring higher on text–audio semantic similarity than a random valid configuration, suggesting that retrieval adds semantic targeting beyond the schema’s baseline coherence under this proxy. This metric is a system-reliability and alignment proxy, not a perceptual-quality measure. LAION-CLAP also informed retrieval-map construction, so retrieval has an expected advantage on this metric; human perceptual validation is planned.

6 Discussion↩︎

G1 (immediacy) and G3 (reproducibility) are met. G5 (continuity) is met—in a 200-prompt benchmark, the embedding backend resolved all prompts successfully ($\(0.3 s config generation,\)\(1.2 s total with synthesis); the external LLM backend resolved 178/200 (\)$5.6 s config generation, 89% success rate), with failed calls falling back to the fast backend to maintain uninterrupted playback. G2 (controllability) is partially met—categorical labels provide precise steering, but retrieval coverage gaps cause occasional semantic mismatches. G4 (learnability) is informally supported but formally unevaluated.

The live generator reframes text-to-music from a one-shot generation into an ongoing stream—decoupling resolution from emission generalizes beyond music to any latency-sensitive domain. The CPU-only, reproducible design responds to Clarke et al.’s [17] concerns about longevity, reproducibility, documentation, and the resource demands of deep generative models in NIME practice—the default backend requires no inference at runtime, amortizing LLM cost into a one-time dataset construction step. The open-source SDK, extensible schema, and shared retrieval dataset contribute to the NIME 2026 Communities theme: the instrument’s parameter space is a shared vocabulary that other developers can extend with new style sets, new retrieval entries, or new synthesizer backends—and the live generator architecture provides the runtime for any such extension to be performed live.

6.0.0.1 Musical outcomes and limitations.

The strongest outputs are ambient, cinematic, and loop-adjacent soundscapes where texture and harmonic color matter more than authored melody. It is less successful for genre imitation or timbral specificity. This trade-off privileges predictable steering and live continuity over timbral range.

7 Ethical Standards↩︎

This work follows NIME ethical expectations. Informal listener feedback was solicited from consenting adult participants on coherence, controllability, and musical character; no personal data was collected, and the feedback is reported as informal design input, not a powered study. The system generates procedural audio rather than imitating recordings or performers; the retrieval dataset uses an openly licensed corpus [14] and releases structured configurations and rendered examples rather than copyrighted music recordings. The CPU-first default backend amortizes LLM cost across all use, addressing NIME’s guidelines on lightweight models and AI environmental impact. AI coding tools assisted with codebase and schema design, and grammatical correction; the open-source release includes an AI_DISCLOSURE.md documenting their scope. A supplementary video demonstrates the live generator.

8 Supplementary Material↩︎

This appendix supplements the main paper with additional technical detail and artifact links. The open-source SDK and demo are available at https://github.com/prabal-rje/latentscore and https://latentscore.com.

The subsections below cover: §8.1 the full configuration schema including co-generated metadata, §8.2 benchmark results across six named controllers, §8.3 example outputs with color palettes, §8.4 dataset and artifact availability, and §8.5 supplementary figures.

8.1 Configuration Schema Reference↩︎

Each output produced by the system is a JSON object with four top-level fields:

• thinking — a short generated rationale summarizing the model’s parameter choices; used as metadata during dataset construction and omitted from the performer’s direct control surface.

• title — a short evocative name (\(\leq\)​6 words) summarizing the vibe, displayed in the UI as a human-readable label for the current soundscape.

• config — the 34-field synthesis configuration (Table 1).

• palettes — three color palettes, each containing five hex colors with size weights (xs–xxl). These drive the UI’s visual feedback: background gradients, particle colors, and ambient lighting shift in real time as configurations change. The co-generation of audio parameters and visual palettes from a single text prompt makes the system an audiovisual instrument.

The web-based UI displays the title, renders the active palette as a color-shifting background, and overlays particle effects responsive to playback state. Together with the text input and parameter steering buttons, this creates a playable audiovisual performance surface where text prompts produce coordinated shifts in sound, color, and motion.

Synthesis Configuration Fields↩︎

The config object contains 34 fields in five groups. Eight fields marked \(\star\) support ordered stepping: Step(+1) or Step(-1) adjustments that clamp at boundaries.

8.2 Benchmark Results↩︎

We benchmarked six controllers on 200 held-out test prompts (TEST split), preventing data leakage into the embedding lookup map.

Controllers↩︎

1. Random Baseline — uniformly random valid configuration (ignores prompt).

2. Base Untrained — unsloth/gemma-3-270m-it (Gemma 3, 270M parameters), no fine-tuning, constrained via Outlines grammar-guided decoding.

3. SFT Fine-tuned — same Gemma 3 270M model after supervised fine-tuning on TRAIN split with LoRA, constrained via Outlines.

4. Claude Opus 4.5 — Anthropic’s commercial API, structured output mode.

5. Gemini 3 Flash Preview — Google’s commercial API (gemini-3-flash-preview), structured output mode.

6. Embedding Lookup — default fast backend; nearest-neighbor retrieval using all-MiniLM-L6-v2 sentence embeddings (Reimers & Gurevych, 2019).

Results Summary↩︎

Key observations:

• Embedding Lookup achieves the highest CLAP (0.163) with lowest total latency ($$1.2 s) and 100% validity. This is the default backend for live performance.

• Gemini 3 Flash Preview produces competitive CLAP (0.158) on valid outputs but fails schema validation 11% of the time; in live use, such failures trigger automatic fallback to the fast backend.

• Claude Opus 4.5 achieves 100% validity but scores below random on CLAP (0.137 vs.) with $$12.6 s mean latency.

• The fine-tuned 270M model only marginally exceeds random on CLAP (0.140 vs.) at $$100 s mean generation time, suggesting mode collapse at this model scale.

• Synthesis time is configuration-dependent: richer configs from Embedding Lookup (0.79 s) and Gemini (0.73 s) render slower than simpler ones from Base (0.38 s) and Random (0.56 s).

CLAP circularity. The retrieval map was constructed with Best-of-5 CLAP-scored selection. The same metric is used for evaluation; the retrieval advantage is partly expected. Human perceptual evaluation is planned.

8.3 Example Configurations↩︎

Three complete outputs from the retrieval dataset, each showing the full structure: thinking (the LLM’s justification of its choices), title, all 34 config fields, and three co-generated palettes with color swatches.

“Warm, comforting recollection.”

Title: Hazy Echoes of a Gentle Memory CLAP: 0.569 Split: SFT-Train

Thinking: “A warm, comforting recollection calls for slow tempo and soft dynamics—like something remembered through a haze. F major gives a gentle warmth. Dark brightness with large space creates the sense of a memory filling a room. Vinyl crackle as texture reinforces the nostalgic, analog quality. Chime accents suggest fleeting moments surfacing. Ambient drift pad with sustained bass provides the continuous, enveloping foundation. Sparse melody with stepwise motion and weak cadences lets the piece breathe without urgency. Lush chord extensions add harmonic richness without tension.”

{
  "tempo": "slow", "root": "f", "mode": "major",
  "brightness": "dark", "space": "large", "density": 3,
  "bass": "sustained", "pad": "ambient_drift",
  "melody": "contemplative", "rhythm": "none",
  "texture": "vinyl_crackle", "accent": "chime",
  "motion": "slow", "attack": "soft",
  "stereo": "wide", "depth": true,
  "echo": "subtle", "human": "natural", "grain": "warm",
  "melody_engine": "procedural", "phrase_len_bars": 8,
  "melody_density": "sparse", "syncopation": "straight",
  "swing": "none", "motif_repeat_prob": "sometimes",
  "step_bias": "step", "chromatic_prob": "none",
  "cadence_strength": "weak",
  "register_min_oct": 3, "register_max_oct": 5,
  "tension_curve": "arc", "harmony_style": "ambient",
  "chord_change_bars": "slow", "chord_extensions": "lush"
}

Palette 1:  #f4e1d2 (xxl)  #e9c46a (xl)  #f4a261 (lg)  #e76f51 (md)  #264653 (sm)

Palette 2:  #fdfcf0 (xxl)  #faedcd (xl)  #d4a373 (lg)  #ccd5ae (md)  #e9edc9 (sm)

Palette 3:  #efd3d7 (xxl)  #feeafa (xl)  #dee2ff (lg)  #8e9aaf (md)  #cbc0d3 (sm)

“Need for continual self-examination.”

Title: Infinite Internal Reflection CLAP: 0.556 Split: SFT-Train

Thinking: “Self-examination is inward-looking, contemplative, cyclical. Dorian mode in D gives a modal ambiguity—not quite sad, not resolved. A drone bass anchors the introspection; ambient drift pad creates a sense of floating within one’s own thoughts. No rhythm—this is internal, not measured by external time. Pad whisper texture adds subliminal movement beneath the surface. Piano note accents surface like individual thoughts emerging. Slow motion, soft attack, medium stereo—contained but not claustrophobic. Waves tension curve for the recurring nature of self-questioning. Often-repeating motifs suggest obsessive return to the same themes.”

{
  "tempo": "slow", "root": "d", "mode": "dorian",
  "brightness": "dark", "space": "large", "density": 3,
  "bass": "drone", "pad": "ambient_drift",
  "melody": "contemplative", "rhythm": "none",
  "texture": "pad_whisper", "accent": "piano_note",
  "motion": "slow", "attack": "soft",
  "stereo": "medium", "depth": true,
  "echo": "subtle", "human": "natural", "grain": "warm",
  "melody_engine": "procedural", "phrase_len_bars": 8,
  "melody_density": "sparse", "syncopation": "straight",
  "swing": "none", "motif_repeat_prob": "often",
  "step_bias": "step", "chromatic_prob": "light",
  "cadence_strength": "weak",
  "register_min_oct": 2, "register_max_oct": 4,
  "tension_curve": "waves", "harmony_style": "ambient",
  "chord_change_bars": "slow", "chord_extensions": "lush"
}

Palette 1:  #1a1a2e (xxl)  #16213e (xl)  #0f3460 (lg)  #533483 (md)  #e94560 (sm)

Palette 2:  #2c3e50 (xxl)  #34495e (xl)  #7f8c8d (lg)  #95a5a6 (md)  #bdc3c7 (sm)

Palette 3:  #000000 (xxl)  #141414 (xl)  #1f1f1f (lg)  #3d3d3d (md)  #ffffff (sm)

“The mood is anticipatory.”

Title: The Growing Momentum of Expectation CLAP: 0.554 Split: SFT-Train

Thinking: “Anticipation means something is building. Medium tempo with slow motion creates a measured but unstoppable forward movement. Dorian mode for tension without darkness. Bright brightness—anticipation is alert, not gloomy. Sub-pulse bass provides the physical heartbeat of waiting. Heartbeat rhythm reinforces this literally. Rising melody enacts the building expectation; ramp tension curve ensures it never resolves too soon. Cinematic pad and cinematic harmony style for the grand-scale, almost filmic sense of approaching something. Sparse melody density with light syncopation—not rushed, but not metronomic. Wide stereo and large space for the sense of something vast about to happen.”

{
  "tempo": "medium", "root": "d", "mode": "dorian",
  "brightness": "bright", "space": "large", "density": 4,
  "bass": "sub_pulse", "pad": "cinematic",
  "melody": "rising", "rhythm": "heartbeat",
  "texture": "shimmer_slow", "accent": "blip",
  "motion": "slow", "attack": "medium",
  "stereo": "wide", "depth": true,
  "echo": "subtle", "human": "tight", "grain": "clean",
  "melody_engine": "procedural", "phrase_len_bars": 4,
  "melody_density": "sparse", "syncopation": "light",
  "swing": "none", "motif_repeat_prob": "sometimes",
  "step_bias": "balanced", "chromatic_prob": "light",
  "cadence_strength": "medium",
  "register_min_oct": 3, "register_max_oct": 6,
  "tension_curve": "ramp", "harmony_style": "cinematic",
  "chord_change_bars": "slow", "chord_extensions": "lush"
}

Palette 1:  #2c3e50 (xxl)  #34495e (xl)  #7f8c8d (lg)  #bdc3c7 (md)  #ecf0f1 (sm)

Palette 2:  #1a1a2e (xxl)  #16213e (xl)  #0f3460 (lg)  #e94560 (md)  #ffffff (sm)

Palette 3:  #000000 (xxl)  #14213d (xl)  #fca311 (lg)  #e5e5e5 (md)  #ffffff (sm)

Note how palette color temperatures correlate with audio parameters: warm/nostalgic vibes produce cream and amber tones; introspective vibes produce deep navy, indigo, and near-black; anticipatory vibes produce cool grays with accent pops. The thinking field is a short generated rationale captured during dataset construction; it is retained as metadata about parameter choices and is not part of the performer’s control surface.

8.4 Dataset and Artifact Availability↩︎

The dataset and benchmark artifacts are archived through public Hugging Face repositories; the repository file listings provide the authoritative artifact record rather than the automatically generated dataset preview.

• Dataset artifacts: https://huggingface.co/datasets/guprab/latentscore-data

• Benchmark artifacts and audio: https://huggingface.co/datasets/guprab/latentscore-clap-benchmark

• Model artifacts: https://huggingface.co/guprab/latentscore-gemma3-270m-v5-merged

8.4.0.1 Availability.

The SDK, retrieval-map dataset artifacts, benchmark outputs, model artifacts, and demo materials are released through the public repository and the Hugging Face file listings linked above. Code and samples are available at https://github.com/prabal-rje/latentscore.

8.5 Supplementary Figures↩︎

References: Reimers & Gurevych (2019), “Sentence-BERT,” EMNLP. LAION-CLAP: Wu et al.(2023), ICASSP. Common Pile: Kandpal et al.(2025). Gemma 3: Google (2025). Outlines: Willard & Louf (2023). Full references appear in the bibliography above.

References↩︎