1 Introduction↩︎

LLM-driven agents are moving from chat into workflows: opening tickets, transitioning issues, updating wikis. A growing literature treats tool invocation as a first-class capability to be elicited via supervised fine-tuning or RL on real APIs [1]–[4], yet public RL benchmarks for agents target the open web [5], software engineering [6], or broad agentic suites [7], [8]. Niche enterprise SaaS workflows — narrow, schema-heavy, dominated by the “check the resource, then mutate it” pattern — get comparatively little attention.

The deeper issue is an objective mismatch: LLMs are optimised against next-token likelihood on internet-scale text, not to act inside a particular API surface. Asked to create a Jira sub-task or a Confluence page in our suite, a prompted Qwen3.5-4B knows the rough shape of the call but fills slots wrong, omits required fields, or stops after the first read — behaviours that read as fluent under a token-prediction objective and score poorly under any check that grounds the output back to the environment (our prompted-baseline scores in Section 6 bear this out). Reinforcement Learning with Verifiable Rewards (RLVR) [9]–[12] gives up reward modelling in favour of programmatic checkers wherever correctness can be inspected. Tool-use is naturally amenable: an agent’s output is a sequence of structured tool calls whose argument values and ordering can be inspected directly. We exploit that observation to train agents against outcome-level rewards in synthetic copies of the target environment, with no live API, learned judge, or human label in the loop.

1.0.0.1 Contribution.

We present a suite of five synthetic, schema-faithful Atlassian environments, the verifiable rewards that score them, prompted baselines on the same rewards, and end-to-end RL training results. Concretely:

• Environments that emulate the Jira REST v3 and Confluence v2 endpoints used in common automation flows (issue read/create, page read/create, transitions, labels), with stateful parent–child constraints and per-batch reset (Section 3).

• Verifiable reward functions that decompose into per-argument correctness, structural bonuses for the validate–mutate–verify pattern, and penalties for hallucinated or duplicate calls — all computable from the tool-call trace (Section 4).

• Prompted baselines for Qwen3-1.7B and Qwen3.5-4B, run via the HuggingFace Inference Router against the identical reward checkers used at training time (Section 6).

• An end-to-end RLVR training recipe using GRPO [12] on Qwen3 models [13] via TRL [14], with a convergence-based early-stopping callback (Section 5), reaching \(\geq 0.95\) average reward on every non-degenerate scenario in tens to low hundreds of generation batches and producing the largest absolute lift over prompted 4B baselines on schema-heavy creation tasks (e.g.\(+0.65\) on Confluence page creation).

The result is a proof of concept: the recipe works on this cheap substrate, and Section 7 foregrounds what it does not yet establish — chiefly that hand-crafted verifiable rewards do not scale to the full Atlassian public surface, and that one of our five scenarios has a saturating reward shape the prompted 4B already maxes out, which we keep as a transparency control rather than part of the headline claim.

2 Related Work↩︎

2.0.0.1 Tool-use agents.

Toolformer [2] and ToolLLM [1] train models to invoke real APIs; ReAct [4] and CodeAct [3] interleave reasoning with action. These works prove that LLMs can use tools but largely rely on supervised data; we focus on the RL phase and on training-time environments that obviate live calls.

2.0.0.2 RL with verifiable rewards.

DeepSeek-R1 [10], DeepSeekMath [12], and Tülu 3 [11] demonstrate that programmatically checkable rewards drive strong reasoning gains, primarily on math and code. Concurrent work extends the recipe to tool-use and agentic settings: ToolRL [15] argues that reward design is the load-bearing piece of tool-use RL, and Agent-RLVR [16] trains software-engineering agents with environment rewards and guidance. Our contribution sits in the same family of methods but is the environment-and-reward design for a different surface: narrow enterprise SaaS APIs where the verifier inspects nested argument values and call ordering rather than a final numeric answer, at sub-second latency and with no live API in the training loop.

2.0.0.3 Agent benchmarks and synthetic worlds.

WebArena [5], AgentBench [7], and SWE-bench [6] evaluate agents on rendered web pages and real GitHub repositories. AppWorld [8] and ToolSandbox [17] provide stateful synthetic worlds spanning many apps. Our work is narrower in surface but tailored for training: every reward signal is computed at sub-second latency from the tool-call trace, and the environment fits in a few hundred lines of Python, making it suitable for the inner loop of RL.

3 Environment Design↩︎

3.1 Design principles↩︎

The environments target four properties:

3.1.0.1 (P1) Schema fidelity.

Tool signatures and response payloads mirror the public Jira REST v3 and Confluence v2 contracts (e.g. POST/rest/api//issue expects fields.project.key, fields.parent.key, and fields.issuetype.id="10003" for a sub-task), so policies trained here speak the same wire format as their live counterparts.

3.1.0.2 (P2) Stateful but resettable.

Each environment exposes a mutable resource pool (issues, pages) so that create_* calls have side effects observable by subsequent get_* calls; a reset_synthetic_data() hook is invoked before every reward computation to restore the pristine state, preventing state-pollution across rollouts.

3.1.0.3 (P3) Determinism.

No randomness in the environment: identical tool-call sequences yield identical responses. This makes reward attribution exact and the substrate usable as a regression fixture.

3.1.0.4 (P4) Verifiability.

The ground-truth solution for each prompt is a small dictionary of expected argument values, so the reward function never needs to invoke an oracle LLM or hit a live service.

3.2 Scenarios↩︎

We instantiate five scenarios spanning Jira, Confluence, and the cross-product (Table 1). All scenarios share a common interaction pattern: the user prompt names a resource by key, the agent is expected to validate that the resource exists, mutate state with a write call, and optionally verify the result with a final read.

3.3 Example tasks↩︎

One user prompt per scenario, drawn verbatim from the training set:

• Ticket transition. “Please resolve ticket ISK1.”

• Sub-task creation. “Create a sub-task ‘Write QA test cases’ under ABC-123 and assign to admin123.”

• Page creation. “Create a page ‘Oncall Runbook - Payments’ under parent page 789 in space PAY.”

• Page labeling. “Can you correctly label page 10001?”

• Cross-product. “Create a sub-task ‘Write Release Notes’ under ABC-123 and assign to admin123. Also create a Confluence page ‘Release Notes 1.0’ under parent 789 in space PAY.”

At convergence on the cross-product scenario the agent reads the named parent issue and the named parent page, then issues both create calls with correct nested arguments; the reward function (Section 4) attributes credit for each correct argument and for the read-before-write ordering. The converged tool-call sequence is reproduced in Appendix 8 (Listing [lst:l3]).

4 Verifiable Reward Functions↩︎

For each scenario we hand-design a dense reward function that maps the list of tool calls in a rollout to a scalar in \([0, 1]\). The reward decomposes into three additive groups:

4.0.0.1 (R1) Per-argument correctness.

For each expected argument in the gold dictionary, we award a small constant (0.10 in the cross-product reward; 0.10–0.15 per slot in the single-product creation rewards) if the agent’s emitted argument matches. This is the dominant signal: e.g., in the sub-task scenario, five fields (summary, parent.key, project.key, assignee, issuetype.id) each contribute 0.10, for a correctness ceiling of 0.50. For Confluence page labeling, R1 is instead a per-label recall term: 0.10 for every gold label the agent eventually adds (Table 1).

4.0.0.2 (R2) Structural bonuses.

We award the validate–mutate–verify pattern: a get_* call before any create_* call, the correct write next, and an optional get_* after the create to confirm. The cross-product reward awards 0.05 per step; the single-product creation rewards tune these higher (typically 0.05–0.20 per step) so that an isolated structural signal still drives learning when the per-argument ceiling is smaller. Bonuses are gated by per-platform “valid state” predicates so that they cannot be earned without producing a usable mutation.

4.0.0.3 (R3) Penalties.

We subtract for behaviour that would be costly or destructive against a real API: missing required create_* calls, invalid payload shape, duplicate creates, hallucinated tool names, and excessive call counts beyond a per-scenario budget. Penalty magnitudes are scenario-tuned (e.g.\(-0.25\) for a missing create in the cross-product reward, \(-0.30\) in the sub-task reward); the cross-product values are listed in Table 2. The final reward is clamped to \([0, 1]\).

Table 2 decomposes the cross-product reward, the most complex; the single-product rewards follow the same R1/R2/R3 template with per-scenario magnitudes, and ticket-transition is the known outlier (Section 7).

5 Training Setup↩︎

We train Qwen3-1.7B [13] (ticket transition) and Qwen3.5-4B [18] (all other scenarios) with GRPO [12] as implemented in TRL [14]. We use 4–16 generations per prompt (4 for the larger-prompt-budget scenarios; 16 for ticket-transition and labeling, which expose fewer distinct prompts per batch), two optimisation iterations per batch, BF16 weights, gradient accumulation of 4, and a maximum completion length of 2048–4096 tokens depending on scenario. A reward-convergence callback stops training once the epoch-average reward delta falls below 0.01 for a per-scenario patience window (5–10 epochs), rotating checkpoints to retain the most recent two on disk.

5.0.0.1 Hardware.

The 1.7B runs fit on a single Nvidia RTX PRO 6000 Blackwell; the 4B runs use two — intentionally sized for a small on-prem or single-node cloud budget.

6 Results↩︎

6.1 Prompted baselines vs.RLVR-trained↩︎

We score prompted Qwen3-1.7B and Qwen3.5-4B against the same reward functions used during training, run via the HuggingFace Inference Router (temperature 0.7, 25-turn cap), with synthetic state reset between every task and tool-call arguments normalised to dicts so baseline scores are directly comparable. Figure 2 reports the average reward per scenario. We keep ticket-transition only as a transparency control — its reward saturates (Section 7), so the prompted 4B already scores 1.00; the four non-degenerate scenarios carry every claim below.

On the four scenarios whose rewards are non-degenerate, the RL-trained policy beats prompted Qwen3.5-4B by a meaningful margin. The largest absolute lifts come on the most schema-heavy creation tasks: Confluence page creation jumps from 0.35 to 1.00 (\(+0.65\)), Confluence labeling from 0.52 to 0.95 (\(+0.43\)), and Jira sub-task creation from 0.68 to 1.00 (\(+0.32\)). On the cross-product task, where the 4B baseline is already strong at 0.92, RL closes the residual tail to 1.00 — every one of the 12 evaluation prompts becomes a perfect rollout rather than 8 of 12. Failure modes in the prompted baselines mirror the objective mismatch flagged in Section 1: plausible-looking calls with the wrong nested key, dropped required fields, and early-stopping after the read.

6.2 Training dynamics↩︎

Figure [fig:dynamics] plots the GRPO reward trajectories (ticket-transition is omitted; its logging schema was coarser). The three from-scratch scenarios reach \(\geq 0.95\) in under 100 generation batches; the Confluence page-creation curve is a warm-started stability trace (flat near 1.0 across 16 batches). The shaded bands report per-batch min/max across the four-generation group, and contract over training as the policy stops emitting low-reward rollouts.

Appendix 8 contrasts the two dominant pre-training failure modes (early stop after the read calls; mis-shaped nested parent arg followed by a chain of duplicate create_* retries) against the converged trace. These traces are human-inspectable evidence that the gain is behavioural, not reward-hacking: the converged policy reads before it writes, shapes nested arguments correctly, and stops without hallucinated or duplicate calls — visible in the call sequence, independent of the scalar.

7 Limitations and Future Work↩︎

-.5 -3.5 Narrow prompt coverage; no held-out evaluation. Each scenario ships with 6–12 training prompts. Convergence to \(\geq 0.95\) average reward on those prompts shows that GRPO with the proposed verifiable reward can fit the training distribution; it does not measure generalisation to unseen prompts. A held-out set — same schema, new entities, summaries, and parent–child combinations — is the immediate next step.

-.5 -3.5 Reward-shape artifacts. The ticket-transition reward awards \(+0.25\) per tool call, clamped at 1.0, with no penalty for excess or hallucinated calls, so the prompted 4B already saturates at 1.00 (one rollout invoked transition_issue nine times and still maxed out). A separate type bug compares the gold transition id as an integer against a string-typed schema, denying the schema-compliant 1.7B a bonus the 4B collects. We keep these shapes verbatim — they are the rewards under which the rollouts were generated — but flag per-scenario reward-hacking audits as unfinished work.

-.5 -3.5 Reward engineering does not scale by hand. Each reward was hand-tuned through pilot GRPO loops that exposed one of three failure modes: no convergence (signal too weak), reward hacking, or premature plateau (group-relative advantages collapse on a partial solution). Scaling this to hundreds of Atlassian endpoints is the load-bearing scalability question. Much of each reward is mechanically derivable, though — R1 per-argument terms from a schema’s required fields, R3 type penalties from its type/enum constraints, leaving only R2 structural bonuses to hand-tune — so OpenAPI-driven reward synthesis, checked by pilot-run signal or an LLM-as-reward-designer, is the most important follow-up.

-.5 -3.5 Live-API transfer. Schema fidelity at the wire format does not guarantee robustness to real-world failure modes (rate limits, eventual consistency, permission errors, partial responses); the synthetic environments are intended as a cheap inner loop paired with a smaller live-tenant validation loop: record–replay of captured responses, fault injection (429/403, stale eventual-consistency reads), and a low-volume sandbox-tenant canary.

-.5 -3.5 Stronger baselines and open questions. Our prompted baselines hold the base model fixed, isolating the post-training objective; they are a controlled test, not a claim that RL is the only route. SFT on successful traces, ReAct- or plan-execute-style scaffolding, and a frontier model with a Python REPL over these environments are the next rungs — separating whether RL is necessary from merely sufficient — alongside an order-of-magnitude more prompts, larger base models, and multi-turn user simulators.

7.0.0.1 Reproducibility.

Rewards are deterministic functions of the tool-call list, so any rollout re-scores post-hoc against the same checker; Tables 1–2, Sections 3.3 and 5, and the Appendix 8 rollouts specify the substrate at the level needed to reimplement it.

8 Pre- and Post-Training Tool-Call Sequences↩︎

The listings below are the ordered tool-call sequences (tool name and argument dictionary) emitted by the policy during a single cross-product rollout, lifted from atlassian_rollouts.json with tool responses and intermediate reasoning omitted. Listings [lst:l1] and [lst:l2] are representative pre-convergence failure modes; Listing [lst:l3] is a converged success.

8.0.0.1 Failure mode A: early stop.

Listing [lst:l1] is a generation-batch 0 rollout (reward \(= 0.00\)). The policy correctly reads both parent resources, then halts before issuing the two required write calls. This is the dominant pre-training failure mode in the cross-product scenario and the one captured by the qualitative claim in Section 6.

# user: Create 'Deploy Staging' sub-task under XYZ-456
# for admin123. And a 'Staging Environment' page
# under 100 in ENG.
get_issue(issueIdOrKey="XYZ-456")
get_page(page_id="100")
# (policy ends turn)

8.0.0.2 Failure mode B: structural OK, nested args wrong.

Listing [lst:l2] is a generation-batch 2 rollout (reward \(= 0.52\)). The policy emits the full read-then-write sequence, but the first create_issue passes parent as a bare string "ABC-123" instead of {"key": "ABC-123"}, the Jira-v3 wire format. The policy then issues five more create_issue calls trying alternate shapes (string parent, dropped assignee, projectKey instead of project.key); each collects the duplicate-create penalty.

# user: Create a sub-task 'Write Release Notes' under
# ABC-123, assignee admin123. Also create a
# Confluence page 'Release Notes 1.0' under
# parent 789 in space PAY.
get_issue(issueIdOrKey="ABC-123")
get_page(page_id="789")
create_issue(fields={"project": {"key": "ABC"},
    "parent": "ABC-123",  # wrong shape
    "summary": "Write Release Notes",
    "issuetype": {"id": "10003"},
    "assignee": {"accountId": "admin123"}})
create_page(spaceId="PAY", title="Release Notes 1.0",
    parentId="789",
    body={"representation": "storage",
        "value": "<h2>Release Notes 1.0</h2>..."})
create_issue(fields={..., "parent": "ABC-123",
    "assignee": "admin123"})  # wrong
create_issue(fields={..., "parent": "ABC-123"})  # wrong
create_issue(fields={"projectKey": "ABC",
    ..., "parent": "ABC-123"})  # wrong
create_issue(fields={..., "parent": "ABC-123"})  # wrong

8.0.0.3 Success: converged.

Listing [lst:l3] is a generation-batch 53 rollout (reward \(= 1.00\)). The policy reads both parents, then issues exactly one create_issue and one create_page with correct nested shape — including the issuetype.id="10003" subtask discriminator that the system prompt names but does not demonstrate.

# user: Create a sub-task 'Write Release Notes' under
# ABC-123, assignee admin123. Also create a
# Confluence page 'Release Notes 1.0' under
# parent 789 in space PAY.
get_issue(issueIdOrKey="ABC-123")
get_page(page_id="789")
create_issue(fields={
    "project": {"key": "ABC"},
    "parent": {"key": "ABC-123"},
    "summary": "Write Release Notes",
    "issuetype": {"id": "10003"},
    "assignee": {"accountId": "admin123"}})
create_page(spaceId="PAY", title="Release Notes 1.0",
    parentId="789",
    body={"representation": "storage",
        "value": "<p>...</p>"})

References↩︎