1 Introduction↩︎

Dance Dance Revolution (DDR), and its modern counterpart In the Groove (ITG), are rhythm games consisting of songs and charts. Players hit one of four arrows (up, down, left, right) as directed by a chart written to be played in time with an accompanying musical track. Player performance is determined by step accuracy, with steps closer to the assigned timing receiving better evaluations. A single chart may have multiple variations to accommodate difficulty settings determined by the stepcharter, and are associated with a coarser verbal difficulty level (Beginner, Easy, Medium, Hard, Expert) as well as a finer integer difficulty assignment. While verbal difficulties are expected to correlate appropriately for each song’s charts, they do not necessarily correlate across distinct songs. That is, the ‘Hard’ difficulty for one song may be more difficult than the ‘Medium’ difficulty for another. The fine integer difficulty is expected to be objective. For example, charts rated ‘10’ should be harder than charts rated ‘8’.

Charts and music are expected to correspond in a rhythmically, musically, and thematically fitting manner. While not a firm rule, steps should generally be placed where musical activity is most prominent. Depending on the desired difficulty, speed of the song (BPM), and overall styling of the music, more or fewer steps, or more or fewer technical steps may be appropriate to place. Such demands constitute a rich and challenging task for the application of modern machine learning (ML) methods.

In 2017, the authors of DDC [1] introduced a bipartite model structure for step placement and selection, using a hybrid convolutional/LSTM (CNN-LSTM) architecture. This model has been referred to as state-of-the-art as recently as 2023 [2]. In 2025, we proposed DDCL [3], a similarly bipartite model employing \(\Delta\)-beat based audio modulation and ConvLSTM encoding. DDCL improved dramatically on generation quality compared to DDC, but did not include any transformer based processing.

Here we introduce ITGPT, a major departure from both DDC and DDCL. We eschew the use of LSTM in favor of a transformer based architecture. While CNN-LSTM and ConvLSTM architectures are effective at encoding local musical context, moving to a transformer based architecture allows us to better encode global context and integrate features such as difficulty and BPM. We maintain the shift to \(\Delta\)-beat based sampling for the step placement process, as well as the integration of audio features into the step selection process. We introduce an encoder which takes a much larger context window at a maximum of \(2000\) beats by default, more than the length of most songs. We maintain the integration of fine difficulty and BPM in the step selection process. BPM detection is handled through the same ArrowVortex [4] algorithm used by human stepcharters, and difficulty is provided at the time of training and generation. We introduce a diagnostic model during the training process for the step placement model, which enforces alignment with BPM and difficulty parameters. Following DDC and DDCL, the ITGPT step selection model operates autoregressively, using each generated step as input for the next. We implement a much larger context window by default at 500 steps, larger than DDC (64 steps) and DDCL (256 steps). We improve generation times over DDCL by around \(7\times\) by preprocessing the full audio stream for step selection before generation. We introduce a ramping multi-step training curriculum similar to [5] in order to better structure the step selection latent state for pattern based predictions.

The ITGPT step placement model outputs 48 evenly distributed steps per beat, while our step selection model outputs 256 possible step combinations for each placed step. We evaluate our model’s performance on an expanded version of the Fraxtil dataset used in the original DDC paper and DDCL papers, adding new charts created by Fraxtil which were not included in the original dataset. While DDCL was a significant improvement over the original work in DDC, ITGPT is a further improvement over DDCL still, providing significant improvement in the step placement and selection processes.

2 Related Work↩︎

Dance Dance Convolution (DDC) [1] introduced the CNN-LSTM pipeline and the bipartite generation process for rhythm game charting. Our work in DDCL [3] introduced beat modulation and ConvLSTM to this pipeline. Similar approaches have been undertaken for the game Taiko no Tatsujin (TaikoNation [6]), osu! [7] [8] [9], BeatMania (GenerationMania [10]), and LoveLive [2]. An approach extending DDC to generate lower difficulties can be found in DDG [11] through creating a ‘gradation’ of generation by restricting the training set to particular coarse difficulties. Dancing Monkeys [12] provides a purely rule based approach to chart generation. The authors of [13] propose a transformer based architecture for the generation of general rhythm game charts, but the architecture appears to be limited to step placement for DDR/ITG charts.

In this work, we will explicitly compare our results to the original DDC architecture, as well as DDCL. We expect the transfomer based approach in ITGPT will significantly improve global cohesion across charts, and improve pattern recognition and reconstruction for step selection. We will also compare ITGPT against the rhythm game transformer from [13], referred to as GOCT, using their osu! four panel fine-tuned version. This is the highest performing model from their paper. DDG is another possible comparison, but in effect this would be a comparison against DDC trained on a subset of the training data. From the DDG paper, it is unclear which model should be considered as a baseline, so we elect to compare against the DDC architecture trained on the full dataset.

3 Data↩︎

We utilize an expanded Fraxtil dataset from the original DDC [1] and DDCL papers, consisting of 8 packs with 253 songs and 952 charts from the single eponymous author. This utilization of the Fraxtil dataset is meant to maintain continuity with prior work in DDC and DDCL, while expanding the available training data to improve model generalization. The expanded Fraxtil dataset contains roughly \(3\times\) the content of the original version from the DDC and DDCL papers. In comparison to DDC, this dataset is larger than the original Fraxtil and ITG datasets from that work combined. Before mirroring, this gives us 584644 steps across 952 charts of information to train our step placement model with.

We maintain the up-down left-right mirroring process used in the DDC paper. In effect, this process multiplies the volume of step data produced by a given collection of charts by 4, providing our symbolic model with an effective 2338576 steps of information across 3808 charts after mirroring. The technicality of a step is unchanged by mirroring all steps left to right or up to down, as patterns are maintained by this transformation [14].

We represent input data in two ways for our distinct step placement and selection models. For the step placement process, each chart is represented as a collection of beats, with each beat represented by the tuple (audio features, auxiliary features). The audio features term consists of a (32, 80, 3) sample of 80 melbands at 32 evenly placed timesteps across the beat. The auxiliary features term contains two constants relevant to the charting process, (BPM, integer (fine) difficulty). The target and model outputs per beat are represented by \(48\) length binary vectors, representing the distribution of steps across each beat.

4 Model description↩︎

Following DDC, we implement a two step process for chart generation. We first determine step placements, then step selections for each step. From DDCL, we maintain the computation of BPM as a pre-generation alignment step using the algorithm described by van de Wetering [15] and applied in the charting software ArrowVortex [4]. ArrowVortex is a commonly used and ubiquitously recommended editor for stepcharting (see [16]). It is worth note that since our algorithm uses the same method for BPM detection as a human stepcharter, the BPM assigned to songs at generation time will be identical to that which would have been assigned for the source material. Thus, there is little to no risk of BPM misalignment between the training and generation processes.

The ITGPT step placement model takes as input audio features sampled across each beat, together with BPM and difficulty information. The goal of the step placement model is to encode each audio feature while taking BPM and difficulty as context, then produce step placements across each beat. Each output is represented as a length 48 binary vector, with 1 representing a placed step and 0 representing an absent step. In the interest of computational efficiency, ITGPT employs a one-shot structure, computing all step placements for a song or a song chunk at once. On average, ITGPT takes \(0.06\)s to generate a chart (see Table 2).

The ITGPT step selection model takes as input the full context of previously placed steps, up to 500 observations, together with audio features sampled in a given radius around each placed step, and auxiliary features indicating the timing between each step. As there are 4 arrows and 4 possible placements for each arrow (no step, step, hold, release), there are \(4^4=256\) possible variations for each step. Each direction (left, right, up, down) can either be represented by a 0, 1, 2, or 3, with \(0\) representing no step, \(1\) representing a placed step, \(2\) representing the start of a hold (where the player is expected to keep their foot on the given step until a release appears) and \(3\) representing the release from a hold. Many charts have more complicated placements (e.g, “mines” which punish the player for stepping on them, and “rolls” which require the player to hit them repeatedly until a release step is observed) but in the interest of computational complexity we restrict ourselves to \(4\) possible arrow selections.

5 Methods↩︎

5.1 Generation↩︎

We extract audio features from a given file and determine BPM as described above. We feed spectrogram audio features together with BPM and desired difficulty information through our step placement model to determine step placement times. Difficulties for each generated chart are determined through distribution across a census of the training set by BPM and song length. By default, 5 charts will be generated at coarse difficulties Expert, Hard, Medium, Easy, and Beginner with evenly distributed difficulties between the top and bottom 10% for each BPM/Song length bracket. The desired difficulties may also be set by the user. We pass these step placements together with audio feature information through our step selection model to determine steps for each chart. Step selections are made after a nucleus sample with a threshold of \(.9\), discarding low likelihood possibilities once a \(90\%\) nucleus has been reached.

To improve generation quality, we introduce a scaling repetition penalty, similar to comparable techniques in natural language processing (NLP) but adapted for DDR/ITG. For a given step, a context window of the previous 20 placed steps is maintained. If a new step will cause a repetition of an \(n\)-gram of length \(l\in[4,8]\subset\mathbb{N}\), we apply a scaling penalty of \(p = 1.07^{l-3}\) to the offending logits before nucleus sampling. This is intended to push the model towards more varied patterns and away from lengthy repetitions.

5.2 Audio features↩︎

We maintain the DDC audio pipeline. This process involves ingesting each audio file as a monolithic representation taken as an average of stereo PCM audio. We then maintain three channels for each observation at the input step, representing short-time Fourier transforms (STFT) with windows of 23ms, 46ms, and 92ms, with a stride of 10ms. This methodology mirrors that used in DDC and DDCL, in turn following methodology from [17]. We reduce this representation to an 80 band mel-spectrogram representation, with log-scaling to better represent the range of human auditory perception. We organize this audio information across each beat by taking \(32\) evenly spaced samples. Duplicate samples are possible if the window for a beat does not contain 32 distinct samples. Due to our larger context window, we provide the full song or 2000 beats of audio to the model for each pass, whichever is shorter.

The final inputs to our step placement model are a tensor of shape \((T, F = 80, W = 32, C = 3)\), where T represents the number of beats in a given song, along with two integers representing BPM and difficulty. For songs with variable BPM, we separate the song into segments by BPM where the BPM is consistent for at least 50 beats. Where this is impossible, we group short segments of variable BPM into segments of at least 50 beats and assign the average of the BPM values observed. We normalize each song to have 0-mean and unit standard deviation across each frequency and channel.

5.3 Step placement model↩︎

The ITGPT step placement model architecture consists of a hierarchical transformer encoder coupled with specialized downstream prediction heads and multi-scale local refiners. The overall framework is regularized via a joint optimization protocol utilizing two auxiliary diagnostic networks.

5.3.1 Audio Feature Extraction↩︎

The input tensor representing the full spectrogram audio of a song or chunk is denoted as \(\mathbf{X} \in \mathbb{R}^{T \times F \times W \times C}\) where \(T\) is the total number of beats, \(F = 80\) represents frequencies, \(W = 32\) represents the frames sampled per beat, and \(C = 3\) represents channels. For each beat, the local slice at time \(t\), denoted \(\mathbf{X}_t\), is processed by a convolutional encoder consisting of four convolutional layers with GELU activation. The layers are:

1. Layer 1: (7,3) Conv2D layer with no padding and stride 1.

2. Layer 2: (3,3) Conv2D layer with padding along the \(W\) axis and stride 3 along the frequency axis. This reduces the frequencies by a factor of 3, similar to the maxpooling used in DDC.

3. Layer 3: (3,3) Conv2D layer with no padding and stride 1.

4. Layer 4: (3,3) Conv2D layer with padding along the \(W\) axis and stride 3 along the frequency axis. Identical to layer 2.

This stack extracts local feature maps of shape \(T \times 64 \times 24 \times 8\). These maps are flattened across the channel and spatial frequency dimensions into a shape of \(T \times 24 \times 512\), which are subsequently projected to match the core model dimension \(d_{\text{model}} = 256\), yielding the base beat sequence representation \(\mathbf{H}_0 \in \mathbb{R}^{T \times 24 \times d_{\text{model}}}\).

Simultaneously, the continuous conditioning variables for the beats-per-minute (\(\text{BPM}\)) and chart difficulty (\(\text{Diff}\)) are mapped into a normalized bounded range \([0, 1]\) based on predefined dataset configuration bounds (set arbitrarily at [0,50] and [30, 500]). BPM is first bucketed within 10 BPM bands, to suppress minor variations. These normalized scalars are projected through distinct MLP layers to construct context tokens \(\mathbf{E}_{\text{diff}}, \mathbf{E}_{\text{bpm}} \in \mathbb{R}^{1 \times d_{\text{model}}}\).

5.3.2 Hierarchical Framework↩︎

To analyze the composition of the chart across multiple time horizons, we process step selection through a hierarchical stack of encoder blocks before processing across the sequence. Let \(C_i\) denote the number of beats per chunk at layer \(l\), and \(F_i\) represent the frame resolution per beat. Then our hierarchical layers are:

1. Level 1 (\(l=1\)): Processes individual beats (\(C_1 = 1\), \(F_{\text{pb}} = 24\)).

2. Level 2 (\(l=2\)): Processes 4-beat bars (\(C_2 = 4\), \(F_{\text{pb}} = 24\)).

3. Level 3 (\(l=3\)): Processes 4-bar phrases (\(C_3 = 16\), \(F_{\text{pb}} = 24\)).

4. Level 4 (\(l=4\)): Processes 8-bar phrases (\(C_4 = 32\), \(F_{\text{pb}} = 24\)).

Following Level 4, we project each beat’s worth of 24 frames to a compressed 6 frame representation. This lower-resolution representation is subsequently processed at

1. Level 5 (\(C_5 = 64\), \(F_{\text{pb}} = 6\)).

5.3.3 Global self-attention and convolution↩︎

At Level 6, the \(6\) compressed frames are projected into a single representation vector. We inject a global learned positional embedding \(\mathbf{P}\), re-append aux token embeddings, then process the sequence through a stack of transformer layers. Before output, we pass the model through a final Conv1D pass to smooth local inconsistencies before output to density and onset prediction heads.

5.3.4 Step Placement Diagnostic Network↩︎

To preserve difficulty and BPM through generation, we pre-train a diagnostic network to estimate auxiliary parameter values from model outputs. Specifically, we train a diagnostic network to estimate BPM when provided difficulty and a chart, and difficulty when provided BPM and the chart. We first RMS normalize, project, and embed the BPM and difficulty information, as in the main model. We then process the placed steps for a given chart through a Conv1D layer with a size 7 filter, before adding the embedded auxiliary value tensor to the chart representation. We then process the resulting time-series tensor through a bidirectional LSTM before applying mean-pooling and a final MLP pass before output. The diagnostic network’s loss, computed using model inputs and outputs, is added to the total loss before optimization.

5.3.5 Training methodology↩︎

We optimize using AdamW [18] with an initial learning rate of \(1e-4\) and scheduled reduction on plateau with a factor of \(0.5\) and 5 epochs of patience, predicated on the validation set’s loss. For loss, we employ binary crossentropy with weighting \(w\) corresponding to positions in the \(48\) vector beat, based on common position prioritization. Positions \([0, 12, 24, 36]\) corresponding to the downbeat, offbeat, and 16th notes are given weight 2, positions \([8, 12, 32, 40]\) corresponding to triplets and 24th notes are given weight \(1.5\), and other positions are given weight \(0.5\). We employ this reweighting scheme as most positions within a beat will be uniformly empty, thus a minor reweighting scheme is appropriate to address class imbalance. We further implement early stopping with a patience of 10 epochs, predicated on the validation set’s PR-AUC. Loss via the diagnostic network is evaluated by the MSE of the target and predicted cumulative probability sums, to increase penalization for substantially different BPM and diff predictions. The diagnostic networks are trained on ground truth and their loss calculations are added to the main model’s per chunk loss with a lambda of \(\lambda_{diag} = 0.005\). Thus, for prediction \(y_p\), target \(y_t\), BPM \(B\), and fine integer difficulty \(D\), our full loss computation is given by

\[\mathcal{L}(y_t, y_p, B, D) = w\cdot\text{BCE}(y_t, y_p) + \lambda_{diag}\mathcal{L}_\text{Diag}\]

where

\[\begin{align} \mathcal{L}_\text{Diag} = \,&\mathcal{L}_{cum}(B, \text{Diag}_{B}(y_p, D)) \;+ \nonumber \\ &\mathcal{L}_{cum}(D, \text{Diag}_{D}(y_p, B)) \end{align}\]

5.4 Step selection↩︎

We approach the process of step selection as an autoregressive sequence generation task. As a departure from predecessor work, we now consider the full song’s audio features as input, or the last 500 steps, whichever is shorter. Each step is paired with 21 frames of audio input, representing 10 frames of past and future information for 200ms of total audio context.

5.4.1 Step selection audio encoder↩︎

Input audio is formed as a 4-dimensional tensor \(\mathbf{X}_{\text{audio}} \in \mathbb{R}^{T \times S \times F \times C}\), where \(T\) represents sequence length, \(S\) denotes the frame dimension, \(F\) denotes frequency bins, and \(C\) represents channels. This representation is processed by a convolutional frontend consisting of four layers with GELU activation. The four convolutional layers are identical to those used in the step placement convolutional encoder, with the exception of the final layer which produces twice the channels, for better audio resolution. Following the convolutional encoder, we collapse the audio representation along the frequency and time within beat axes, producing a representation of shape \((T, 104, d_{model}/2)\).

We apply residual vector quantization to better generalize across our dataset. Residual vector quantization in audio encoding is a highly standardized practice intended to contextualize and discretize audio representations for more comprehensive encoding (see [19],[20],[21]). In particular, we implement the RVQ from [21], found here. We employ 4 codebooks with \(\dim=d_{model}/2\), using cosine similarity to evaluate codebook and commitment losses, and a dead code threshold of \(2\). Codebook and commitment losses are passed through the model and evaluated as a sum with the total loss.

After RVQ, we add two positional embeddings for the frequency and frame axes, reshaped to match the quantized representation, then pass through a transformer encoder layer. This layer is intended to better extract and contextualize local audio relationships highlighted by the convolutional layers. We apply attention pooling across the pooled frame/frequency dimensions to produce a \(128\)-dim feature vector for each placed step. We project these vectors to a single \(d_{model}\) feature vector per-step for the main model.

5.4.2 Step selection model↩︎

Along with the encoded and compressed audio signal of shape \((T, d_{model})\), the ITGPT step selection model takes as input a tensor of shape \((T, 2)\) representing the \(\Delta\)-beat time between the last step and the next step, respectively. The model iterates autoregressively through placed steps, taking as input the last 500 steps, or the full song if fewer than 500 steps have been placed.

For a step with \(S\) previously placed steps, the step history tensor has shape \((S, 256)\), representing the 256 valid step classes. The aux tensor is encoded through two embedding layers, to a tensor of shape \((T, d_{model})\) which will be referred to as the audio aux tensor, and a tensor of shape \((S, d_{model})\) which will be referred to as the step aux tensor. The step aux tensor is sliced to match the number of steps in memory. We employ pervasive FiLM conditioning before every self attention block with each respective aux tensor. The audio tensor is passed through two self-attention blocks with FiLM conditioning applied before each attention pass. We then perform gated fusion between the audio and aux tensors to form a single audio context tensor of shape \((T, d_{model})\).

The step tensor of shape \((S, 256)\) is passed through a linear layer to project to \((S, d_{model})\), and then passed through six self-attention blocks with FiLM conditioning applied before each attention pass. Two cross attention blocks are then applied with the step tensor as the query and the audio context tensor as the key and value. We then apply six more self-attention blocks before output through a linear layer to a head of shape \((S+1, 256)\) representing the logits for each step class. We then apply softmax to produce a probability distribution over the 256 possible step classes.

5.4.3 Training methodology↩︎

Our step selection model is trained using AdamW [18] with an initial learning rate of \(1e-4\) and scheduled reduction on plateau with a factor of \(0.5\) and 5 epochs of patience, predicated on the validation set’s loss.

During training, we have the network instead output 4 split heads, representing predictions for the current step as well as the following \(3\) steps. This process is intended to improve the model’s ability to learn patterns across multiple steps, inspired by a similar technique from ProphetNet [5]. At generation time, the additional predictions are not used. During training, we apply scaling dropout on the step history, scaling up to \(.2\). That is, eventually \(20\%\) of the step history is blanked during training to aid in generalization. The warmup period consists of \(100000\) charts passed through the model, during which the dropout is scaled linearly.

Loss is calculated as the weighted sum of categorical cross entropy over the \(4\) predicted steps. Step \(0\) received weight 1. For each step \(i\in\{1,2,3\}\), we compute a step exponent:

\[\lambda_i = 1 + (\frac{i}{4}).\]

For a training run of \(N\) total steps and a current step of \(s\), we compute the weight for step \(i\) as \[w_i(s,N) = (s/N)^{\lambda_i}.\]

This weighting scheme causes the model to focus on earlier steps earlier on in the training procedure, with later predictions receiving increased weight further into training. Close to the end of training, \(s/N\) becomes close to \(1\), and so the model begins to weight all predictions nearly evenly. The final loss is then computed as the sum of the cross entropy for each step, weighted by the step weights, along with the VQ loss from the residual vector quantization.

6 Experiments↩︎

We apply the same 8/1/1 split used in the original DDC and DDCL papers for the expanded Fraxtil dataset. We further maintain the chart difficulty segregation implemented in DDC, as charts for the same song at different difficulty levels often present with similar or identical sections. That is, all difficulties for any given chart will appear in the same split.

All testing was performed on a system using a single Nvidia RTX 3080 with 10GB of VRAM.

6.1 Step placement evaluation↩︎

We compare the ITGPT step placement model against two alternate versions, intended to highlight the value of choices in the training and design methodologies. The ITGPT (NH) or no hierarchy version of the model omits the hierarchical processing, in favor of a more aggressive downsampler before passing to global attention. This is a much more traditional version of an encoder only step placement model structure. The ITGPT (ND) or no diagnostic version of the model omits the diagnostic network, and is trained solely on the binary crossentropy loss for step placements. We also compare our models to DDC and DDCL as predecessor works, as well as the pre-trained model from [13], denoted GOCT, which we test using the checkpoints the authors provide. Following conventions from that paper, GOCT is compared to the threshold maximized versions of each model as GOCT is a token based classifier and cannot be optimized further in this way. DDC is benchmarked using the same \(20\)ms accuracy window methodology used in the original paper, and in the \(20\)ms version of DDC from the DDCL paper.

6.1.1 Metrics↩︎

We report mean performance across all predictions for F1score, precision, and recall with a flat threshold set at .5 in Table 1. Following methodology from DDC and DDCL, we also report metrics with threshold optimization for F1score. To better comprehend performance for full charts, we also report these metrics averaged across charts instead of by prediction. Metrics averaged this way are denoted with the ‘cht’ subscript. Precision-recall AUC (PR-AUC) and loss (BCE) are reported with chart averaging. For a more granular breakdown of model performance at different difficulty levels, we report model performance averaged across each coarse (verbal) and fine (integer) difficulty level in tables 2 and 1, respectively. GOCT metrics are not reported for coarse difficulties as their architecture discards this identifier at evaluation time.

We observe the greatest benefit from threshold tuning from the DDC model, with a smaller but still significant benefit for DDCL. ITGPT shows the smallest improvement, which appears to be a result of the diagnostic model. Threshold tuning still provides some improvement, but not to such an extent that tuning is required for acceptable ITGPT performance. The ITGPT (ND) model, trained without the diagnostic model, exhibits a similar level of performance improvement from threshold tuning to that of DDCL, implying that one benefit of the diagnostic model is the elimination of the need to tune for thresholds in generation. This is an especially plausible conclusion, since the original generation method used by DDC relied on threshold selection to generate different difficulties.

One of the main benefits exhibited by the DDCL step placement model was a dramatic improvement among lower difficulty charts. ITGPT extends this trend, with the greatest improvement observed among charts at the Beginner difficulty level, and significant but more modest improvements across the board. The shift to a transformer with access to global context appears to be a dramatic improvement over both the DDC and DDCL models, as ITGPT is superior to DDCL in almost every metric, for every difficulty.

GOCT is fairly close to ITGPT if no threshold tuning is performed. However, as GOCT is a token based classifier model, the most likely token is selected before decoding, and thus there is no threshold to optimize to improve performance.

6.1.2 Generation times↩︎

Here we make brief note that while most models evaluated have negligible generation times for step placement due to one-shot architectures, the model used in [13] is autoregressive and thus has significantly increased computational cost compared to other models shown here. While the increased cost is not so severe that GOCT cannot be used, ITGPT is both more accurate and nearly \(50\times\) faster.

6.2 Step selection evaluation↩︎

We compare our ITGPT step selection model against a version of the same model with a smaller encoder, which we denote ITGPTs. Compared to the version we describe above, the smaller ITGPT version eschews the audio input level transformer in favor of a large convolutional compression block with a \((13,3)\) kernel, to reduce each step’s audio context to a single feature vector. While this model is substantially faster to train, it is not meaningfully faster at generation time (see Table 3). We also compare our model against the DDC and DDCL step selection models. GOCT is timing only for DDR/ITG, and as such does not have the capability to select steps for ITG charts.

6.2.1 Metrics↩︎

We report average accuracy and loss (CCE) across all placed steps for each model. We further report top-\(k\) accuracy for each model, where \(k\in\{1,2,3,5\}\). In some cases, top \(2\) accuracy is likely a better judge of true model accuracy than raw accuracy, since in stepcharting for DDR/ITG, some sequential step choices (left then down vs left then up for example) are genuinely a choice. We further report accuracy and loss averaged across charts instead of across placed steps. Loss as reported for ITGPT contains loss for the \(4\) step prediction method we describe above, and is thus not comparable to DDC/DDCL. We also report accuracy for held notes and non-held notes separately, as held notes are a less common step type and thus more difficult to predict.

Similar to results observed from the DDCL paper, ITGPT exhibits significant improvement over DDCL and DDC, but not to the same degree exhibited by the step placement model. This is likely attributable to the fact that step selection is a more difficult and creative task, where multiple interpretations (and indeed, full mirrorings) of the same chart can be correct at the same time. Step placement, on the other hand, is much closer to onset detection and is thus a task with mostly right and wrong answers.

Strangely enough, DDCL exhibits better performance on held notes than ITGPT. While DDC remains catastrophically bad at held note prediction, DDCL is able to achieve a held note accuracy of \(0.4115\) compared to ITGPT’s \(0.3782\). It is likely this is attributable to the ConvLSTM encoder, since a held note often corresponds with a held tone, which would be especially easy to extract across beat-separated frames through a ConvLSTM encoder.

6.3 Generation times↩︎

We measure average step selection generation times across the test split of our 8/1/1 expanded Fraxtil dataset. Step selection, as the autoregressive portion of the generation pipeline, is by far the most consequential factor in user experienced wait times for chart generation. To whit, in table 2, we observe an average generation time per chart of \(0.06\) seconds. Notably, ITGPT improves on generation times compared to DDC, and is around \(7\times\) faster than DDCL. Improvement over DDC is likely attributable to the architectural ability for ITGPT to process convolutional input once per song, then iterate purely over tokens for the remaining generation steps. This allows ITGPT to leverage a faster transformer based architecture in purely symbolic generation. In turn, DDC is much faster than DDCL since no audio input is processed whatsoever.

7 Discussion↩︎

There are a number of areas for improvement to our approach. Primarily, a much larger dataset may exhibit some performance improvement. While GOCT from [13] is trained on osu! datasets, their substantially improved results using a transformer architecture with a much larger dataset (\(10\times\) Fraxtil-sized osu! vs our \(3x\) Fraxtil) imply that a larger DDR/ITG dataset would produce an even better model, whether ITGPT or GOCT based. However, the stronger performance of ITGPT in comparison to GOCT, even using a one-shot architecture, implies that any larger dataset should likely be DDR/ITG native and not sourced from other rhythm games. Further, if a stronger GOCT model is constructed, the computational cost advantage of ITGPT would remain. It is also possible that ITGPT could be improved by adopting the GOCT approach to sampling at a BPM determined window at the time of spectrogram generation, instead of resampling along 10ms windows. However, the performance difference between GOCT and ITGPT is fairly uniform across difficulties (which tend to correlate with higher BPM), so we expect any benefit would be limited.

A unified step placement/selection pipeline remains a possibility, but a distant one. For step placement, each beat contains \(48\) binary positions, which is a representation that could possibly be reduced to around \(8\) positions if we limit the maximal complexity to \(24\)th notes. Already this is a \(2^8=256\) class classification problem if performed per-beat, or an \(8\) label multi-label binary classification problem if performed per-step. If the additional \(256\) per-step classes are added, the problem balloons to at minimum a completely unreasonable \(256^8\) class classification problem, or at best an \(8\) label, \(256\) class multilabel/multiclass classification problem. Perhaps each step within a beat could be generated autoregressively, reducing each beat to an \(8\) step, \(256\) class multi-label classification problem, but we suspect the effect of this would be solely to exacerbate the drift issues associated with autoregressive step placement.

8 Conclusions↩︎

We introduce ITGPT, a transformer based model for the automatic generation of DDR/ITG charts. ITGPT represents a significant improvement over prior works, especially for lower difficulty charts. We further introduce a new dataset of DDR/ITG charts, expanding the existing Fraxtil dataset commonly used to evaluate DDR chart performance.

We introduce a new hierarchical transformer architecture for step placement, which allows the model to consider the full chart structure when making predictions. We consider this contribution individually significant, as it is bespoke for DDR/ITG charts as a task and particularly well-suited to tasks with well-defined boundaries and temporal structure, such as beat-delineated music.

We further introduce a diagnostic model for improved consistency during training for auxiliary parameters such as BPM and difficulty. While not itself an original idea in general, we are unaware of any similar approaches taken in the pursuit of rhythm game generation, and thus consider this contribution individually novel.

We introduce a new transformer based model for step selection, dramatically improving generation times through the use of mixed processing. Audio is processed once per chart, then used throughout generation, yielding even faster times than DDC, which does not use audio at all. ITGPT is about \(7\times\) faster than DDCL (see Table 3). We further introduce a robust generation pipeline, integrating domain knowledge from rhythm games with techniques from NLP to perform highly accurate charting.

9 Code Availability↩︎

All code for this paper can be found at this project’s github.

10 Funding↩︎

We gratefully acknowledge partial funding from the Alexander von Humboldt Foundation.

Appendix↩︎

References↩︎