1 Introduction↩︎

Human visual attention is a mechanical process that people follow to select particular visual stimuli with their eyes [1]. In this paper, the humans are software engineers and the visual stimuli are words in source code of programs the software engineers read or build. Visual attention over source code is distinguished from generalized computer vision in that movement and object cues are absent, and it is distinguished from reading text because the eye tends to scan many areas of code with frequent regressions spread over many files with many hundreds of lines of code each [2], [3]. The underlying cognitive goal is different: software engineers reading code are building a mental model of software program behavior, versus spacial awareness or story comprehension [4]–[7].

Predicting human visual attention in source code involves a forecast of which words in a program’s source code will be read most often, reread most often, and/or to what degree and in what order. Predictions of human attention are useful for determining what information software engineering tools should display [8], [9], designing user interfaces [10], and developing bio-inspired machine learning models [11], [12]. Predictions of human attention also holds academic value as a means of studying human cognitive processes [13]–[16].

Typical strategies for predicting visual attention have their origin as either general computer vision research (where the task is variously known as “gaze tracking”, “gaze prediction”, etc. [17]–[19]) or from models of reading text [20], [21]. But models from general vision research treat the prediction problem as coordinates in a plane or 3-d space, tending to ignore the semantic information in words or code items. And reading models assume particular patterns of eye scan which are not found over source code [2]. Lately an alternative has been proposed based on a hypothesis that artificial neural networks natively exhibit similar patterns of attention as humans, but as Eberle et al. [22] point out, this hypothesis is not borne out for highly task-specific gaze such as reading source code.

In this paper, we propose a computational model of human visual attention over words in source code. The input to our model is a snippet of source code, fixation tokens and tokens next to fixation tokens. The output is a measure of the total gaze time a person is likely to look at each word. The model predicts the metric percent total gaze time (ptgt), which is used as an aggregate metric of attention per word in code in related literature [23], [24]. Our model is novel in two key ways: First, we use eye-tracking datasets to finetune the pretrained language models proposed by [25]. Second, we designed a novel loss function that computes the similarity between human attention (ptgt) and machine attention (self-attention) for finetuning.

We evaluate our model in an experiment comparing the model’s predictions to actual eye tracking data. We source the eye tracking data from studies of two programming languages and two programming tasks to maximize generalizability of our experiment. Two datasets are studies of Java programmers tasked with reading code to write documentation (the Rodeghero study and Wallace study) [3], [24] and another is a study of C programmers tasked with locating bugs (the Smith study) [26]. We compare our model to the most related and current baseline from software engineering literature [23] and computer vision literature [27]. We also create an ablation version of our model to study the effect of a novel component of our finetuning procedure. Compared with baseline [23], our method achieves 64%, 16%, and 467% improvements in Pearson correlation on Wallace, Rodeghero, and Smith study.

We conduct an additional experiment on scanpath prediction to demonstrate the effectiveness of our model on the task that requires the understanding of human thought process. We used the dataset designed for the scanpath prediction task proposed by Bansal et al. [11]. We compared our model with the LLMs-based approach proposed by Bansal et al. [11] and commercial models. We show that our model outperforms Bansal baseline and both GPT-5 and Claude on this task.

5pt

2 Background & Related Work↩︎

In this section, we discuss the research related to predicting human attention during software engineering tasks and scanpath prediction. We summarize related papers on predicting human attention during software engineering tasks in Table 1.

2.1 Biometric-based Approaches↩︎

Biometric-based approaches predict programmer attention based on measurements of body activity [29]. These approaches tend to target coarse-grained signals of attention. The goal is to predict whether a person is, e.g., encountering a stressful workload [44]. Biometric-based sensing of engagement and attention level has long been studied in other domains [45], and has been applied to programmers [30]. Likewise, Fritz et al. [29] trained a Naive Bayes classifier to predict the difficulty of programming tasks and programmer resilience to intellectual challenges.

2.2 Eyetracking-based Approaches↩︎

Eye-tracking is a technology that measures the location of a person’s gaze over time, along with other eye data such as pupil dilation [46]. Eye-tracking has been used for decades to study human attention during many types of tasks including software engineering – the community is mature enough to have guides describing best practices for eye tracking in software engineering [47], [48] and surveys chronicling its history [16], [49], [50]. However, predicting human programmer visual attention is much less common. An early example uses eye tracking data as a guide for crafting weight modifications by hand to an information retrieval system that generates source code documentation [24]. A different group of researchers aimed to predict programmer navigation through code by identifying influential visual features during software engineering tasks such as bug localization and traceability link recovery [32], [35]. Nuances of behavioral signals have been studied as well, such during code changes, cross-language environments, and scanpaths [11], [13], [28], [31]. Yet by far the most similar work to this paper predicts the degree of human visual attention using a graph neural network (GNN). The graph input to the GNN is an abstract syntax tree (AST) of a method written in the Java programming language, and the output is a measure of the amount of gaze time spent on each word in the method [23]. We use this GNN approach as a baseline in our evaluation later in this paper.

2.3 Computer Vision-based Approaches↩︎

General computer vision-based approaches formulate these problems as gaze prediction [27] and the prediction of neural response in visual systems [51], [52]. The input of these approaches is in the 2-d or 3-d space. The goal is to predict the target area or compare the models with human brains. However, these approaches tend to ignore the semantic information in vocabularies, e.g., source code [53]. In this paper, we propose an alternate computational model for source code.

2.4 ML and LLM-based Approaches↩︎

Recently, different approaches to predict programmer attention based on classical machine learning (ML) techniques have given way to approaches using large language models (LLMs). Examples in this vein include efforts to predict the next fixation token during code navigation using recurrent neural models (RNNs) [38] and an approach that uses a GPT2-like language models to predict the scanpath of programmers during code documentation [11]. Another strategy is to modify the structure of the attention mechanism of artificial neural networks based on experimental observations of programmers [12], [43]. These approaches tend to be highly novel but have niche applications such as attention to function call graphs during code summarization [54] or prediction of attention levels for the purpose of studying the similarity of machine and human attention [55].

2.5 Summary↩︎

Table 1 shows a trend from biometric, through hand-crafted eye-tracking-based, to data-driven ML and LLM-based approaches most recently. While eye-tracking data itself has been used consistently, these have been used as training inputs to ML and LLM techniques rather than as a basis for hand-crafted features. The gap in the literature this paper fills is predicting attention at a fine level of granularity (specific words in source code) using an LLM-based computational model trained with eye-tracking data.

2.6 Scanpath prediction↩︎

Scanpath is a is a task of predicting sequence of fixations made by a person while observing a visual stimulus [11], [56]–[60]. Before neural models, Reichle et al. [57] proposed E-Z Reader that models a eye-movement based on the reading pattern for different types of word and passages. Alternatively, Engbert et al. [58] implemented a mathematical models to model the eye movement in the general reading comprehension. More recently, neural-based approaches for scanpath prediction have been proposed. For example, Deng et al. [56] proposed a LSTM-based approach that consider different reading pattern, e.g., non-left-to-right reading pattern. However, these approaches focus on reading comprehension instead of programming tasks. Bansal et al. [11] first proposed a GPT2-like language model to predict the scanpath of human programmers while doing source code summarization. Therefore, we use this as our baseline models.

3 Computational Model↩︎

Our computational model is a Transformer neural architecture that we train with a custom loss function. We finetune a pretrained Transformer using actual eye-tracking data over source code. Our loss function for finetuning takes the machine attention vectors in the last layer of the Transformer model, aggregates those vectors for fixation, then computes a similarity between the machine’s attention and a reference human visual attention calculated for the fixation tokens in the input. This section discusses this custom loss function, the ingredients required to work, and the model’s prediction target.

3.1 Prediction Target↩︎

Our prediction target is percent total gaze time (ptgt). The ptgt metric was proposed by Rodeghero et al. [24] to quantify the “importance” of a word in source code to a person reading that code. Formally, ptgt is: \[\mathrm{ptgt}(r) \;=\; \frac{G_r}{G_{\text{total}}}.\]

Where \(G_r\) denotes the total gaze time (in milliseconds) spent on a region of interest (ROI) \(r\) (where \(r\) is usually a word in source code), and \(G_{\text{total}}\) denotes the total gaze time spent viewing the entire code artifact (e.g., all the words in a code subroutine). Percent total gaze time quantifies the proportion of a programmer’s visual attention allocated to a specific code region relative to all viewed code. By normalizing gaze time by the total viewing duration, ptgt enables comparisons across different regions, methods, and participants independent of absolute reading time or size of the ROI.

3.2 Ingredients↩︎

The two key ingredients to our computational model are:

1. Pretrained Transformer-based Language Model. Almost all current large language models are based on the Transformer architecture in one form or another, with the main difference among models being the size of the model and the data on which it is trained. While it is possible to train our computational model from scratch, the large amount of data required to train language models and the relatively small amount of available eye-tracking data leads us to finetune a pretrained model. The Methods section describes the pretrained language model in our experiments.

2. Eye-tracking Data over Source Code. We require samples of actual human visual attention to calculate ptgt and finetune the pretrained model. We obtain these samples from experiments of human programmers doing software engineering tasks at a computer equipped with an eye-tracking device. The samples include source code and human gaze data. We used leave-one-out validation, i.e., we holdout one participant at a time and averaged results. We describes our datasets in Section 4.1.

The input sequence is composed of words from a section of source code, fixation tokens, and tokens next to the fixation tokens code. We tokenized input sequence with the GPT2 tokenizer as in [25]:

\[\mathbf{S} = (x_1, \dots, x_{T_x},\; y_1, \dots, y_{T_y},\; z_1, \dots, z_{T_z})\] Where \(x_{t_x}\) is the source code tokens, \(y_{t_y}\) is the fixation tokens, and \(z_{t_z}\) is the tokens next to the fixation tokens. The model \(\mathcal{M}^{\text{pre}}\) is pretrained using the typical categorical cross-entropy (CCE) loss denoted \(\mathcal{L}_{\text{CCE}}\).

3.3 Custom Loss Function↩︎

To create our computational model, we finetune \(\mathcal{M}^{\text{pre}}\) to create \(\mathcal{M}^{\text{ft}}\) using a our custom loss function, \(\mathcal{L}\). During finetuning, we calculate loss in three steps:

3.3.1 Attention Readout↩︎

Our attention readout mechanism extracts the model’s attention to each fixation token in the input sequence. In a Transformer model, \(Q\) and \(K\) are “attention” matrices used to scale the value matrix \(V\), which represents the input tokens. A typical LLM will have many Transformer layers – what we do is extract \(Q\) and \(K\) from the final Transformer layer (note, Wang et al. [61] found that the attention heads in the final layer are most like human attention). Formally, in the final transformer layer with \(H\) heads and input sequence length \(T\), the model computes query, key, and value matrices for each attention head as:

\[Q^{h},K^{h} \in \mathbb{R}^{ T \times d_v},\; V^{h} \in \mathbb{R}^{ T \times d_v}, \quad \text{where } d_v=d_{\mathrm{model}}/H .\] Each scaled dot-product attention is computed as: \[A^{h} = \operatorname{softmax}(\frac{Q^{h} {K^{h}}^\top}{\sqrt{d_k}}), \quad A^{h} \in \mathbb{R}^{T \times T}.\] We obtain our attention readout by averaging the attention scores from attention heads 10 to 16. Wang et al. [61] observed that attention heads in deeper levels are closer to human attention. Since our task is to predict human attention, we used attention heads from 10 to 16. We formally define our attention readout as: \[\mathbf{\bar{A}} = \frac{1}{H-10} \sum_{h=10}^{H} A^{(h)},\] Each row of \(\bar{A}\) represents a probability distribution over key tokens attended by a given query token.

3.3.2 Model-Derived Attention Score↩︎

Next, we define an attention score by averaging attention probabilities from a subset of query tokens and key tokens in \(\bar{A}\). Formally, we define the attention score as: \[\hat{\mathbf{a}} = \frac{1}{|\mathcal{Q}'|\,|\mathcal{K}'|} \sum_{t_z \in \mathcal{Q}'} \sum_{t_y \in \mathcal{K}'} \bar{A}_{t_z,t_y}, \quad \mathcal{Q}' \subseteq \mathcal{Q}, \quad \mathcal{K}' \subseteq \mathcal{K}.\] Where \(\mathcal{Q}'\) is a set of tokens next to the fixation tokens and \(\mathcal{K}'\) is a set of the fixation tokens in the input. This scalar represents the probability mass that the fixation tokens are attended to by its next token.

3.3.3 Human Attention Supervision↩︎

Finally, we compute the similarity of the predicted attention \(\mathbf{\hat{a}}\) to the reference human attention \(\mathbf{a}\). Reference human attention \(\mathbf{a}\) is ptgt for each token (see the Metrics section under Methods for a complete definition of \(\mathbf{a}\)). Then, we combine that similarity with the standard CCE loss. To compute the similarity of \(\mathbf{\hat{a}}\) to \(\mathbf{a}\), we use mean squared error (MSE) and Pearson correlation. Our rationale is that MSE will enforce absolute accuracy of the predicted fixation time, while Pearson correlation will enforce the relative attention structure (high vs. low attention tokens). The combination encourages the model to predict correct magnitude and to preserve human-like attention distributions across tokens. Formally, we use MSE as: \[\mathcal{L}_{\text{MSE}} = \frac{1}{m}\sum_{i=1}^{m}(\hat{a}_i - a_i)^2,\] Where \(m\) is the batch size.

We denote \(\rho\) as the Pearson correlation. We normalize and convert \(\rho\) to a loss as: \[\mathcal{L}_{\text{corr}} = 1 - \tfrac{1}{2}(\rho + 1).\]

To prevent the auxiliary attention losses from being dominated by the language modeling loss, we dynamically rescale them to match the magnitude of the cross-entropy loss. The final training objective is: \[\mathcal{L} = \mathcal{L}_{\text{CCE}} + \lambda_{\text{MSE}}\,\alpha_{\text{MSE}}\,\mathcal{L}_{\text{MSE}} + \lambda_{\text{corr}}\,\alpha_{\text{corr}}\,\mathcal{L}_{\text{corr}},\] where \(\alpha_{\text{MSE}}\) and \(\alpha_{\text{corr}}\) are scaling factors based on the ratio between the cross-entropy loss and the corresponding attention loss, and \(\lambda_{\cdot}\) are fixed dominance coefficients. Table 2 gives the values for all the training hyperparameters in our evaluation. We used these hyperparameters based on our experiments.

3.4 Finetuning↩︎

We follow conventional finetuning procedures for Transformer-based language models of source code established in many years of related work [62]–[64]. Note above that our loss function uses \(\mathcal{L}_{\text{CCE}}\) as is conventional – the model’s goal is still to learn to generate samples in an autoregressive fashion like almost all current language models. The model will see source code samples from the eye-tracking experiments as though they are normal samples during normal training. However, the effect of our loss function is to increase or decrease the penalty the model receives based on how similar its own internal attention is to human visual attention observed experimentally. This process yields a model \(\mathcal{M}^{\text{ft}}\) which is operationally identical to any other autoregressive language model but which has attention scaled to be more like a human programmer. Note, however, that we can extract \(\mathbf{\hat{a}}\) as above to obtain attention predictions from the model after finetuning.

1.5pt

4 Aggregated Attention Prediction↩︎

This section describes our research methods for studying machine and human attention using the computational model in Section 3. Our research objective is 1) demonstrating the degree to which our model mimics the attention people pay to those words by comparing attention on words in source code with humans and our computational model 2) comparing our model to the most-similar computational baselines. To this end, we ask the following Research Questions (RQs):

  RQ1

What is the performance of our computational models compared with the baseline and human reference?

  RQ2

What is the performance of our computational models with the Pearson correlation loss compared with the one without Pearson correlation loss?

The rationale behind RQ1 is that we use the attention scores found to be closer to human attention to finetune the models. However, it is possible that the models do not learn the meaningful information during finetuning, so the baseline could still outperform our model or the results could still be far from human reference. We compare our model with the baseline from software engineering and computer vision.

The rational behind RQ2 is that it is possible that the Pearson correlation loss does not help the model to learn the meaningful information. To this end, we conduct an ablation study to compare the model with the Pearson correlation loss and the one without Pearson correlation loss.

4.1 Datasets↩︎

We use three datasets in our experiment. The first dataset (the Smith study) involves 21 programmers who examined up to eight memory bugs in three different C programs on a machine equipped with a Tobii Pro Fusion eye-tracking device [26]. The second dataset (the Wallace study) involves 10 programmers who read 40 subroutines in the Java programming language in five different sessions [3]. The task is to read Java source code and write an English sentence that describes the purpose of that code on a machine equipped with a Tobii Pro Fusion eye-tracker. The third dataset (the Rodeghero study) involves 10 professional programmers who were hired to read Java source code and write an English sentence that describes the internal of that function on a machine equipped with a Tobii TX300 eye-tracking device [24]. All studies are independent. We chose these three different studies to maximize the generalizability of this study.

4.2 Computational Model Settings↩︎

We use the pretrained language model jam as \(\mathcal{M}^{\text{pre}}\) in this study. The jam model is a GPT-2 style language model that has around 350m parameters [63]. We select this model for two reasons. First, it includes Java and C versions each with a tightly-regulated training set intended for controlled experiments on software engineering datasets. Second, subsequent experiments demonstrated that the 350m parameter model is a “goldilocks” size in that it is large enough to be competitive with large commercial models on select software engineering tasks, and yet small enough to be trainable on commercial hardware [25], which is important to ensure reproducibility of results and control of experimental variables. Note that we finetune Rodeghero dataset with the model that has been finetuned with Wallace dataset because its size is smaller. We freeze parts of the weights to finetune our models with the proposed loss function \(\mathcal{L}\). The purpose is to increase the batch size because Pearson correlation is less effective when the batch size is small [65]. We finetune layer 14 to 24 because finetuning with only the final layer may not be large enough for models to learn meaningful information and Wang et al. [61] found that these layers better align with human attention compared with other layers.

4.3 Comparisons with Human Reference↩︎

We compare the human participants in one study to other participants in the same study, e.g. one participant in the Smith study to others in the Smith study. The purpose of this comparison is to establish a baseline for expected similarity of the computational model to human participants. We would not expect the computational model’s attention similarity to substantially exceed the attention similarity that humans have to each other – an excellent benchmark would have similarity near to that which humans have to one another.

4.4 Comparison with Baselines↩︎

We use two previous machine models of attention for comparison, plus one variant of our model as an ablation test. One is the software engineering model proposed by Bansal et al. [23]. A second comparison is the most-similar model from the computer vision research community [27]. Almost all general-purpose computer vision models are not suitable for the task we present in this paper because they use 2-d images as input and output coordinates in those images – they rely on color, shape, and other real-world visual cues not in source code. However, the model by Tafasca et al. [27] includes a context encoder for semantic information akin to that in words in source code. We create a faithful re-implementation of both approaches using source code words as input and ptgt as output. These models are useful as comparison points for our model while also showing the trainability of different artificial neural models to mimic human attention.

We also create a variant of our model which is identical except that the loss function does not include a correlation component as an ablation study. We denote this variant ours-nocorr when reporting results. Formally speaking, the variant’s loss function is: \[\mathcal{L} = \mathcal{L}_{\text{CCE}} + \lambda_{\text{MSE}}\,\alpha_{\text{MSE}}\,\mathcal{L}_{\text{MSE}}\]

4.5 Metrics↩︎

We use four different measures of correlation as similarity metrics. We calculate each metric the same way, as an aggregate comparison of ptgt predicted by the computational model for each word, to the actual ptgt for each word observed experimentally. However, a mismatch between machine and human attention is that the model outputs one ptgt score for each word, but each study has multiple people as participants and these participants do not look at all the same words, and often the same person reads the same word multiple times. We solve this mismatch by aggregating the ptgt for each word for each participant in each session. The correlation is calculated over pairs, which have the average of all ptgt values for a word from a single experimental session involving one participant, and the predicted ptgt value for that word.

We have canonical word tokens and virtual word tokens for our comparison as follows:

4.5.1 Canonical Word Tokens↩︎

We define a canonical word token as the pair \((f,t_{src})\). The set of all canonical word tokens is: \[\mathcal{W} = \{(f,t_{src}) : f \in \mathcal{F},\; t_{src} = 1,\dots,L_f\}.\] Where \(\mathcal{F} = \{1,\dots,|\mathcal{F}|\}\) denotes the set of functions (subroutines), and function \(f \in \mathcal{F}\) contains \(L_f\) word tokens indexed by \(t \in \{1,\dots,L_f\}\). Also, we define \(W'\) as the subset of \(W\), which represents a set of canonical fixation tokens in a function \(f\).

We extract a single attention score for each canonical fixation word token from the model. These predictions are invariant across participants and sessions: \[\hat{a}_{f,t_{src}} \in \mathbb{R} \qquad \forall (f,t_{src}) \in \mathcal{W}', \quad \mathcal{W}' \subseteq \mathcal{W}.\]

4.5.2 Human Study Sessions↩︎

From a human study, we have \(\mathcal{S} = \{1,\dots,|\mathcal{S}|\}\) experimental sessions. Each session \(s \in \mathcal{S}\) could have multiple participants. Thus, the set of canonical fixation word tokens observed in a session \(s\) may be denoted: \[\mathcal{W}'_s = \{ (f, t_{src}) \in \mathcal{W}' \}.\]

For each observed canonical fixation word token in a session \((f,t_{src}) \in \mathcal{W}'_s\), we compute ptgt, denoted: \[a_{s,f,t_{src}} \in \mathbb{R}\]

4.5.3 Virtual Word Tokens↩︎

We define a virtual word token as the triple \((s,f,t_{src})\), representing an observation of canonical token \((f,t_{src})\) in a session \(s\). The set of all virtual tokens is: \[\mathcal{V} = \{(s,f,t_{src}) : s \in \mathcal{S},\; (f,t_{src}) \in \mathcal{W}_s\}.\]

If a canonical word token \((f,t_{src})\) appears in multiple sessions, then it gives rise to multiple virtual tokens, each with its own human attention measurement but sharing the same model prediction. We define a subset of virtual word tokens that represents the set of virtual fixation word tokens as \({W}'_s\).

4.5.4 Vector Construction for Correlation↩︎

We build two vectors, \(\mathbf{a}\) and \(\hat{\mathbf{a}}\), as an input to a correlation metric. We let \(N = |\mathcal{V}|\) and construct the two vectors of length \(N\): \[\begin{align} \mathbf{a} &= \big( a_{s_i,f_i,t_{\text{src}_i}} \big)_{i=1}^{N}, \\ \hat{\mathbf{a}} &= \big( \hat{a}_{f_i,t_{\text{src}_i}} \big)_{i=1}^{N}. \end{align}\]

\(\hat{\mathbf{a}}\) contains repeated values whenever the same canonical word token appears in multiple sessions.

4.5.5 Correlation Computation↩︎

Finally, we compute the correlation between model predictions and human attention over all virtual tokens: \[r = \mathrm{Corr}(\hat{\mathbf{a}}, \mathbf{a}),\] Where \(\mathrm{Corr}\) is one of four correlation metrics.

4.5.6 Calculation for Human Comparisons↩︎

For human comparisons, we follow the same procedure as above, except that the model prediction is replaced by the attention of a held-out participant. Let \(p^\ast\) denote the held-out participant. For each virtual fixation word token \((s,f,t_{src})\) associated with \(p^\ast\), we define \(a\) and \(\hat{a}\) as: \[\hat{a}_{f,t_{src}} \;=\; a_{p^\ast,f,t_{src}}, \qquad a_{s,f,t_{src}} \;=\; a_{s,f,t_{src}} - a_{p^\ast,f,t_{src}},\] Where \(a_{p^\ast,f,t_{src}}\) is the attention score of participant \(p^\ast\) for word token \((f,t_{src})\), and \(a_{s,f,t_{src}}\) ranges over all other participants’ observations of the same token. We then compute the correlation between \(\hat{a}\) and \(a\) as before.

4.5.7 Correlation Functions↩︎

We use the following correlation metrics for the function \(\mathrm{Corr}\). We provide our rationale for each as follows:

• Pearson correlation measures linear association between model and human attention values. It captures whether the model assigns proportionally higher attention to words that humans attend to more, assuming a linear relationship to absolute magnitudes.

• Spearman rank correlation evaluates monotonic agreement between model and human attention rankings. It captures whether the model ranks words correctly from most to least attention.

• Kendall’s \(\tau\) measures ordinal agreement based on pairwise concordance between rankings. It provides a more conservative estimate of rank agreement than Spearman and is particularly robust to ties and small-sample effects common in token-level attention data.

• Cosine similarity measures directional alignment between attention vectors, independent of their overall magnitude. It captures whether the model and humans distribute attention across words in a similar pattern, even when absolute attention values differ.

All things considered, these metrics assess agreement in attention magnitude (Pearson), ordering (Spearman, Kendall), and distributional shape (Cosine) between machine-predicted and human attention.

5 experiment results↩︎

This section discuss our experiment results.

5.1 RQ1: Comparison with Human Reference and Baselines↩︎

Table 3 showcases our results. Our model reaches a higher level of similarity to the human reference attention in all three datasets. However, we find that all models achieve a degree of similarity to the reference, implying that human visual attention patterns over source code are at least somewhat learnable by a range of artificial neural models. We find that human visual attention is partially consistent according to our measurement metrics, and machine models even exceed similarity to the aggregated reference than some humans do. Human Reference Similarity We report aggregate human reference similarity for each study in Table 3 for a comparison point, but with the caveat that differences among individual participants can be large. Consider the Table ¿tbl:tab:human-reference-sim? which shows Pearson correlation for each participant to all other participants for the Java methods in the Wallace study. Two points stand out. First, no participant shows uniformly high agreement across all methods. However, second, correlation tends to be in the neighborhood of positive 0.40 throughout. These points hint at a signal of human attention patterns hidden among noisy individual, task, and artifact-dependent factors. Thus, the human reference we report should be interpreted as a “ballpark” comparison point for our model rather than as an idealized notion of perfect agreement.

Model Similarity to Human References We observe that our model has the highest similarity to a human reference in all three of the datasets and all four metrics. For example, Pearson correlation is 0.45 in the Wallace study versus 0.27 for one baseline and 0.21 for another. One possible explanation is our readout mechanism for extracting attention from the final transformer layer. Indeed, Saha et al. [66] conclude that when modeling the visual system with artificial neural networks, the “choice of readout mechanism significantly impacts prediction accuracy.” We extract attention directly from the component of the model intended to mimic and leave the model’s original autoregressive design intact.

At times the machine models exceed the human reference according to the correlations we calculated. What this means is that, for certain functions and words, the model’s predicted attention aligns more consistently with the aggregate human attention pattern than any single individual participant aligns with the rest of the group. This does not imply that the model is “more human” than the humans. It reflects that the model estimates human attention overall, whereas individual human measurements are subject to personal variability. Note that at all times cosine similarity is the highest among humans, implying that people distribute attention across words in more similar patterns to each other than a model. This observation is consistent with findings in other cognitive studies, where averaging across observers reveals robust attentional structure that may be obscured at the level of single-subject measurements [67]–[69].

Dataset Differences Similarity measures are overall lower in the Smith dataset than in the Wallace or Rodeghero datasets. We note that in some cases model performance is quite low over the Smith dataset, as the baselines appear to struggle to have a positive correlation at all. Even the human reference attention similarity is relatively low. One likely explanation is the nature of the C programming language itself. Functions in C tend to be shorter than in Java, and the words used tend to be shorter and to contain less semantic information [70], [71]. Therefore, even an expert human reader has less reason to read one word more than another as the program behavior must be understood from the interaction of many different variables. Another possible explanation is the task the participants performed – some research suggests task type has more effect on eye movement than language [72]. The Smith study concerns locating bugs, which could require different patterns of skimming of the eyes [73]. Both the Wallace and Rodeghero concern writing code descriptions, with Wallace allowing navigation through code projects and Rodeghero limited to one function at a time.

5.2 RQ2: Ablation Study↩︎

We observed that our model with Pearson correlation outperforms the model without Pearson correlation. We show the results in Table 3. One possible explanation is that Pearson correlation guides model to learn the linear relationship in the human attention. However, the increased performance of our model is not only due to Pearson correlation in the loss function coming from the three other metrics. Spearman and Kendall are both higher in all cases, implying better rank of words. Specifically, our model with Pearson correlation is more likely to produce a rank similar to that of a person’s visual attention than baselines in a task of ranking words in source code. The finding is corroborated by cosine similarity, which helps adjust for differences in total eye dwell time. Some people may read more slowly, so they could have higher attention to all words they read, even if they pay more attention to the same words as other people do. A higher similarity of these three metrics implies that our model is superior in terms of comparing words relative to each other. This relative comparison may be more useful in practice than the absolute attention because software engineering tools often need to make binary decisions about what to show or not show [74].

6 scanpath prediction↩︎

This section demonstrates the effectiveness of our models with scanpath prediction, which is a task of predicting a sequence of fixations

6.1 Research Questions↩︎

Our research objective is to evaluate whether the model finetuned with the human attention improves the model’s capability to solve the problems that require the understanding of human thought process. We use scanpath prediction as an example and ask the following Research Questions (RQs):

  RQ3

What is the performance of models finetuned with human attention in scanpath prediction compared with the baselines?

The rationale behind RQ3 is that the model finetuned with human attention may not improve the task that requires the understanding of human thought process. We compare our model (i.e., the mode trained with the Pearson correlation loss) with the baseline proposed by Bansal et al. [11].

6.2 Dataset↩︎

We use the dataset designed specifically for scanpath prediction proposed by Bansal et al. [11]. They conducted an eye-tracking study with 27 programmers while doing the source code summarization. The study is conducted on the machine with Tobii Pro Fusion eye-tracker. Bansal et al. [11] filter the noise by using velocity-based fixation and low-pass filter. This results in the scanpath for both reading source code and writing summary. We holdout one function each time and calculate the average score for each metrics. We do not use the data from Wallace study [3] and Smith study [26] for scanpath prediction because the goal of both Smith study and Wallace study is to investigate how programmers understand source code to write a summary that describes the purpose of the function and localize bugs in the bigger context. These studies provide the participants the entire projects and investigate what context the programmers use to finish the task. Remember that the context size of the jam model that we used is only 1,024, which is not possible to have the entire project as input. The investigation of the dataset that requires larger window size is beyond the scope of this paper. We do not consider Rodeghero study [24] because this dataset is an older dataset with smaller sample size. All things considered, we only use the dataset proposed by Bansal et al. [11] because it aligns better with the current literature in scanpath prediction for software engineering tasks.

6.3 Metrics↩︎

We used four different metrics based on the current literature, i.e., Normalized Levenshtein Distance (NLD), Normalized Discounted Cumulative Gain (NDCG), Recall@k, and Precision@k. We provide our rationale for each as follows:

• Normalized Levenshtein Distance (NLD) measures how many edits, adds, and substitutions are required to transform one string to another string. NLD normalized the value of Levenshtein distance to a range between zero and one by dividing it by the length of longest sequence, where zero means two sentences are identical and one means two sentence require the maximum edits, addition, or substitutions to transforms one sentence to another.

• Normalized Disounted Cumulative Gain (NDCG) measures whether the model gives the highest rank to the most relevant tokens. The range of NDCG is between zero and one, where zero means the model does not demonstrate the ranking capability and one means the perfect ranking.

• Recall@k measures how many tokens are correct in the k predicted tokens. The range of recall@k is between zero and one, where zero means the model does not generate any correct tokens in k predicted tokens and one means all of the generated tokens are correct.

• Precision@k measures how many top k predicted tokens are correct. The range of precision@k is between zero and one, where zero means the model does not generate any correct tokens in k predicted tokens and one means top k generated tokens are correct.

Together, these metrics assess whether two strings are similar (NLD), the ranking capability of the model (NDCG), and whether the model generate the correct tokens (recall@k and precision@k).

6.4 Baseline↩︎

We used the baseline in software engineering proposed by Bansal et al. [11] for comparison. We used this baseline because this baseline is most similar to our model. Bansal et al. [11] modeled scanpath prediction as a finetuning task. They used the jam model as their experiments. We used the same configurations and prompt as in the original paper for the baseline. We used the same prompt as in the baseline for our models. We also include two different close-sourced commercial models (i.e., Claude and GPT-5) as our baseline. We used the claude-haiku-4-5-20251001 as our Claude model. We used gpt-5-nano-2025-08-07 as our GPT-5 model, which is the newest GPT-5 Nano model.

6.5 Experiment Results↩︎

We observed that our model significantly outperforms the baseline proposed by Bansal et al. [11] in NLD on the reading task. We show the results in Table 4. We also observed the positive improvement in NDCG, precision@k, and recall@k. For example, compared with the baseline proposed by Bansal et al. [11], we observed 8% and 17% improvement in NDCG for five and six tokens prediction tasks respectively. This shows that our model is able to generate the scanpath similar to the human programmers and rank the most relevant tokens in the highest positions. One possible explanation is that our model has already been finetuned with human visual attention. This finetuning provides the model with guidance on where to focus in the software engineering tasks. We also observed that the performance of those commercial models (i.e, GPT-5 and Claude) is even worse than our baseline. One of the possible explanations is that those models are trained with large-scale data from the Internet. However, those commercial models could look at different tokens to do software engineering tasks compared with human programmers [75] and demonstrate an alien thought process [76]. Overall, we observed that our model is more capable of the task required the understanding of human thought process.

We observed a positive improvement on the writing task compared with Bansal baseline. We show the result in Table 4. Specifically, we observed a 2% improvement in NLD and a 7% improvement in NDCG on the six-token prediction task although the differences between ours and the baseline proposed by Bansal et al. were not statistically significant. A possible explanation is that human programmers could have the different visual attention in the writing task compared to the reading task. In fact, Karas et al. [15] found the farther reading distance when programmers are doing writing tasks. Overall, our results still show the positive improvement over commercial models and the Bansal baseline on the writing task although our model is only trained on visual attention during source code comprehension.

We show one reading example in Table ¿tbl:tab:reading95example?. We observed that all of the commercial models (GPT-5 and Claude) first focus on the words in the function signature. For example, we observed that the first token of those commercial models is genSql. This aligns with the current literature that function signature provide more information for source code summarization models [77]. However, we observe that the first token that the human programmer look is SqlcPrettyPrinter, which is in the body of the function. Our model is able to predict SqlcPrettyPrinter as the first token and public as the last token, which aligns better with the human programmers. We also observe that Bansal baseline is not able to genearte the correct tokens. Overall, we found our model finetuned with human attention generates a scanpath closer to the scanpath of human programmers although we also caution that our model is more human-like compared with baselines.

public void genSql() throws PositionedError {
    try {
      SqlcPrettyPrinter         spp;
      spp = new SqlcPrettyPrinter(ref.getFile());
      spp.printCUnit(elems);
      spp.close();
    } catch (IOException ioe) {
      ioe.printStackTrace();
      System.err.println("cannot write: " + ref.getFile());
    }
  }

7 threat to validity↩︎

We follow Wohlin et al. [78] to organize threat to validity into threat to internal validity, external validity, and construct validity. The threat to internal validity is that the baseline that we compare may not be a fair comparison. We mitigate this threat by including the baseline from both computer vision and software engineering. We mitigate this threat on scanpath prediction by including both the baseline from the current literature and the strong commercial models as our baseline. The threat to external validity is that our model might not be able to generalize to other software engineering tasks and programming languages. We mitigate this threat by using diverse dataset. We use the dataset from two different tasks and two different programming languages although it is possible that our model does not generalize to tasks other than source code summarization and bug localization. The threat to construct validity is that the metrics that we use may not be appropriate to measure our tasks. We mitigate this threat by using the metrics from the current literature.

8 limitation↩︎

First, our supervision signal, ptgt, aggregates visual attention across time and collapses reading order and eye movement regressions into a single scalar per token [24], [40]. As a result, our model captures where attention is allocated but not how attention may change over time. Second, we caution that we do not intend to take a side in the “attention equals explanation” debate [79]–[81] even though our model design may be characterized as a “surgical intervention” [82]. We are studying human and machine attention itself in this paper, not its downstream effect and we do not claim that our model implements the same mechanisms that give rise to human attention (our model is not a surrogate human). We also caution that our results may or may not generalize over other datasets, languages, or tasks. Our model is not a complete account of all aspects of attention, but is nonetheless a step forward in computational modeling of human programmer visual attention. Finally, we caution that our results for scanpath may not be able to generalize to a dataset that requires large context due to our model size.

9 Discussion↩︎

A central finding of this work is that a Transformer language model can predict aggregate human visual attention over source code with a level comparable to that of individual human participants relative to their group when we finetuned with eye-tracking supervision. This finding reflects the well-known phenomenon that averaging across observers reveals stable attentional structure even if that structure is partially obscured by individual variability. In our results, this effect is measurable when comparing machine-human similarity and the human reference baseline. Individual programmers exhibit idiosyncratic reading strategies, but they form a consensus on overall need to attend to certain words. This finding is consistent with studies comparing, for example, word-level attention of blind and sighted programmers [74].

We demonstrate a model that is optimized against an aggregate attention signal (percent total gaze time). In effect the model learns a population-level attentional prior over tokens in software source code. In this way, the model should be viewed as an estimator of aggregate attentional structure across programmers rather than as a surrogate human. Consider the example in Figure 1. Subfigure (a) of the example shows the aggregate ptgt we calculate, where darker shades of red indicate more attention. The list of words includes all which were read by any person in the source code for the Java method compareIds() above. Consider the token compareTo. This token is the name of a method in a method invocation. Method invocations are critical to program behavior. There is a long-observed tendency of artificial neural networks to attend to method invocations due to their position as connection points among source code components and co-occurrence with other words [83]–[86]. Thus, it is not surprising that ours-nocorr predicted relatively high attention to this word, visible as medium-dark red highlighting in Figure 1b.

Nevertheless, the aggregate observed human attention to compareTo is low (Figure 1a). The model labeled ours correctly predicted low attention to that word (Figure 1c). The possible explanation is that this model received the ptgt visual attention as a signal during training. The aggregate attention paid to method names in an invocation is actually fairly low. In fact, [87] found that programmers typically read the inputs to the invocation much more than the invoked method’s name, a finding also consistent with the high attention prediction for the word fco2, which is an input to the compareTo method. Therefore, even though neural models tend to have high attention to method invocation names, this high attention is inconsistent with human visual attention, and our model’s prediction was in closer alignment with this human reference.

From a technical perspective, one of the more consequential design decisions in our model is to extract attention from the final Transformer layer rather than training a new output layer specifically to generate attention predictions. This decision allowed us to keep the model’s architecture intact and instead only use a custom loss function during finetuning. The model \(\mathcal{M}^{\text{ft}}\) we create has the same autoregressive behavior as \(\mathcal{M}^{\text{pre}}\). Therefore our model retains the ability to respond to prompts. An intriguing possibility, hinted to by related work [23], is that the model \(\mathcal{M}^{\text{ft}}\) may exhibit more human-like responses to prompts than \(\mathcal{M}^{\text{pre}}\) because its attention mechanism is more human-like. We showed that our model has positive improvement over scanpath prediction tasks, which is a task that requires the understanding of human thought process.

10 Conclusion↩︎

We move the stat-of-the-art threefold. First, we develop a novel loss function to train a model with the data from the eye-tracking studies. We used two programming languages (Java and C) and two programming tasks (bug localization and source code summarization) as examples. We showed that our model outperforms the close baselines in both computer vision and software engineering.

Second, we conducted an ablation to study to investigate the impact of the loss function with Pearson correlation. We found that Pearson correlation guides models to learn the linear relationship in the data from the eye-tracking studies. We also show that Pearson correlation is not the only contribution to the performance based on our experiments results with other metrics (i.e., Kendall’s \(\tau\), Spearman rank correlation, and Cosine similarity).

Finally, we demonstrated that our model is more capable of the tasks that require the understanding of human thought process. We used scanpath prediction as examples. We show the improvement in both reading and writing tasks compared with the close baseline and commercial models (i.e., GPT-5 and Claude). We also caution that our model may not be able to generalize to a task with large context windows size.

We release the source code in the online appendix at https://github.com/apcl-research/human-attention-prediction.

ackowledgment↩︎

This work is supported in part by the NSF grants CCF-2100035, CCF-2211428, and CCF-2211429. Any opinions, findings, and conclusions expressed herein are the authors’ and do not necessarily reflect those of the sponsors. We also sincerely thank participants of our qualitative study.

References↩︎