1 Introduction↩︎

Language models (LMs) have great potential as tools for studying language. The success of a domain-general statistical learner on many linguistic tasks has prompted some researchers to argue that LMs should be more central in evaluating and developing linguistic theory [1], [2]. In this work, we focus on constructionist approaches, or Construction Grammar ([3]–[5], inter alia), a family of linguistic theories which posit that language is primarily structured as form-function mappings of gradient complexity. Construction Grammar accounts for not only the most integral parts of linguistic structure (e.g.verb argument structures; [3]), but also rare phenomena traditionally relegated to the “periphery”, arguing that both can be accounted for with similar representations [6]. Construction Grammar’s ability to describe and account for rare linguistic phenomena makes it a valuable framework for studying language models, as mastery of rare and complex constructions is a crucial part of humans’ linguistic knowledge, and may be particularly challenging for language models due to scarcity in their input.

At the same time, the general success of neural language models, which do not explicitly distinguish between syntax and semantic information, has led some researchers to argue that Construction Grammar is more aligned with LM processing of language relative to other frameworks [7]–[9]. In order to fully evaluate these claims, it is worth investigating the extent to which LMs can serve as “model learners” [1] of constructionist theories of language.

Because Construction Grammar posits that form and meaning are not separable parts of linguistic knowledge, a constructionist “model system” should learn and understand constructions with respect to both their forms and functions. However, up to this point, there are somewhat mixed results regarding LMs’ syntactic and semantic knowledge of rare constructions. There is ample evidence that even relatively small language models learn formal properties of rare constructions, including Article-Adjective-Numeral-Noun [11], [12], [13], and the Comparative-Correlative [14]. However, in many cases, these smaller models seem unable to grasp the semantics of these rare constructions [13], [14]. On the other hand, extremely large LLMs have been shown to have relatively nuanced semantic understanding of a range of constructions [15], [16]. Since most LLM evaluation has been done in the prompting setting on closed-source LLMs, it remains unclear whether semantic understanding of constructions can be observed in the raw probabilities of (smaller) language models, particularly for exceedingly rare constructions.

Additionally, it is not clear that all current evaluation datasets are well suited for testing constructional semantic understanding. Past work relies on unnatural sounding test items [13], [14] or more complex metalinguistic tasks which may limit LM performance [17]. Additionally, particularly for rare constructions, it is likely that constraints on their behavior are related to more general phenomena in language [11], [18], and it is reasonable to expect that the semantics of constructions are shaped and constrained by their interactions with other domains of semantics.

In this work, we address the above gaps with a new dataset for testing a rare² family of four constructions: , , , and , collectively known as constructions because they conjoin two phrases that are both in focus:

. He doesn’t like shrimp, let alone squid. [6]

We evaluate syntax and semantics on a much wider range of models than has been tested in past work on constructional semantics. We ground the creation of our dataset in other factors in language production which are likely to inform learning of the constructions’ semantics: scalar adjectives and general world knowledge.

This allows us to ask and answer detailed research questions about the acquisition of semantics.

1) How do training data, parameter count, and pretraining objective impact knowledge of meaning? We show that a range of medium-sized open-source models show sensitivity to the construction-level semantics in their raw probability distributions. However, we do not observe any above-chance knowledge of semantics for models trained on human-scale data (BabyLMs, [19]). 2) When are form and meaning learned throughout pretraining? We show that form is consistently acquired prior to semantics. Furthermore, we show that inferences early in learning are often influenced by typicality (world knowledge) rather than constructional meaning, particularly for weaker models. We find that the different constructions we test are highly correlated with one another in terms of both syntactic and semantic performance. 3) How does meaning correlate with performance on other linguistic benchmarks? We examine the learning dynamics of both the form and meaning of the constructions, in relation to performance on a range of existing linguistic benchmarks. We find that performance on our syntactic evaluations of the constructions plateaus early in pretraining (mirroring learning curves for the BLiMP [20] grammatical benchmark), while functional knowledge is acquired much later and more closely mirrors the learning trajectory of the world-knowledge-based EWoK [21] benchmark.

Overall, our results show that constructions can be learned by relatively small models (under 400M parameters). They also shed light on the interrelatedness of constructions with scalar semantics, and with domains of world knowledge in which such scalar semantics are relevant. Furthermore, the failure of all “human-scale” models on our semantic evaluation points calls into question the ability of human-scale pure text LMs to serve as model systems of linguistic theories that posit joint acquisition of form and meaning.

2 Background↩︎

2.1 Paired-Focus Constructions↩︎

In this work, we focus on four related constructions: , , , and . We focus on these constructions because their syntactic and semantic properties are well established in Construction Grammar theory [6], and have also been studied in past computational work (e.g.[12], [13], [17], [22]), though past work on investigating language model understanding of semantics has yielded mostly negative results.

[6] provide a comprehensive treatment of the construction, though much of their analysis can be extended to the other constructions in this work. Syntactically, these constructions function somewhat similarly to coordinating conjunctions (Examples [ex:pf] and [ex:and]), but resist various types of syntactic movement (Example [ex:movement]), and generally behave like negative polarity items (Example [ex:npi]).

. He doesn’t like shrimp, or squid.

. ??It’s shrimp, not to mention squid, that he doesn’t like.

. ??I like shrimp, much less squid.

Regarding semantics, the constructions indicate a relationship between two conjoined and focused elements which are being compared. The comparison between the focused elements evokes a scalar relationship with the two elements representing “two points on a scale” [6]. Generally, the second focused element has a higher value on the scale than the first focused element. Overall, the semantics of constructions are related to scalar semantics more generally (in the invocation of a scale) and to world knowledge, which informs what scalar properties are natural for the focused items in the construction.

2.2 Related Work↩︎

2.2.1 Linguistic Capabilities of LMs↩︎

Broadly, this work follows in a long line of research seeking to investigate LM knowledge of linguistic phenomena by examining LM probabilities for grammatical and ungrammatical sequences ([23], inter alia). Among the most relevant of these works are those which seek to design datasets to assess broad domains of linguistic abilities, whether that be syntactic, conceptual, or world knowledge. We leverage several of these datasets as comparison points for learning dynamics of constructions. Furthermore, investigations of scalar semantics of adjectives [24]–[27] and adverbs [28] are of particular relevance to this present work, though most past work has focused on scalar implicature and relative intensity of adjectives, which is orthogonal to the present study. Contributing to this line of work are several datasets containing scalar adjectives of varying intensities [29]–[31]. We rely on the dataset of [30] for the scales and adjectives in our dataset.

2.2.2 LM Understanding of Rare Constructions↩︎

Past work on evaluating LM understanding of constructions has primarily focused on evaluating various constructions that have been outlined in theoretical linguistics, such as argument structure constructions [32]–[34], the Comparative-Correlative [14], Causal-Excess [35], NPN [36], PiPP [18], AANN [11], [37], [38], and [13]. Other work opts for a more general investigation of a range of constructions [12], [16], [39], or discovers constructions using unsupervised methods [40]–[42]. While most work on constructional understanding in LMs has focused on English, there is growing work on evaluation of constructions in languages beyond English [43]–[46].

constructions have been addressed in several past works. [22] and [12] use Global Affinity measures to show that both RoBERTa and a range of BabyLMs have knowledge of collocations for the constructions and . Our results robustly replicate theirs. [17] and [16] include as one of several constructions and find that large, closed-source LLMs can perform a metalinguistic grouping task and natural language inference successfully on examples of these constructions. Our work diverges from these approaches in that we test smaller models using direct, probability-based evaluations.

Our work is most similar in approach to [13], who design probability-based measures for syntactic and semantic properties of . As discussed previously, we argue that their dataset for semantics relies on arbitrary contrasts which do not clearly implicate scalar semantics. Furthermore, they only test a single model architecture (OPT-125M) and data scenario (100M tokens of BabyLM), and only address and not the highly related , , and .

3 Paired-Focus Dataset↩︎

We construct a novel dataset to test the semantics of the constructions. Past work [13] used a fully arbitrary dataset, where there was no obvious scalar relationship between the focused elements (outside of what may be communicated by the construction itself):

. I couldn’t lift the orange crate, let alone the green crate. [13]

Failure on fully arbitrary evaluations could be due to several factors, apart from the models truly not understanding the construction. It is also possible that they may be unable to map the arbitrary relationship between the two focused elements onto any scale, which would also lead to a failure to understand the construction overall. If this is the case, we would expect the models to fail to comprehend arbitrary examples (like Example [ex:arbitrary]) but succeed at understanding examples with a coherent scale. Furthermore, it is possible that the models have only limited knowledge of the construction, where the constructions can only be interpreted in contexts where the scalar relationship between the focused elements is already obvious due to world knowledge. We would then expect the models to succeed in cases where there is a clear scalar relationship that makes sense in the context of world knowledge but fail in cases where there is a clear scalar relationship which conflicts with world knowledge.

To distinguish between these possibilities, we construct a new dataset that contains examples where the focused elements have clear scalar relationships between them. We use adjectives and scales from [30] as a starting point for the focused elements that will be compared within the construction. In all examples, we pair adjectives from their dataset with a set of common nouns that are natural sounding with the chosen scalar adjectives³. In total, we generate 198 starting templates across 4 unique scales from [30], and then fill the resulting templates with a set of appropriate adjectives and nouns, resulting in a dataset of 3.5k example sentence pairs per construction. Example sentences for all scales are shown in Table 4 in Appendix 8.

3.1 Evaluation of Dataset↩︎

To test model sensitivity to semantics, we measure the effect that the constructions have on model output probabilities on a follow-up sentence that would be entailed by the construction. Because constructions imply that the second focused element is a higher value on the scale than the first focused element [6], we include follow-up sentences which either entail or contradict this scalar relationship. Models are expected to prefer the follow-up which reinforces the semantics of the construction to one which contradicts the construction (compare 6 and 7 in Table 2).

Directly comparing the model probabilities of follow up-sentences is confounded by the logical plausibility of the follow-ups independent of the construction. We expect that, irrespective of the presence of a construction, a good model will assign a higher probability to a follow-up that aligns with world knowledge, all other things being equal. To control for this fact, we compare example pairs with a construction to examples without a construction with scalar semantics, specifically the simple conjunction “or” (see 8 and 9, Table 2). Intuitively, if the model is sensitive to semantics, we expect a model to assign higher probability to a follow-up that aligns with the construction, beyond what is assigned to the follow-up due to world knowledge.

Concretely, given a base sentence \(S_{+{\mathbf{Cxn}}}\) and a follow-up sentence \(T_{\pm{\mathbf{Plausible}}}\), we calculate the difference \(\Delta P\) as follows. Here, \(\iota_{\theta} = - \log p_{\theta}\) denotes the surprisal value derived from a language model parameterized by \(\mathbf{\theta}\). \[\begin{align} \Delta P(+\mathbf{Cxn}) &=\iota_{\theta}\!\left(T_{-\mathbf{Plaus.}} \mid S_{+\mathbf{Cxn}}\right) \\ \notag &\quad - \iota_{\theta}\!\left(T_{+\mathbf{Plaus.}} \mid S_{+\mathbf{Cxn}}\right) \end{align}\]

\(\Delta P(-\mathbf{Cxn})\) for the control condition with “or” is computed similarly. Finally, given a dataset with \(k\) example pairs, we then derive an accuracy score by comparing the condition and the control (“or”) condition: \[\label{eq:acc} \mathbf{Acc_{PF}} = \frac{1}{k} \sum_{i=1}^{k} \mathbb{1}[\Delta P(+\mathbf{Cxn}) >\Delta P(-\mathbf{Cxn})]\tag{1}\] There are other confounding factors (beyond the construction and world knowledge) that could influence LM probability associated with the follow-up sentences. Since the two follow-ups are minimal pairs (with only one word different between them) there is a potential confound of lexical bias if the correct sentence has a more frequent/probable word irrespective of context (e.g.since “easier” is more frequent than “harder” overall). Another potential confound is if there is an ordering effect of the two focused elements, where a follow-up sentence is preferred or dispreferred for presenting focused elements in the same order as the sentence. We control for both of these confounds by balancing both the plausible and implausible follow-ups in our dataset by ordering and by the lexical items that appear in the entailed follow-up.

3.2 Syntactic Evaluation↩︎

Beyond evaluating semantics, we also wish to test if models have knowledge of the forms of constructions. We create a syntactic evaluation suite which adapts the syntactic tests from [13] to our dataset. Specifically, we select three syntactic manipulations from [13] which are ungrammatical for constructions but grammatical for simple conjunctions (see 3). Similarly to our semantic tests, we evaluate the \(\Delta P\) values between grammatical and ungrammatical sentences compared to simple conjunctions. For more details on the syntactic evaluation suite, as well as evaluations of Global Affinity [22], see Appendix 10.

4 Experiment 1: Impact of Model Size and Training Data↩︎

4.1 Models and Evaluation↩︎

We test a total of 36 models, varying in parameter count, pretraining data size, pretraining objective (masked vs. autoregressive)⁴, and model family. See Table 9 in Appendix 12 for model details. In addition to computing accuracy, we run a linear mixed effects model analysis to test the impact of model size, pretraining data, and architecture across all models. The dependent variable is accuracy (see Equation 1 ). We include pretraining data (log # of tokens), parameters (log # of parameters), and architecture (causal vs. MLM) as fixed effects, and a random intercept for model name.

4.2 Results↩︎

Figure 1 visualizes the accuracy⁵ as a function of model size and training data amount. Below a parameter threshold of roughly 400M, model performance is generally at or below chance regardless of the amount of training data (with the exception of Ettin-encoder-150M). Beyond this parameter threshold, there is a generally positive relationship between training data and accuracy as well as parameter count and accuracy. However, there is substantial variation between individual models, regardless of parameter count, training data, and training objective.

For our linear modeling analysis, we find that only parameter count has a significant main effect (\(\beta= 6.055\), \(p=.011\)). As a follow-up analysis, we additionally fit a series of linear mixed effects models assessing each predictor independently. For single predictor models (again with a random intercept for model name), we find that both parameter count (\(\beta= 6.607\), \(p=003\)) and pretraining data (\(\beta= 2.651\), \(p=.034\)) are significant, though the effect of parameter count remains larger. While we note that MLM and autoregressive scoring is not directly comparable, we find no significant effect of pretraining objective.

Regarding our form evaluations, we find that even very small models have strong performance on our syntactic evaluation suite, and assign high Global Affinity scores to fixed slots in the construction (see Tables 7 and 8 in Appendix 10), echoing past results [12], [13], [22]. The dissociation between tasks for small models underscores the need to evaluate both form and semantics when assessing LMs’ overall knowledge of rare constructions.

In summary, these results indicate that models far smaller than frontier LLMs can grasp constructions, as we find that a few models as small as 400M parameters succeed at learning (with 90%+ accuracy) both the form and semantics of the constructions, and most of the larger models are above chance. There seems to be a broadly positive relationship between parameter count and semantic accuracy, while the effect of training data independent of parameter count is not well supported by our results. Knowledge of the form of constructions is present even in the smallest models we test.

5 Experiment 2: Learning Trajectory Experiments↩︎

In this experiment, we investigate when constructions are learned in training in relation to other linguistic knowledge. We hypothesize that prior to model knowledge of semantics, we will observe knowledge of constructional forms. We further hypothesize that examples which align with appropriate scalar semantics (given general world knowledge) will be acquired first, especially for models that are less proficient at understanding the constructions.

To operationalize semantic and formal knowledge of constructions, we use accuracy for semantics (Equation 1 ) and our syntactic test suite identical to Experiment 1. In order to test how alignment of scalar semantics with world knowledge interacts with learning of constructions, we further test models on sentences where constructional examples entail implausible follow-up sentences and contradict plausible sentences (see 10 and 11 in Table 2). Like in Experiment 1, we consider an example correct if the presence of the construction shifts increases the probability of the follow-up entailed by the construction, relative to a baseline conjunction “or”. Intuitively, if a model has a fully abstract understanding of semantics, we expect the presence of a construction to increase the probability of an otherwise implausible statement that is consistent with the scalar relationship implied by the construction.

5.1 Comparison with Linguistic Benchmarks↩︎

We further evaluate the learning dynamics for three other linguistic benchmarks as comparison points throughout training. We evaluate BLiMP [20], COMPS [47], and EWoK [21]. BLiMP tests model knowledge of a range of syntactic constructions. We expect this dataset to be learned relatively early. COMPS tests knowledge of conceptual properties of noun classes. Knowledge of some properties tested in COMPS may be relevant to linking the noun phrases in our dataset with appropriate scalar semantics, and thus we expect high accuracy on COMPS to precede semantic accuracy. EWoK tests a range of domains of world knowledge, including physical and material properties. Understanding of these properties is crucial to interpreting the scales in our dataset, and thus we expect that high performance on EWoK will correlate with semantic accuracy.

5.2 Models↩︎

We focus on learning dynamics of three models in Experiment 1: Pythia-12b [48], Ettin-encoder-400m, and Ettin-decoder-1b [49]. We select these models because they are the top three performing models at their final checkpoints in Experiment 1. For Pythia-12b, we sample 19 logarithmically spaced checkpoints from start to finish in training. For Ettin models, where logarithmically spaced early checkpoints are not available, we evaluate the first 50 checkpoints⁶ for each model. For each of these model checkpoints, we evaluate semantic accuracy (for both plausible and world-knowledge implausible examples), syntactic accuracy, and accuracy on each of the comparisons.

5.3 Results↩︎

For the purpose of clarity, we primarily discuss and visualize results for Pythia-12b in the main body of the paper. Results on Ettin models (for details see Appendix 11) are broadly qualitatively similar, but display noisier and more inconsistent learning trajectories. We also find that the Ettin models display a more substantial difference in constructional effect on plausible compared to implausible follow-ups, especially early in training. In contrast to the Ettin models, which seem more reliant on world knowledge cues for their interpretations of constructions, Pythia-12b displays a sensitivity to the construction which is robust to implausible contexts.

Results for Pythia-12b are visualized in Figure 2. We find strong evidence that form is learned prior to semantics. Syntactic accuracy reaches a peak much earlier in training than accuracy. We find performance on semantic evaluations follows similar trajectories regardless of the plausibility of the follow-up sentence, though plausibility does lead to higher absolute performance at the end of training.

For both BLiMP and COMPS, accuracy plateaus relatively early in training, substantially before the accuracy rises above chance. For EWoK, performance gains are more gradual, similar to gradual gains on semantics. ⁷

We report results for each construction individually and graph results for Pythia-12b in Figure 3. We observe that performance across the constructions is generally strongly correlated, with all constructions following broadly similar trajectories. Of the four, is learned more linearly across training, while performance curves for the other constructions are more logarithmic and peak later in training. While we cannot establish a causal relationship, we note that is the most frequent and least ambiguous of the four (see Table 1) and thus it is perhaps unsurprising that it is learned earlier in training.

5.4 Correlation Analysis of Learning Trajectories↩︎

We run a first-difference correlation between semantic performance and performance on other benchmarks. We find negligible correlation between form and semantics, and between semantics and BLiMP. We find a moderate correlation between semantic performance on plausible and implausible follow-ups, further showing Pythia-12b has substantial knowledge of scales implied by constructions beyond the scalar relationships evident through world-knowledge alone. When subdividing EWoK into its different world knowledge domains, we find a moderate correlation between semantics and the physical relations domain (\(\rho=.48\)). The first-order correlations between formal accuracy, semantic accuracy, and EWoK physical relations accuracy are shown in Figure 4.

5.5 Discussion↩︎

Overall, these results provide strong evidence that form and meaning are learned with vastly different amounts of training input. We find that performance across individual constructions is strongly correlated, and that overall performance is moderately correlated with performance on relevant world knowledge domains in EWoK. We find no evidence that form and meaning acquisition are correlated in any way, nor is semantics significantly correlated with the more syntactic BLiMP benchmark. Regarding model comparison, we find that the learning trajectories of Pythia-12b are much more stable than the trajectories of the smaller Ettin models, which are more susceptible to spikes and valleys in performance across training, and are less proficient at semantics when follow-up sentences are implausible.

6 Discussion↩︎

In this work, we have shown that medium-sized models (\(\approx\)​400M parameters) can acquire knowledge of semantics. However, small models, and models trained on human-scale data, show a clear gap in performance on form and meaning at their final checkpoints. Furthermore, we have shown that even models which eventually learn meaning only do so much later in training relative to form. The performance gap we observe for constructions echoes past results on Comparative-Correlative [14] and Causal-Excess [35] constructions, which similarly found that LMs fail at semantic evaluations which target those constructions. However, we note that our syntactic tests are inherently different from our semantic tests, and interact with different parts of grammar (e.g., other constructions like conjunction and pseudoclefting for syntactic tests vs. scalar semantics for semantic tests). While our results do not provide definitive evidence that form and meaning are necessarily learned separately by LMs, they underscore that modeling joint learning of constructional form and meaning (as would be ideally possible for a “model system” of constructional acquisition) has not been clearly demonstrated by LMs thus far.

While we do not claim that are results prove that LMs could never jointly acquire form and meaning via text-only language modeling, there are obvious reasons why LMs would be unlikely to model a constructionist account of language development. Humans learn language in social settings in order to perform communicative goals. We are also exposed to rich non-linguistic input in the form of embodied experience. Pure text language models do not have access to these rich sources of input, which are likely particularly relevant for extracting semantic and pragmatic features of language. We look towards future work on multimodal human-scale models [50], [51] and models that incorporate human feedback [52] or learning in situated contexts [41], [53] as potential avenues for future work.

Finally, we highlight the importance of evaluation design. By improving on past evaluations, we find evidence of semantic learning in models that are much smaller than previously reported. Further innovation in evaluation methods for constructional meaning may yet reveal that functional abilities emerge earlier in training, and at smaller parameter scales, than found here.

7 Conclusion↩︎

In this work, we have investigated if LMs can learn nuanced semantic interpretations for a rare family of constructions. We find that models larger than \(\approx\)​400M parameters—both encoders and decoders—are broadly successful at the task, with larger models generally performing better. We find that smaller models completely fail at our novel semantic benchmark despite robust knowledge of the constructions’ forms. This seeming divergence underscores a broader lack of evidence that text-only LMs are jointly modeling form and meaning of constructions. Turning to an analysis of learning dynamics, we show that learning of semantics occurs after learning formal knowledge of the construction, and is correlated with learning of relevant world knowledge. Our largest high performing model is robust to changes in plausibility when interpreting examples, while smaller models are more sensitive to the plausiblity of the construction. Overall, this work shows that relatively modest-sized models can acquire nontrivial semantic knowledge of rare constructions, and finds correlational evidence relating learning of semantics to learning trajectories of other realms of linguistic knowledge.

Limitations↩︎

This paper is limited in terms of its coverage of constructions. While we evaluate a range of constructions in English, it is not clear if our concrete findings would generalize to other constructions which may be less closely related than the set we test here. In addition, while we hope that our work serves as a test case of linking the semantics of a set of rare constructions to other semantic properties of language, our evidence in this paper is not causal: while performance on semantics is correlated with certain realms of world knowledge, it is not clear how learning would be impacted if such knowledge were perturbed or removed altogether from training. Furthermore, while models displayed some robustness to implausible follow-up sentences, it is not clear how plausibility is causally linked to learning of constructions. The number of scales that we used to create our dataset was somewhat small; increasing the number of scales used (and using less common scales) could impact the results here, especially for smaller models, which display some borderline knowledge of semantics. Finally, constructions are not unique to English, but our evaluations were limited to English constructions and primarily monolingual models.

Acknowledgments↩︎

Leonie Weissweiler was supported by a postdoctoral fellowship from the German Research Foundation (DFG, WE 7627/1-1). This research was supported in part by NSF award IIS-2144881. We thank CoNLL reviewers and members of the NERT and PiCOL labs for their insightful comments which improved the final version of this paper.

8 Dataset Examples↩︎

This Appendix contains examples for each frame and construction. See Table 4. All examples in our evaluations are presented with all four constructions: , , , and .

4pt

9 Experiment 1 MLM Replication with Pseudo-Log-Likelihood↩︎

Here, we replicate the semantic tests from Experiment 1 for MLMs using Pseudo-log-likelihood (PLL, [54]), using the formulation from [55]. We use the minicons library for computing all PLL scores. Table 5 reports the accuracy on the semantic benchmark as measured by PLL (compare with Table 6, which contains the full results for the semantic evaluations from Experiment 1). We see similar trends in accuracy as we do in our main evaluation for Experiment 1, where we compare probabilities of masked tokens directly. All MLMs with less than 150 million parameters achieve chance or near chance performance in both settings. We find that ettin-400m, ettin-1b, and ModernBERT-large are the top performing models in both settings.

10 Experiment 1: Syntactic Evaluations↩︎

We develop a syntactic evaluation suite which is based upon several tests from [13]. The tests focus on three grammatical properties, specifically targeting alternations which are grammatical with simple conjunctions but generally ungrammatical for constructions (see Table 3). Similar to our evaluations on semantics, we measure \(\Delta P\) between two conditions, relative to the difference in those conditions when a simple conjunction is present. Unlike the semantic tests, there are no follow-up sentences; rather, we evaluate the differences in probabilities for sentences with constructions directly. More specifically, given a base sentence with a construction \(S_{-\mathbf{Manip.,+\mathbf{Cxn}}}\), we expect that an ungrammatically manipulated sentence \(S_{+\mathbf{Manip.,\pm\mathbf{Cxn}}}\) will have a relatively high surprisal value. Thus, we compute for a example:

\[\begin{align} \Delta P(+\mathbf{Cxn}) &=\iota_{\theta}\!\left(S_{+\mathbf{Manip.,+\mathbf{Cxn}}} \right) \\ &\quad - \iota_{\theta}\!\left(S_{-\mathbf{Manip.,+\mathbf{Cxn}}}\right) \end{align}\]

We then compare to \(\Delta P(-\mathbf{Cxn})\), which is computed similarly using “or” sentences with manipulations. We calculate accuracy based on \(\Delta P\) values identically to the semantics dataset: \[\label{eq:acc2} \mathbf{Acc_{PF}} = \frac{1}{k} \sum_{i=1}^{k} \mathbb{1}[\Delta P(+\mathbf{Cxn}) >\Delta P(-\mathbf{Cxn})]\tag{2}\]

Results on our syntactic evaluation suite are reported in 7.

Additionally, to test general familiarity with the wordforms associated with each construction, we use the Global Affinity metric presented by [22]. Global Affinity is defined as the probability assigned to a target word when it is masked in a string. Given an original string \(s\) and an index \(i\), Global Affinity defines \(P_{s \setminus\{i\}}\) as the probability distribution at position \(i\). The Global Affinity is then the probability assigned to the correct word \(w_i\) when position \(i\) is masked:⁸ \[\label{eq:glob95aff} \mathbf{GlobalAff}_{s,w_i} = P_{s \setminus \{i\}}(w_i)\tag{3}\]

In our case, we calculate the Global Affinity for each word in our target constructions (e.g., “much” and “less” for “much less”) and average across all words in the construction. We find that regardless of model size, most models have a very high global affinity for all fixed words in constructions (Table 8). This indicates that models are confident that these constructions are collocations and the fixed words in the constructions are relatively easy to predict for models of all sizes.

11 Experiment 2 Full Results↩︎

In this section, we provide extended results on learning dynamics for Ettin-encoder400m and Ettin-decoder1b. Results are shown in Figures 5 and 6 respectively. In general, the performance trajectories for semantics are substantially noisier in these models than in Pythia, with large peaks and valleys throughout training. For both Ettin models, we observe early spikes in performance on plausible examples that are not accompanied by spikes on implausible examples. Generally, performance on implausible examples does not rise consistently above chance until much later in training. Taken together, these results seem to indicate that the smaller Ettin models do learn nontrivial semantics, but may be limited to more natural contexts where the scalar relationship entailed by the construction is further supported by world knowledge. Performance on other benchmarks is generally more stable than on semantics, though performance on EWoK is noticeably less stable relative to Pythia.

12 Model Details↩︎

Table 9 presents details about the models that we test in Experiment 1. In total, we test 36 models in Experiment 1. In Experiment 2, we test 3 models: Pythia-12b, Ettin-encoder400m, and Ettin-decoder1b, which are selected due to their strong performance in Experiment 1 and the availability of their intermediate checkpoints.

References↩︎

---

1. https://github.com/WesScivetti/Meaning_Alone↩︎

2. For approximate corpus frequencies see Table 1.↩︎

3. constructions can focus other syntactic phrases besides noun phrases. To control for the confound of how syntactic structure could influence understanding, we only place noun phrases in the focused slot of the construction.↩︎

4. For MLM models, we evaluate the probability at the target region (e.g.”easier” vs. “harder”), whereas for the decoder models, we evaluate the probability of the entire follow-up sentence conditioned on the first sentence. See Appendix 9 for a replication with MLM scoring using Psuedo-log-likelihood.↩︎

5. We average accuracy scores across each of our four constructions.↩︎

6. Each checkpoint is equivalent to roughly 8–9 billion tokens of pretraining data, and 50 checkpoints is thus equivalent to approx. billion pretraining tokens.↩︎

7. Chance accuracy for EWoK is 25%.↩︎

8. Global Affinity is only defined for Masked Language Models [22]. Thus, we only evaluate a subset of our models on this metric.↩︎