1 Introduction↩︎

Over the years, the field of Natural Language Processing (NLP) has been at the forefront of understanding the core principles of intelligence. The emergence of Large Language Models (LLMs) such as GPT-4 [1], PaLM 2 [2] and LLaMA 2 [3], showcase a significant breakthrough. Thanks to unparalleled scales of model architecture and the vastness of training data, these LLMs now exhibit exceptional linguistic competence and can execute complex tasks requiring abstract knowledge [4] and reasoning [5].

Previous research has revealed that LLMs naturally capture cross-linguistic similarities in their representation space, facilitating zero-shot cross-lingual transfer [6]–[8]. The model is fine-tuned on one language, enabling the acquisition of comparable capabilities in another language [9], [10], and exhibits the phenomenon of code-switching when generating context [11], [12]. Attempts to improve LLMs’ cross-lingual generalization abilities have been successful through parameter and information transfer learning [13], [14], aligning languages compulsorily [15], [16] and utilizing in-context learning techniques [17], [18]. However, a detailed investigation into the internal mechanisms of how LLMs possess cross-linguistic alignment capability remains elusive.

To delve deeper into the intrinsic mechanisms of LLMs’ linguistic competence, this paper focuses on the LLMs’ parameter importance and investigate the linguistic regions of LLMs based on 30 distinct languages’ performance, with the purpose of figuring out the following questions:

Q1: Does a core linguistic region exist within LLMs that facilitates cross-lingual alignment and generalization? By conducting further pre-training across six languages and evaluating models’ parameter importance (Section 2.2), we discover a region in LLMs corresponding to the core linguistic competence, which accounts for approximately 1% of the model’s total parameters. As shown at the top of Figure 1, removing this region (setting parameters to zero) consistently leads to a significant decline in performance across 30 test languages (Section 3.2).

Furthermore, by visualizing the core linguistic region (Figure 2), we observe that the linguistic core region of LLMs exhibits significant dimensional dependence. In certain dimensions, only perturbing a single parameter could lead to the model losing its linguistic competence (Section 3.3). Additionally, ablation study in 3.3.0.4 shows that beyond outlier dimensions, other non-outlier dimensions in this region are also critical.

Q2: Beyond the core linguistic region within LLMs , do distinct monolingual regions exist that specifically influence individual languages? While LLMs possess strong multilingual capabilities, we discover that each individual language (or language with similar compositional elements or grammatical structures) encompasses independent regions within the LLMs. As shown in the middle of Figure 1, the analysis of the Russian sentences identifies a particular linguistic region that likewise exerts influence both on the Russian and Ukrainian language, both of which belong to the Slavic group (Section 3.4).

Q3: If and how core linguistic regions affect further pre-training, how to utilize it to optimize further pre-training? After pre-training, core linguistic parameter regions of the LLMs are established for multilingual alignment. Notable shifts in these regions potentially lead to a decline in model lingual capabilities. Our findings reveal that freezing this core region can mitigate the issue of catastrophic forgetting [19], [20], a common phenomenon observed during further pre-training of LLMs. As shown at the bottom of Figure 1, we investigate the impact of selectively freezing 5% key parameters of all parameters during further pre-training, compared to the full-scale fine-tuning technique. Findings indicate that this method facilitates comparable learning of the target language while concurrently decelerating the rate of language attrition for previously learned languages (Section [sec:Further32Pre-training32Optimization]). Significantly, our methodology is compatible with the data-replay techniques [21], [22], with no necessity for integrating extra components into the model. Unlike regularization methods [23], [24], our approach restricts to a minimal core region in LLMs.

The main contributions of our work are summarized as follows:

• We discover that LLMs possess a core linguistic region, and removing this region (setting parameters to zero) results in a significant loss of the model’s linguistic capabilities. Furthermore, perturbations to specific dimensions or even a single parameter can lead to a substantial decline in the model’s linguistic abilities.

• We observe that distinct monolingual regions exist in LLMs for different languages. Removing a specific monolingual region causes a significant deterioration in the linguistic capabilities within corresponding language.

• We perform further pre-training for specific languages within the core linguistic region of LLMs frozen, achieving comparable performance in the target language while mitigating catastrophic forgetting in non-target languages.

2 Background and Metric↩︎

2.1 Model Pre-training↩︎

Pre-training is a crucial process by which LLMs acquire linguistic competence and gain general knowledge about the real world. Formally, given a large corpus \(\mathcal{D}\), the training objective for auto-regressive language modeling is to find the optimal \(\theta\) that minimizes the following loss \(\mathcal{L}\): \[\mathcal{L}{(\mathcal{D},\theta)} =\sum_{x\in\mathcal{D}}\sum_i\log p_\theta(x_i|x_1,...,x_{i-1}),\] where \(x=\{x_1, ..., x_n\}\) denotes an input token sequence and \(\theta\) denotes parameters of the model.

2.2 Parameter Importance↩︎

Drawing upon the observations of linguistic alignments, we propose that particular parameters regions within the model exert significant influence on its inherent language alignment capabilities. Evaluating parameter sensitivity is a crucial metric for determining the significance of parameters in model pruning [25]–[27]. If removing a parameter (zero-out) significantly affects the loss, the model is sensitive to it. More specifically, given a large corpus \(\mathcal{D}\) and \(\theta = [\theta_1, \theta_2, \ldots, \theta_d] \in \mathbb{R}^d\) as the parameters of a model, with each \(\theta_j \in \mathbb{R}\) denoting the \(j\)-th parameter, the training objective is to minimize loss \(\mathcal{L}{(\mathcal{D},\theta)}\) (defined in 2.1). The importance of each \(\theta\) is denoted as \(\mathcal{I}(\theta) \in \mathbb{R}^d\), where its \(j\)-th index \(\mathcal{I}_j(\theta)\) signifies the importance for \(\theta_j\).

Under an independent and identically distributed data (i.i.d.) assumption, the importance of a parameter \(\mathcal{I}_j(\theta)\) is measured by the increase in prediction loss when it is removed, calculated as the absolute difference between prediction losses with and without the parameter(\(\theta_j\)): \[\label{eq::absolute32difference32loss} \mathcal{I}_j(\theta) = \left| \mathcal{L}{(\mathcal{D},\theta)} - \mathcal{L}{(\mathcal{D},\theta|\theta_j = 0)} \right|.\tag{1}\]

Calculating \(\mathcal{I}_j(\theta)\) for each parameter, as outlined in 1 , is computationally expensive because it involves \(d\) distinct versions of the network computing, for each removed parameter. This becomes particularly challenging as the number of model parameters, \(d\), grows to hundreds of billions. However, similar to several prior works [27], [28], using the Taylor expansion formula for \(\mathcal{L}\) at \(\theta_j = 0\): \[\begin{align} &\quad \mathcal{L}(\mathcal{D}, \theta) = \mathcal{L}(\mathcal{D}, \theta | \theta_j = 0) \\ &+ \frac{\partial \mathcal{L}}{\partial \theta_j}(\theta_j - 0) + \frac{1}{2!} \frac{\partial^2 \mathcal{L}}{\partial \theta_j^2}(\theta_j - 0)^2 + \cdots, \end{align}\] we can estimate \(I_j(\theta)\) with its first-order Taylor expansion, eliminating the requirement for \(d\) distinct networks computation: \[\label{eq::gradient32accumulate} \mathcal{I}_j(\theta) \approx \left| g_j\theta_j \right|,\tag{2}\] where \(g_j = \frac{\partial \mathcal{L}}{\partial \theta_j}\) are elements of the parameter gradient \(g\), and the importance is easily calculated since the gradient \(g\) can be obtained from backpropagation.

3 Experiments↩︎

3.1 Experimental Setup↩︎

To localize the functional regions corresponding to linguistic competence within LLMs and analyze their nature, we perform language further pre-training (next token prediction) on various languages and observe the relationship between internal parameter removal and external output quality. We utilize LLaMA-2-7B/13B [3] as our model instance, as it stands out as one of the most notable state-of-the-art open-source LLMs in current academia.

Our experimental dataset comprises materials from Chinese platforms like Zhihu and Wechat, English sources from Arxiv and Falcon, and a corpus including books from 28 languages, totaling 30 languages in all. Six languages, namely Arabic, Spanish, Russian, Chinese, Korean, and Vietnamese, are chosen for language further pre-training and region localization, with \(100,000\) samples for each (distinct from the samples in the test set). All 30 languages are employed for model testing and functional region analysis, with the specific languages and token count detailed in 6. We use perplexity (PPL) as the criterion for evaluating the linguistic competence of a language model.

3.2 Core Linguistic Competence Region↩︎

In this section, we conduct further pre-training experiments on LLaMA-2 across six languages, aiming to explore and identify the core linguistic region in LLMs. Here, we define the region as "core linguistic", attributed to its minimal proportion within LLMs, constituting only 1% of the total parameters, and its association with model’s linguistic competence modeling.

Specifically, according to Equation 2 , we cumulatively compute \(\mathcal{I}^*(\theta)=\Sigma\mathcal{I}(\theta)\) values across six different languages’ training, positing that the set of parameters exhibiting maximal importance score \(\mathcal{I}^*(\theta)\) during the language further pre-training may have a strong correlation with the model’s linguistic competence. We provide both logical and empirical evidence to support this hypothesis.

3.2.0.1 Logical Evidence

The phenomenon of code-switching suggests that the LLMs can align languages and may possess core linguistic regions. As discussed in Section 2.2, if a parameter \(\theta_j\) is crucial for the LLMs’ core linguistic competence, the model should be sensitive to \(\theta_j\), shown by a significant increase on the loss \(\mathcal{L}\) when \(\theta_j\) is removed, severely impairing the LLMs’ linguistic performance. Conversely, other parameters impact rarely on core linguistic capabilities.

3.2.0.2 Empirical Evidence 1

Table 1 illustrates that even a 3% removal on the ‘Top’ region leads to a substantial increase in perplexity (PPL), reaching over \(40,000\) across 11 languages, indicating a complete loss of linguistic competence. In contrast, removing the ‘Bottom’ region is comparable to non-removal ‘Base’ in model PPL, and a ‘Random’ removal of equal magnitude has no significant impact on the model’s linguistic competence. Moreover, refer to Appendix 7, additional experiments with reducing training samples to \(10,000\) or adjusted the region selection ratio to 1% and 5% yield consistent findings: removing the ‘Top’ region deprives LLaMA-2 of its capability across all 30 languages. This suggests the model’s linguistic competence is directly influenced by the ‘Top’ region, while removing the ‘Bottom’ and ‘Random’ region don’t have a significant direct impact on language capabilities. See Appendix 7 for evaluations on 30 languages and further experiments.

3.2.0.3 Empirical Evidence 2

In the experiment corresponding to Table ¿tbl:tab:zero95and95freeze951percent?, we initially zero out various regions within LLaMA. Consistent with the findings from Table 1, removing the ‘Top’ region leads to a loss of linguistic competence, whereas the ‘Bottom’ region don’t. However, in this experiment, we sought to ascertain if LLaMA could reacquire its lost cross-lingual generalization competence. Thus, we train on different amounts of Chinese Zhihu corpus and evaluate on Chinese Wechat and English Falcon corpora. The results indicate that unlike the ‘Bottom’ region, if the ‘Top’ region is removed and frozen, the model have to relearn basic language rules in other regions based on the provided Chinese Zhihu corpus, but these rules are inherently biased towards Chinese. Consequently, while its proficiency in Chinese is restored, the English perplexity remains high (\(5301.527\)). If the ‘Top’ region is removed but not frozen, the model can rebuild its linguistic competence in-place. As its proficiency in Chinese is restored, so is its proficiency in English.

This implies that the ‘Top’ region encodes multilingual linguistic competence. When ‘Top’ region is zeroed-out and frozen, other regions difficultly adapt to regain the core linguistic competence.

3.3 Dimensional Dependence↩︎

To provide a more intuitive revelation of the spatial distribution characteristics of the linguistic competence region within the model, we visualize the ‘Top’ region. As shown in Figure 2, whether in the attention mechanism layer or the feed-forward layer, the linguistic region displays a distinct concentration in both the rows and columns of the matrices. In Appendix 7.0.0.1, we also discover that in various layers, the core linguistic region is concentrated on different heads of the Attn.o matrix. Such distribution features seem to imply that the model’s linguistic competence is localized in specific rows and columns.

3.3.0.1 Structured Removal

Instead of discretely removing different unstructured parameters, we selectively remove structured certain rows or columns for each matrix, especially those dimensions encompassing a significant number of ‘Top’ region parameters, termed as ‘Top’ dimensions. As illustrated in Table ¿tbl:tab:att95ffn?, we attempt to remove the columns of FFN.down and Attn.k/q/v, as well as the rows of Attn.o. The results indicate that removing just these ‘Top’ dimensions leads to a substantial decline in the model’s linguistic competence. However, removals to the ‘Middle’ ,‘Bottom’ and ‘Random’ dimensions do not yield noticeable effects. Selecting the dimensional region only from the Attention matrix or inverting rows and columns removals lead to similar findings, as described in 8.

3.3.0.2 Single Dimension Perturbation

Here, we explore whether a specific dimension significantly impacts the model’s linguistic competence. As illustrated in Figure 3, we iterate through the key dimensions mentioned in Section 3.3.0.1, attempting to perturb (random initialization) the same dimension across all Transformer layers. The results indicate that the impact of dimensions 2100 and 4743 on the LLaMA-2-13B substantially surpasses other dimensions, even when compared to the other three in the ‘Top 5’ dimensions. In contrast, perturbing two randomly selected dimensions, 2800 and 4200, yields linguistic performance almost indistinguishable from the unperturbed state.

3.3.0.3 Single Parameter Perturbation

We discover that even a slight modification to a single parameter in a model with over 13 billion parameters can lead to a significant decline in its output quality. In Table 3, merely resetting the \(2100\)-th parameter in the ‘Input_LayerNorm’ module of the \(1\)-st layer to its initial value causes LLaMA-2-13B’s PPL value to skyrocket from 5.865 to 83224.078. If this weight parameter is multiplied by \(10\), the PPL value also rises to \(4363.462\). However, randomly altering the parameters at dimensions 2800 and 4200 doesn’t noticeably impact the model. For more details, refer to Appendix 9.

3.3.0.4 Ablation Study

Considering that the collapse of PPL may be possibly caused solely by the removal of outlier dimensions, rather than the collective effect of the entire linguistic region, we conduct an ablation experiment without removing the parameters of outlier dimensions (1512/2533 for LLaMA-2-7B and 2100/4743 for LLaMA-2-13B). The results for the 13B model, presented in Table ¿tbl:tab:ablation95outlier95dimensions95on95LLaMA-2-13B?, indicate that while outlier dimensions are contained within the core linguistic region, non-outlier dimensions in this region are also critical. Not altering any row or column of the matrix parameters for the two outlier dimensions also results in a great increase in PPL, although the collapse of PPL is slowed. This finding reveals that all dimensions in the core linguistic region are tightly interrelated, and disrupting even a small part of it can lead to a PPL collapse. For a more detailed analysis of the experiments and results for the 7B model, please refer to Appendix 10.

3.3.0.5 Output Under Perturbation

To illustrate the impact of the linguistic competence region on the model’s output quality, we use "Fudan University is located in" as a prompt input and observe the model’s outputs under different parameter perturbations. The results are shown in Figure 4. Compared to randomly selected 4200-th dimension, perturbing model on 2100-th dimension significantly leads to model loses its linguistic competence, producing error or even nonsensical strings.

3.4 Monolingual Region↩︎

In this section, we wonder if LLMs possess distinct regions within different individual languages. Unlike the core linguistic regions described in Section 3.2, a monolingual region only has a strong correlation with certain languages, and removing it will only cause significant influence on LLMs’ proficiency in those corresponding languages.

3.4.0.1 Region Localization

Different from Section 3.2, we initially identify and select the 1% ‘Top’ and ‘Bottom’ regions for each of the six languages (Arabic, Spanish, Russian, Chinese, Korean, and Vietnamese) according to Equation 2 , then deduplicate these regions. For the target language region, we exclude any regions that overlap with the ‘Top’ and ‘Bottom’ regions of the other five languages, aiming to eliminate the core regions and critical dimension corresponding to the model’s fundamental linguistic competence. We denote \(L\), \(\mathcal{S}\) and \(\mathcal{S}^*\) as the total set of six languages and the ‘Top/Bottom’ regions before and after deduplication, respectively. Language \(l\)’s own region \(\mathcal{S}_l^*\) is computed as follows: \[\mathcal{S}_l^* = \mathcal{S}_l \,\, - \bigcup_{l^{'} \in L\setminus\{l\}}\mathcal{S}_{l^{'}}.\] In Appendix 11.0.0.1, we visualize the distribution of Attn.q matrix in ‘Arabic’ and ‘Vietnamese’ regions and discover minimal overlap between them.

3.4.0.2 Region Removal

Unlike removing core regions or dimensions in Section 3.2 and 3.3, we discover that removing monolingual regions will only significantly affect the ability of the target languages and their closely related languages with similar letter elements or sentence structure. For example, if we remove only the region \(\mathcal{S}_{Russian}^*\) for Russian alone, selected from \(10,000\) or \(100,000\) samples respectively, as shown in Table 2, only Russian itself and Ukrainian have significant increases in PPL when removing ‘Top’ \(\mathcal{S}_{Russian}^*\) region. We speculate this to the fact that Russian and Ukrainian are relatively similar in terms of sentence structure and constituents, both belonging to the Slavic group. A similar phenomenon is observed if removals are changed to the regions for each of other five languages, see Appendix 11 for more details.

3.4.0.3 Downstream Task

We conduct downstream tasks on MMLU [29] and ArabicMMLU [30]. The former is in English while the latter is in Arabic. Our experiments demonstrate that removing monolingual regions significantly impacts the model’s ability to perform downstream tasks in the targeted language. Specifically, our findings are as follows:

1) The average probability for option ‘A’ on the ArabicMMLU evaluation increases from 0.36 to 0.64 if remove the ‘Arabic’ region. 2) As illustrated in Figure 5, the model’s accuracy in the ArabicMMLU (filtered) evaluation, where the correct answer is one of the options ‘B/C/D/E’, drops significantly from 25.6% to merely 1.5% when the ‘Arabic’ region is excluded. Conversely, the removal of the ‘Vietnamese’ region has a negligible impact on accuracy, which remains at 26.7%. 3) For the MMLU evaluation, the model’s accuracy is minimally impacted by the removal of monolingual regions. Compared to a baseline accuracy of 42.46%, the removals of the ‘Arabic’ and ‘Vietnamese’ regions result in accuracies of 39.27% and 39.68%, respectively.

Furthermore, using "There are 365 days in a year and 12" as a prompt for generation, we test outputs on the removal of the ‘Arabic’ and ‘Vietnamese’ regions. More details are provided in Appendix 11.0.0.3.

3.5 Further Pre-training Optimization↩︎

In this section, we demonstrate that stabilizing the core linguistic regions (identified in Section 3.2) during further pre-training mitigates the catastrophic forgetting (CF) issue [19], [20] in LLMs, while maintaining learning proficiency comparable to full-scale fine-tuning in target language. Our experimental setup involves further pre-training LLaMA-2-7B on 100,000 Arabic sentences, with a batch size of \(256\), a maximum token length of \(512\), and learning rates (\(lr\)) of \(5e{-5}\) or \(5e{-6}\), employing perplexity (PPL) as evaluation criterion.

3.5.0.1 Full-scale Model Fine-tuning

Traditional full-scale fine-tuning, when increasing the learning rate or the amount of corpus data, enhances learning in the target language but aggravates forgetting in non-target languages. As shown in Table ¿tbl:tab:zero95and95freeze951percent?, since LLaMA-2 is primarily trained on English corpora, conducting a second stage of pre-training solely on large-scale Chinese corpora can lead to the forgetting of English competence. Additionally, as depicted on the left side of Figure 6 in blue line, increasing \(lr\) from \(5e{-6}\) (dotted line) to \(5e{-5}\) (solid line) under full-scale fine-tuning boosts the acquisition of the target language (Arabic), while simultaneously accelerates the forgetting rate of the non-target languages (English and Chinese), as shown in the middle and right side.

3.5.0.2 Freeze Core Regions Fine-tuning

We hypothesize that CF problem occurs due to the amplification of parameter adjustments when increasing the learning rate, which leads to significant shifts in the core linguistic region, adversely affecting language alignment. To mitigate this, we protect the core linguistic region and key dimensions by freezing the ‘Top 5%’ core language area for fine-tuning, as shown by the red line in Figure 6. At a \(lr\) of \(5e{-6}\) (dotted line), the difference between freezing fine-tuning and full-scale fine-tuning is minimal. However, when the \(lr\) increases to \(5e{-5}\) (solid line), freezing fine-tuning not only similarly facilitates faster learning in the target language (preserving comparable performance in Arabic PPL: 3.557 vs. 3.566), but also significantly reduces the forgetting of non-target languages (showing improvements in English and Chinese PPL: 18.796 vs. 20.557 and 90.84 vs. 563.423, respectively).

3.5.0.3

The potential reason for this phenomenon may lie in the preservation of the core regions within the cross-lingual alignment competence. Restricting the magnitude of updates in the core region’s parameters is a future strategy that we intend to employ. Notably, unlike regularization methods [23], [24], such approaches restricts to a minimal core region in LLMs, and can be implemented alongside blending previous data, retraining the entire network, or possibly only the final layers, without adding additional components to the model.

4 Related Work↩︎

4.0.0.1 Intrinsic Regions

Prior works aimed to extract a sub-network capable of executing specific downstream tasks [27], [31] or task-specific subspaces to limit parameter fine-tuning within it [32], [33]. For parameter importance estimation, an effective metric is to use parameter magnitude [34]–[36]. Another metric involves estimating the sensitivity of parameters [25], [26], [28], [37]. In this work, we employ the latter method to select the most crucial parameters to unveil linguistic regions. Unlike [31], which extracted a complete sub-network for downstream tasks, as exemplified by the optimal scale of the lottery network at 21.2%, our findings indicate that the linguistic region is a much smaller (1%) functional region.

4.0.0.2 Cross-lingual Transfer

Multilingual language models exhibit significant zero-shot and few-shot cross-lingual transferability across diverse tasks [6], [8]. Fine-tuned on one language enables model to obtain comparable capabilities in another language [9], [10], often displaying code-switching behavior in context generation [11], [12]. While enhancements in cross-lingual generalization through parameter and information transfer learning [13], [14], compulsory language alignment [15], [16] and in-context learning techniques [17], [18] have been effective, a comprehensive understanding of the internal mechanisms enabling cross-linguistic alignment in LLMs is still lacking.

4.0.0.3 Linguistic Abilities Probing

Prior works have shown that multilingual LMs rely on a shared subword vocabulary and joint pre-training across multiple languages [6], [7], [38]. However, new insights highlight these models’ capacity for learning universal semantic abstractions [39], [40] and demonstrate that embeddings of similar words in similar sentences across languages are approximately aligned already [41]–[43]. Analysis from a hierarchical perspective reveals that classifiers linked to different BERT [44] layers assess semantic features through varied probe tasks [45], [46]. In this work, we introduce a parameter partitioning perspective within LLMs, identifying core linguistic and monolingual regions, which underpin cross-lingual alignment and language-specific characteristics, respectively.

4.0.0.4 Outlier Dimensions

Multiple studies have identified outlier dimensions in Transformer-based LMs [47], [48]. Researches have found that certain outlier dimensions in pre-trained LMs are highly sensitive to the fine-tuning of downstream tasks [47], [49]. Furthermore, [49], [50] discovered that these outlier dimensions encode task-specific knowledge, and disabling these dimensions significantly degrades model performance. [51] attributed positional embeddings to the emergence of outliers, while [52] reported inconsistent results. Additionally, [53] found that these outlier dimensions exhibit significantly larger activation values than others in LLMs. Our findings demonstrate that beyond the outlier dimensions, other non-outlier dimensions within the core linguistic region also play an indispensable role in the model’s core linguistic competence.

5 Conclusion↩︎

This paper explores the pivotal role of certain parameters in Large Language Models (LLMs), identifying a core region essential for multilingual alignment and generalization. Removing this region causes a complete loss of linguistic competence in LLMs. Furthermore, we discover that this core region is concentrated in specific dimensions, perturbing only one dimension can cause a significant decrease in linguistic ability. Moreover, beyond the core linguistic regions, we observe that monolingual regions exist within LLMs that affect specific languages. Importantly, we note that the catastrophic forgetting phenomenon during further pre-training may be related to drastic changes in core linguistic regions, as freezing this part during further pre-training alleviates the issue substantially. Our analysis and findings provide new perspectives and explanations for LLMs’ linguistic competence.

Limitations↩︎

In this paper, while we discover the core linguistic region and distinct monolingual regions within Large Language Models (LLMs), our work presents two notable limitations. First, our experiments are based on LLaMA-2-7B/13B, and it remains to be further determined whether the same phenomenon are observable in larger or differently architected models. Despite this, our focus on LLaMA-2-7B/13B reveals the existence of linguistic regions within the model, providing an explanation for the model’s linguistic capabilities. Secondly, we optimize full-scale fine-tuning through the freezing operation, which is not suited to extensive datasets. A more feasible approach is to limit the magnitude of parameter updates, which is the direction of our future experiments. Nevertheless, it is important to emphasize that slowing down forgetting through freezing core region suggests that in further pre-training, the core region is different from the other regions. Range of variation amplitude in core region should be smaller to maintain the cross-lingual generalization capabilities of the model. Additionally, while our study focuses on linguistic regions, beyond language, knowledge is a higher-level semantic representation, which is a critical direction for us to explore in the future.

Acknowledgements↩︎

The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by National Natural Science Foundation of China (No.62076069,62206057), Shanghai Rising-Star Program (23QA1400200), and Natural Science Foundation of Shanghai (23ZR1403500).

6 Languages in Evaluation Corpus↩︎

We use evaluation data composed of 30 languages to assess the model’s linguistic competence. The 30 languages and their respective token counts (use LLaMA-2 Tokenizer) are as follows: Arabic (4702998), Chinese (2869208), Czech (1362041), Danish (36467), Dutch (3991305), English (1216599), Finnish (372303), French (6755281), German (2884921), Greek (474622), Hungarian (1229433), Indonesian (19226), Italian (6332560), Japanese (501899), Korean (2730794), Malay (5842), Malayalam (1489244), Norwegian (42289), Persian (1736589), Polish (4948702), Portuguese (7598161), Romanian (1381598), Russian (5205716), Spanish (7163860), Swahili (630), Swedish (1450236), Tamil (2920808), Turkish (2484186), Ukrainian (455720), Vietnamese (3606202).

7 Core Linguistic Region↩︎

The regions are localized from six languages: Arabic, Spanish, Russian, Chinese, Korean, and Vietnamese, respectively. Our work does not alter the embedding layer, as we think it equates to a mapping of tokens, which does not involve modeling linguistic competence.

7.0.0.1 Region Visualization

In Figure 9, we present the distribution of the ‘Top’ 5% regions in the Attn.o matrix for the LLaMA-2-13B model. The results indicate that across various layers, the core linguistic region on Attn.o matrix is concentrated on different rows. This difference is observed among the 40 various attention heads.

7.0.0.2 Removal 3% ratio (100K)

LLaMA-2 perplexity on 30 languages when the removal ratio is 3% ratio, with \(100,000\) samples for each language. Refer to Table 9 for more details.

7.0.0.3 Removal 3% ratio (10K)

LLaMA-2 perplexity on 30 languages when the removal ratio is 3% ratio, with reduced \(10,000\) samples for each language. Refer to Table 10 for more details.

7.0.0.4 Removal 1% and 5% ratio (100K)

LLaMA-2-7B perplexity on 30 languages when the removal ratio is changed to 1% and 5% ratio, with \(100,000\) equivalent samples for each language. Refer to Table 11 for more details.

8 Attention Dimensional Removal↩︎

Figure 7 (left) illustrates that the columns of the Attn.k/q/v matrices in the attention layer, as well as the rows of the Attn.o matrix, correspond to different attention head parameters. Conversely, the rows of the Attn.k/q/v matrices and the columns of the Attn.o matrix are closely associated with dimensional features in the representation space.

We remove the ‘Top’ dimensions in the attention layer, and the results is displayed in Tables ¿tbl:tab:att? and ¿tbl:tab:att95reverse?. Table ¿tbl:tab:att? reveals that removing the Attention layers’ ‘Top’ dimensions continues to produce more detrimental effects than other dimensions. The visualizations in Figure 2 show that these dimensions are largely concentrated in a few attention heads, suggesting that some attention heads contribute more significantly to the model’s linguistic competence. Table ¿tbl:tab:att95reverse? indicates that the removals under the second setting cause more damage than the first. Considering that, in the second setting, the ‘Top’ dimensions in the matrix directly interact with the corresponding dimensional features in the representational space, we can conjecture that these features are tightly linked with the model’s linguistic competence.

9 Single Parameter Perturbation↩︎

In a Transformer block, each column in the Attn.o and the MLP.down matrix of the FFN layer can be considered as the input weights of a neuron. Thus, perturbing a column can be seen as disturbing the input weights of a neuron. Viewed from another angle, if we disturb the output activation value of this neuron, a similar effect should be observed. Within LLaMA, there is a specific module called RMSNorm, where each dimension is associated with a weight. Perturbations to these weights can be regarded as disturbances to the output activation values of the corresponding neurons. In Figure 7 (right), we visually demonstrate how RMSNorm affects a column of the Attn.o and the FFN.down matrix.

10 Ablation Study↩︎

Tables ¿tbl:tab:ablation95outlier95dimensions95on95LLaMA-2-7B? illustrate the perplexity of LLaMA-2-7B after removing core regions with and without outlier dimensions, respectively.

10.0.0.1

The ablation experiments reveal that different methods of disruption and varying model sizes exhibit different rates of PPL collapse:

1) Removing according to Attention.Head (attn.k/q/v.col + attn.o.row) results in a slower collapse than according to Dimensional Features (attn.k/q/v.row + attn.o.col). 2) The 13B model shows a slower rate of collapse. 3) The abnormal dimension is mainly concentrated in the FFN layer of the “core linguistic region”. If preserving outlier dimension, the speed of PPL collapse by removing FFN layers decreases most obviously, while Attention.Head is almost unaffected.

11 Monolingual Region↩︎

11.0.0.1 Region Visualization

In Figure 10, we present the distribution of the Attn.q matrix for ‘Arabic’ and ‘Vietnamese’ in 4 different layers. The results reveal that across various layers, the two monolingual regions are concentrated in different columns of the matrix.

11.0.0.2 Region Removal

Tables 4-8 demonstrate LLaMA-2-7B perplexity after removing Arabic, Spanish, Chinese, Korean, and Vietnamese regions, respectively. The region is obtained by removing the intersections with other languages’ respective regions from the 1% ‘Top/Bottom’ regions, selected from 10,000 or 100,000 sentences during further pre-training according to Equation 2 .

11.0.0.3 Case Study

In Figure 8, we use the prompt "There are 365 days in a year and 12" to test the model’s output in English, Arabic, and Chinese, respectively. The results indicate that removing the monolingual regions causes the model to lose the relative language competence, leading the model to generate repetitive, nonsensical responses rather than correct answers like "12 months in a year".

References↩︎