1 Introduction↩︎

Recently, there has been a growing tendency to employ small language models (SLMs) as encoders and embedders. Such small models are often sufficient, since representation learning tasks require capturing semantic similarities rather than performing complex reasoning or text/code generation. Compared to large generative models, SLMs offer significant advantages in terms of computational efficiency, latency, and deployment cost. Furthermore, they are able to produce high-quality text representations when properly trained for domain-specific data. In addition, the use of SLMs for encoders and embedders provides important privacy advantages. Their reduced computational and memory requirements make it feasible to deploy them locally on edge devices, minimizing the need to transmit sensitive data to external servers. This local processing capability reduces exposure risks and enables privacy-preserving applications in many domains. Thus, small models not only offer efficiency benefits, but also play a critical role in building secure and trustworthy semantic systems.

Equally important is the role of semantic chunking in representation-based systems. Rather than dividing text into fixed-length segments, semantic chunking partitions content according to its meaning, discourse structure, or topical coherence. This approach improves embedding quality by ensuring that each chunk represents a self-contained semantic unit. As a result, semantic chunking enhances retrieval accuracy and downstream performance in tasks such as search, retrieval-augmented generation (RAG), and document understanding.

Our goal is twofold — use small language models, and create semantic chunking. We demonstrate that SLMs can effectively segment text, with a specific focus on the relatively underexplored task of code chunking. We show that, despite the challenges of this specific domain, small models can produce high-quality and semantically meaningful code segmentation.

As the use case, we chose C-family languages. Large codebases written in C, C++ and C# remain difficult to process using language-model-based systems due to their complex syntax, heavy use of macros, and weak structural consistency. Existing chunking strategies, such as fixed-size token windows, heuristic splitting, or syntax-based tools such as Tree-sitter, often fail to align with actual semantic units of code. These alignment problems may lead to poor retrieval performance, and to inefficient context construction for downstream LLM tasks. In this work, we introduce SemChunk-C - a family of lightweight language models capable of performing semantic chunking of C-family files (.c, .cpp, .cs, .h, .m/.mm). The models a) identify meaningful code chunks (e.g. cohesive functions, interface blocks, data structure definitions and other components), and b) assign each chunk a descriptive attribute that captures its functional role. Despite the modest model sizes (17M - 150M parameters), they demonstrate strong semantic understanding and robustness to real-world complexities, such as nested declarations, macros, and template-heavy C and C++ code. We evaluate our approach against rule-based chunker and also the use of larger language models. Across a curated benchmark of C/C++ source files, our method achieves higher chunking accuracy and produces more coherent and functionally homogeneous chunks. Also, our approach yields substantial improvement in downstream tasks, such as code retrieval and generation. One of the key advantages of our token classification approach is that it helps avoid hallucinations, which is a key limitation of purely LLM-based chunking. Large models may introduce spurious content, such as additional code, extraneous text, or unwanted attributes, which we have observed in practice.

Our results suggest that small domain-specialized LMs can outperform significantly larger models on targeted semantic segmentation tasks, offering a practical and efficient alternative for code indexing, repository summarization, and intelligent developer toolkit.

Although the chunking process for large existing repositories may appear time-consuming, the advantages of employing semantic chunks for repository-based downstream tasks are substantial. Therefore, incorporating semantic chunking into LLM-based code generation workflows can be highly beneficial.

The remainder of this paper is organized as follows: we first review related work, then present our method, and finally describe the experimental setup and results.

2 Related Work↩︎

The efficacy of modern code-intelligence tasks, ranging from RAG and automated code completion to repository-level understanding, is fundamentally governed by how source code is partitioned. As context windows remain a finite resource, the transition from heuristic-based splitting to semantic chunking has become an important software engineering research.

2.1 Foundation of Semantic Chunking↩︎

The challenge of dividing a continuous stream of text into meaningful segments has its roots in classical natural language processing. Early foundational work, such as the TextTiling algorithm proposed in [2], utilized lexical cohesion and multi-paragraph patterns to identify subtopic boundaries. As the field shifted toward vector space models, research transitioned to using semantic word embeddings to detect shifts in coherence [3], while subsequent supervised learning approaches [4] laid the groundwork for modern "semantic splitting."

In the current era of LLMs, the focus has moved toward identifying the optimal "retrieval unit." Recent work by[5] introduces the concept of Propositional Chunking, arguing that retrieving atomic "propositions"—sentences distilled into independent factual units—significantly outperforms traditional paragraph-based retrieval. This evolution is driven by the realization that chunking strategy is a primary "failure point" in production-level RAG systems [6]. By calculating embedding similarity across these units, modern systems can establish boundaries where the thematic divergence exceeds a statistical threshold, such as an interquartile range or standard deviation shift, ensuring that each chunk captures a complete, independent idea.

2.2 Code-Specific Structural Challenges↩︎

Unlike natural language, source code is best represented as a structured semantic graph rather than a linear narrative. Effective chunking must take into account programming constructs such as functions, classes, and variable scopes, in order to avoid breaking logical dependencies. The foundational work in this area established that capturing intent requires moving beyond raw text to identifier-based embeddings [7] and deep semantic parsing [8].

Recent literature has focused on bridging the gap between syntax and semantics through Abstract Syntax Trees (ASTs). For example, CodeRAG [9] and CodeWisp [10] utilize AST-guided structures to provide highly relevant context for repository-level completion. Similarly, cAST [11] uses recursive parsing methods to merge semantically linked blocks while respecting strict token limits. However, a significant limitation persists: these syntax-reliant methods depend on strict syntactic validity. In C-family languages, the presence of complex macro expansions often causes standard parsers to produce fragmented or failed trees, leaving "functional atoms" obscured.

2.3 Model-Driven Segmentation and Efficiency↩︎

To overcome the fragility of ASTs, a new wave of "model-first" chunking has emerged. Frameworks like CODE2JSON [12] and RANGER [13] utilize zero-shot LLM agents to extract natural language representations and knowledge graphs directly from code. The value of this semantic approach is evident in downstream tasks; FuncVul [14] demonstrates that precise function-level chunking is critical for vulnerability detection, while LongCodeZip [15] employs a two-stage strategy to compress and chunk long-context repositories.

Furthermore, empirical evidence suggests that models may now surpass human performance in this domain. Recent investigations by [16] found that LLM-generated partitions yield summary comments that are up to 20% more factual than human-created boundaries. However, this superior performance comes at a prohibitive computational cost. Relying on massive LLMs for initial indexing of large-scale C-family repositories is not feasible for real-world deployment.

Our work addresses this gap by introducing a family of lightweight chunking models based on Ettin encoders [1] (17M to 150M parameters) that identify flexible semantic boundaries within irregular patterns of C-family code, achieving LLM-level precision with the throughput required for scalable software engineering.

3 Data, Models and Training↩︎

We investigate the potential use of small language models for code understanding related tasks. We focus on a subset of the Ettin family of encoder models with 17M, 32M, 68M, and 150M parameters [1]. These models share an identical architecture, which allows them to be fine-tuned using the same training recipe. This, in turn, allows us to investigate in a systematic way a smallest model which is still capable of good chunking performance.

First, we fine-tuned the models in an unsupervised manner on publicly available repositories covering multiple programming languages (PLs). Then, using a Qwen2.5-Coder-32B-Instruct LLM we automatically generated a training dataset consisting of (chunk, attribute) pairs specific to C-based languages. Finally, fine-tuned Ettin encoder models of various sizes were trained for token classification to segment files into chunks, and to assign an attribute to each chunk. More details are provided below.

3.1 More Code Knowledge Means Better Performance↩︎

As the base model for our chunker, we train a small encoder model with strong capabilities in code syntax. Although the selected Ettin models were already trained on more than 20B tokens of code data, we further enhanced them with additional code, placing particular emphasis on C-related programming languages.

We downloaded nearly 27B tokens from multiple repositories, containing more than 90 PLs, prioritizing C-related languages. Other PLs were used as regularization and augmentation data to enhance robustness and generalization. Figure 1 illustrates the distribution of the downloaded PLs.

Training hyperparameters were selected as in Ettin recipe, with masking ratio of 15%. To partially mitigate the effects of catastrophic forgetting, during the training we included general, non-code data which is a part of C4 dataset [17] containing nearly 5B tokens.

To evaluate the model’s learned ability to better understand C code structure, we compared the base (BS) encoders with the fine-tuned (FT) ones on an internal masked-completion dataset. This dataset was constructed from 100 C language source files in which the function types were masked. Table ¿tbl:tab:function-type-masking? presents a comparison of the base and fine-tuned models in terms of top-1, top-5 and top-25 masked-completion accuracy. As shown, the completion accuracy for the code-specific entity (function type) improves across all the models after fine-tuning. The greatest gain is observed for the smallest model (17M), which achieves a 17.35% increase in the top-1 accuracy.

After our encoders are trained to gain a better understanding of various C-code syntax, we transform them into chunker models. In the following sections, we describe the chunker training procedure and the test sets, and further demonstrate the benefits of the fine-tuning stage for the trained chunker model (see Sec.3.5).

3.2 Taxonomy of Semantic Chunk Types↩︎

Our chunking scheme follows a simple principle: each chunk corresponds to a self-contained, semantically meaningful unit of code. To ensure that our chunker operates at the semantic level rather than as a purely rule-based text splitter, we incorporate additional constraints that require analysis and understanding of the code. These include, for example, handling nested structures and grouping semantically related elements into coherent units.

When building a cross-language repo analyzer it is useful to create a clear structure of semantic chunk types. Below we roughly split semantic chunks into groups, aiming to combine the following types of chunks: C-specific, Object-oriented C++, C#-specific, and common C-family. We also add to each group an attribute (category) that describes its functional purpose. Our goal is to achieve a balance between the granularity of semantic meaning and the number of possible attributes, ensuring that a small model can associate each code snippet with the most appropriate description. We believe that organizing the code into these categories improves the formation of logically and semantically consistent structures. This organization also facilitates repository-level operations, including the systematic addition and removal of code components. Furthermore, it supports the identification of missing or superfluous dependencies. And, finally, it enhances the ability to analyze and answer questions regarding the code’s overall structure and functionality.

The following presents the list of selected chunk categories.

• Category: ‘comment’. Comments may be a single line //.., block comments /*..*/, or doxygen/documentation comment: ///, /**..*/. Comments can be standalone or attached to a declaration/implementation, so they should be chunked differently. We expect standalone comments to form a separate block, while comments that, e.g. belong to a function, should not be separated from the function body. See examples in Fig. 2 A and

  F.

• Category ‘function’ contains function/method declarations, definitions, implementations and modules. A function may be a free function, member or inline function, template, lambda definition, operator overloads, record, orextern for functions. If a function contains sub-functions or compiler directives inside, we assume the chunker still defines it as a single ‘function’ block. See for example Fig. 2 A,D,F.

• Category ‘class’ covers the code related to classes, methods, constructors and destructors.

• Category ‘directive’. This category contains pre-processor and compiler directives: #define, #undef, #if / #ifdef / #ifndef / #else / #elif / #endif, #pragma, #error etc.

• Category ‘include’. An include may be one of the following types: #include, using, import, @import or export.

• Category ‘type definition’ covers structs, unions, constant definitions, enums, declarations, typedefs and alias.

• Category ‘namespace’. Even lengthy code segments that contain heterogeneous categories within namespace scopes should be treated as a single chunk and assigned the category namespace.

• Categories ‘protocol’, ‘implementation’, ‘interface’. These categories are relatively straightforward to detect, as the corresponding code begins with well-defined identifiers and has clearly delineated boundaries.

• Category ‘variable’. If the code defines global or local variables, delegate declarations, or external variable declarations, these are treated accordingly. Variables declared within functions, structs, or other constructs are not separated from their enclosing implementation bodies. Furthermore, certain variables defined outside functional code blocks are associated with the corresponding code segments, as this better reflects their semantic relationships and is expected to improve performance in downstream tasks. See Fig. 2 E for example.

• Category ‘misc’. If a code snippet does not correspond to any of the types, described above, the attribute ‘misc’ is added.

Nested and interleaved structures. Some categories, such as protocols or interfaces, are relatively straightforward to identify, since their boundaries are clearly defined and easily detectable. Others may present greater challenges. Specifically, long comment blocks associated with short functions, or interleaved sequences of directives and function definitions. In the presence of nested structures, we select the largest enclosing entity as the chunk. For example, when one or more functions are wrapped by a compiler directive, the resulting segment is defined as a ‘directive’ chunk (Fig. 2 E). In contrast, if a compiler directive appears within a function, the entire function is treated as a single ‘function’ chunk (Fig. 2 D and F). Many semantically related code elements can and should be grouped into coherent units to simplify subsequent code processing (Fig. 2 B, C and F).

Fine-grained chunking. More fine-grained chunking can be performed iteratively to extract functions from directive blocks, as well as to isolate specific class sections (public, protected or private) and namespace components.

Large files processing. The input sequence length is limited to 8K tokens. Files exceeding this limit are processed iteratively by partitioning them into consecutive 8K-token segments. After processing the first segment, the corresponding portion is removed from the input, and the procedure is repeated on the remaining content until the entire file has been processed.

3.3 Chunker Training Procedure↩︎

We used Qwen2.5-Coder-32B-Instruct model (hereafter referred to as Oracle Chunker) to generate semantic chunks with attribute associated with each chunk. The model was applied to a dataset of 39,371 files (about 73M tokens) using an instruction-based prompt augmented with a small number of illustrative examples (see Appendix 1 for the prompt). Tables ¿tbl:tab:train-data? provide a detailed description of the data generated by Oracle Chunker for model training. To assess the performance of the Oracle Chunker, we first evaluated it on a manually created dataset, called the Manual Test set. We manually annotated several hundred code files with a few extensions. The files were randomly selected from the following repositories [18], [19], [20],[21], [22]. This manually curated dataset, although limited in size, is highly reliable. A separate part of it was also later used as a validation set, to evaluate the accuracy of our trained models.

The Oracle Chunker achieved high average accuracy of 90% for chunk boundaries detection and 96% for category labeling on the Manual Test set. This ensures that the resulting chunks are mostly correct and that the training data is of high quality. The distribution of files and chunk categories of the Manual Test set, as well as the Oracle Chunker results, are presented in Table 1. In all experiments, we used the exact match (EM) evaluation metric.

Training procedure. We found that the following two-stage incremental learning procedure yields the best performance, likely due to the relatively small size of the models.

• In the first stage, each of the four encoder models was trained as a token-level classifier, where each token was assigned to one of three classes: chunk start, chunk end, or chunk interior.

• In the second stage, the classification task was made more challenging by requiring the model to assign interior tokens to one of 12 predefined categories, described in Sec.3.2. Accordingly, the label space was expanded to 14 classes: chunk start, chunk end, and 12 interior categories. After this stage, each token within a chunk interior was classified into one of these 12 categories, and the final category assigned to a chunk was determined based on the classification of its last token.

Training details. We used the AdamW optimizer with a linear learning-rate scheduler and warmup. To mitigate overfitting, dropout and weight decay were applied during training. All models were trained for 5 epochs. In the first stage, the learning rate was set to 5e-5 for the 17M, 68M, and 150M parameter models, and to 8e-5 for the 32M model. In the second stage, a uniform learning rate of 1e-5 was used for all four models.

3.4 Evaluation Methodology and Test Data↩︎

We evaluated the accuracy of chunk boundaries and their associated categories using two test sets: the Manual Test set described above, and a larger Automatic Test set generated by the Oracle Chunker using the same prompt as that employed for training set generation.

Automatic Test set contains nearly 1,000 examples of each of C, C++, C#, H, M/MM files selected mostly from the repositories [18], [19], [20],[21], [22], and processed by Oracle Chunker with corresponding ground truth attributes. We compared the Oracle chunks with those produced by our method. Detailed results for chunking performance and category classification accuracy are presented in Table ¿tbl:tab:automatic-test-data-chunks?. The results demonstrate that, despite the model small size of up to 150M parameters, they are capable of achieving strong semantic code understanding, with performance comparable to that of much larger models. This is in particular evident for the smallest model with only 17M parameters.

We also evaluated a larger language model from the same family as our Oracle chunker, specifically a Qwen-Coder-7B model, using the same prompt configuration as in the Oracle setting. We compared the accuracy of chunk boundary detection and category assignment produced by our lightweight classification models against this substantially larger LLM (see Table ¿tbl:tab:automatic-test-data-chunks?). The results indicate that employing a larger model, even of moderate scale such as 7B parameters, is not always justified. In many cases, a properly trained 17M-parameter model matches or even outperforms the 7B model. Moreover, when averaging results across programming languages, our smaller classifiers demonstrate strong capability in capturing programming language semantics and structural relationships.

Notice that the ability of general purpose SLMs to follow complex instructions in code understanding tasks may degrade substantially as model size decreases. For example, the Qwen-0.5B model (with the same prompt as for its bigger counterparts) exhibits extremely low accuracy on the Automatic test set, achieving on average less than 15% both for chunking and for category classification. We do not report the detailed results for the Qwen 0.5B model due to their limited relevance.

2pt

2pt

3.5 Pretraining Justification and Impact↩︎

To justify encoder pretraining on diverse code data (Section 3.1), we trained an additional family of chunker models built on the original Ettin encoders. We then compared the accuracy of chunkers initialized by the original encoders with those initialized by encoders after the fine-tuning stage described in Section 3.1. Table 2 reports the chunking accuracy (EM) on the Automatic test set for models trained based on the original encoders, along with the corresponding accuracy gains obtained when the chunkers are initialized by the fine-tuned encoders. As shown in the table, incorporating code knowledge through fully unsupervised pretraining improves chunking performance in nearly all cases (with one exception), yielding an average gain of 3-6% across models and PLs. The greatest impact of pretraining is observed for the smallest (17M) model and for C# language.

2pt

4 Experiments and results↩︎

In the previous section, we argued that our chunker models are compact yet accurate. In this section, we evaluate the impact of our accurate chunking on downstream repository-related tasks. We consider two tasks - retrieval and code generation, and evaluate them using publicly available datasets.

4.1 Retrieval Task↩︎

We evaluated the ability of our chunking methods to support repository-level retrieval tasks using the RepoQA benchmark. The authors built a Searching Needle Function (SNF) task that challenges models to locate functions based on their natural-language descriptions.

In our experiments, we selected ten of the fifty repositories included in the benchmark, focusing specifically on programming languages from the C family. Each repository was preprocessed by splitting files into chunks and embedding each chunk using the modern-gte embedder [23]. Each code snippet corresponding to a “needle” function in RepoQA is associated with two types of natural language annotations: a general description and a shorter purpose. The retrieval task consisted of identifying the correct code snippet by finding the embedding vector most similar to the embedding of the description and the purpose, separately. Table3 reports the total number of chunks generated by each method, along with the retrieval performance, measured by the nDCG@10 metric. We compare the performance of our four models of different sizes with the standard Tree-sitter chunker. As shown in the table, our approach achieves comparable or superior retrieval accuracy while producing substantially fewer chunks, thereby significantly accelerating vector database search. Also, restricting retrieval to chunks of specific types (e.g., ‘function’) or excluding certain categories (e.g., ‘directive’ or ‘comment’) improves performance while reducing computational cost.

4.2 Code Generation Task↩︎

YABloCo benchmark [24] targets long-context code generation at the function level, particularly for C and C++ - languages, and context that many previous benchmarks did not cover. The benchmark consists of over 200 tasks, drawn from large repositories: [18], [19], [20], [21]. Models are evaluated by their ability to generate correct function body given extensive surrounding context.

We evaluate the code generation accuracy of Qwen2.5-Coder-32B-Instruct model under five settings: (1) without context, (2) with context provided by the Tree-sitter-based chunking tool, (3) with context generated by an LLM (Qwen2.5-Coder-32B-Instruct), (4) with context produced by our four chunker models, and (5) with the ‘oracle’ context provided by the [24]. In the original YABloCo paper, the ‘oracle’ context was defined by the authors as all the functions in the repository along the docstrings and function bodies that are invoked into the target function body. For the settings (2)-(4), repository files were first segmented into chunks using the corresponding methods. These chunks were then embedded using the gte-modernbert embedder [23], after which the top 10 most relevant chunks were retrieved and supplied as context to the code generation model. We evaluated code generation performance by computing pass@1 and pass@10 metrics across the YABLoCo benchmark repositories (see Table ¿tbl:tab:yabloco-generations?).

As evident from the Table ¿tbl:tab:yabloco-generations?, the context provided by our chunkers substantially improves code generation accuracy compared to the ‘no context’ setting, and in most cases outperforms the context produced by the tree-sitter and the large model. In some instances, it even surpasses performance achieved using the oracle context. Notably, larger chunker models do not yield consistently better results; in several cases, the 17M model provides sufficiently high-quality chunks. This suggests that more efficient, relatively small models are very effective for repository processing in a range of downstream tasks.

4.3 Inference↩︎

The inference times on both GPU and CPU, for a batch size of 1 and two context lengths (512 and 8K tokens), are reported in Table 4. The reported results reflect model inference only and exclude both tokenization and postprocessing. Tokenization process introduces an additional latency of approximately 0.1–0.2 seconds, with substantial variability across iterations. Furthermore, the GPU used in this study does not support FlashAttention, resulting in comparatively slower performance. The analysis of inference times suggests that at least the two smallest models (17M and 32M parameters) are suitable for execution on a CPU, particularly in scenarios involving minor, localized code changes.

5 Conclusion↩︎

This paper presents SemChunk-C - a new lightweight approach to semantic chunking of code written in C-family programming languages. By training the models of various sizes (from 17M to 150M parameters) we investigate the minimal model size capable of capturing semantic nuances in code, and accurately aggregating lines of code based on contextual requirements. In addition, we examine the role of repository-level chunking and demonstrate its importance for improving performance in downstream tasks.

6 Appendix↩︎

Below is a prompt we used for Qwen32B-Coder-Instruct to generate code chunks and their corresponding attributes.

""" Instruct: You are provided with a code snippet. Analyze this code and split it into blocks. Do not generate new text or code, just add delimiters and block attribute. Add delimiter ‘block start’ at the beginning of the block and delimiter ‘block end’ at the end of the block. Blocks may be of the following types.

A comment: if a comment is standalone (like starting comments in the file). If comments precede the function, put the comments and the function into a single block. If it’s a comment, add ‘attribute: comment’ after ‘block end’.

A function: it may be a free, member or inline function, template, constructor or destructor, lambda definition, operator overloads, record, extern for functions. If a function or struct contains sub-functions or compiler directives inside, don’t split it, put into a single block. If it’s a function, add ‘attribute: function’ after ‘block end’.

A class: if a code relates to a class or method. If it’s a class, add ‘attribute: class’ after ‘block end’. A type definition block: if it’s a typedef, union, struct, enum or alias. If it’s a type function, add ‘attribute: type_definition’ after ‘block end’.

A namespace: if it’s a namespace. If it’s a namespace, add ‘attribute: namespace’ after ‘block end’.

An include: if the code is #include, using, import, @import or export. If it’s an include, add ‘attribute: include’ after ‘block end’.

A compiler directive: if the row starts with # (like #ifdef, #error etc), but is not an include. Put everything between #if and corresponding #endif into a single block, even if there are functions, directives or includes inside it. If it’s a directive, add ‘attribute: directive’ after ‘block end’.

An implementation: if it starts from @implementation and ends with @end. In this case add ‘block end’ only after the corresponding @end. If it’s an implementation, add ‘attribute: implementation’ after ‘block end’.

A protocol: if it starts from @protocol and ends with @end. In this case add ‘block end’ only after the corresponding @end. If it’s a protocol, add ‘attribute: protocol’ after ‘block end’.

An interface: if it starts from @interface and ends with @end. In this case add ‘block end’ only after the corresponding @end. If it’s an interface, add ‘attribute: interface’ after ‘block end’.

Variables: if the code defines global or local variables, delegate or extern for variables. If it’s a variable, add ‘attribute: variable’ after ‘block end’.

If a code snippet is not one of the above types, also wrap it by ‘block start’ and ‘block end’, and add ‘attribute: misc’ after ‘block end’. Don’t explain and do not write what is the purpose of the block.

Follow the example below: {}

Code snippet: {} """

References↩︎