1 Introduction↩︎

GitHub hosts tens of millions of public repositories, but the platform provides no native indication of which industry a repository serves. Knowing whether a project is fintech, agritech, healthcare software, or educational tooling matters to policy makers tracking the geography of innovation, to companies measuring open-source adoption, and to economists studying the labor and capital allocation of the software sector. We address this gap by releasing the first multi-region, publicly available corpus of GitHub repositories labeled with NAICS—the industry-classification standard used by the United States, Canadian, and Mexican statistical agencies.

1.0.0.1 Contributions.

• Dataset. We release NAICS-GH, 6,588 GitHub repositories from the USA, EU, and Australia labeled with 2-digit NAICS codes, alongside the full pipeline outputs (sector, score, rationale, repository URL).

• Pipeline. A reproducible two-stage retrieve-and-verify labeling pipeline (BAAI/bge-large-en embeddings + FAISS retrieval, followed by GPT-4.1 rubric scoring) suitable for adapting to other industry taxonomies. The full pipeline narrows \(\sim\)​1.37M source repositories to 31,178 candidate pairs via retrieval, then to 6,588 high-confidence labels via LLM verification at score \(\geq 8\). An end-to-end re-run reproduces the candidate set to within \(\pm 0.03\%\).

• Validation. A 2,421-repository manually re-checked gold subset, confirming 96.98% label precision overall and monotonically increasing precision as the GPT score rises from 8 to 10.

• Benchmark. A head-to-head comparison of six pretrained encoders (RoBERTa, ModernBERT, DeBERTa-v3 in base and large sizes); RoBERTa-large is strongest at 86.45% test F1, and the fine-tuned checkpoint is available on the Hugging Face Hub.

1.0.0.2 Pipeline at a glance.

Figure 1 summarizes the end-to-end retrieve-and-verify pipeline used to construct NAICS-GH from raw public-repository data.

2 Related Work↩︎

2.0.0.1 GitHub repository classification.

GitHub repositories have long lacked standardized industry labels, and existing work has approached this gap from three angles. The first relies on user-declared topic tags as the label space: [1] use a keyword-driven hierarchical model, [2] cast it as a multi-label problem, and [3] rank GitHub topics by relevance with active sampling. The second labels repositories by software application domain: [4] build a 5,000-repository, five-domain classifier achieving \(\sim 70\%\) precision, and [5] scale to 825,000 repositories with ground-truth topics from the Software Heritage archive, reporting F1\(@5 = 60.8\%\) with file-tree and README features. None of these works adopt a standardized economic taxonomy or use a modern LLM verifier. NAICS-GH differs from each on all three axes: (i) we adopt the NAICS 2-digit hierarchy, (ii) we use GPT-4.1 verification on top of dense-retrieval candidates, and (iii) we cover three jurisdictions (USA, EU, AU) rather than a single one.

2.0.0.2 NAICS classification of text.

The United States Census Bureau uses NAICS to label every business establishment, but published machine-learning tools for the taxonomy are scarce. The closest prior work is BEACON [6], a text-classification system deployed in the 2022 U.S. Economic Census that helps respondents self-classify their business activity via NLP, machine learning, and information retrieval; BEACON was used over half a million times during the 2022 Census. Two structural differences distinguish it from our work: BEACON’s unit of analysis is a business establishment described by its owner (we classify GitHub repositories described by community metadata), and BEACON’s algorithm and training data are not publicly released (we release both the labeled corpus and the pipeline code). To our knowledge, NAICS-GH is the first publicly available NAICS-labeled corpus of software repositories.

2.0.0.3 LLM-as-labeler and weak supervision.

Our retrieve-and-verify pipeline sits in the weak-supervision tradition where heuristic or programmatic functions assign noisy labels that are then aggregated and refined [7], [8]. LLMs have recently emerged as a powerful source of such labels: [9] report that ChatGPT exceeds crowd-workers in accuracy and agreement on text-annotation tasks at a fraction of the cost, and [10] develop cost-aware majority voting across multiple LLMs to mitigate individual-model bias. NAICS-GH instantiates this tradition with a specific domain (software repositories), a specific taxonomy (NAICS), and a verification step that uses one high-capability model (GPT-4.1) with a structured rubric and human re-validation on a stratified subsample.

2.0.0.4 LLMs applied to GitHub content.

A growing body of work uses LLMs to classify or extract structured data from the GitHub platform, but with different units of analysis. [11] fine-tune BERT, RoBERTa, and DistilBERT (with LoRA) to classify the sections of a README into eight structural roles (What/Why/How/\(\ldots\)), reaching F1 = 0.98 on the 4,226-section [4]-derived benchmark. [12] use an LLM-driven framework to construct a smart-contract vulnerability dataset at scale, and [13] propose a multi-LLM pipeline that extracts community knowledge from GitHub issue discussions. Our work is complementary to these: we use the README alongside the description and topics to classify the whole repository into a 19-class industry taxonomy, an output space that is broader and more semantically overlapping than the section- or vulnerability-typed analogs above.

2.0.0.5 Datasheets.

We provide a Datasheet for Datasets [14] in Appendix 17.

3 Dataset Construction↩︎

3.1 Source repositories↩︎

We extract source repositories from GitHub’s internal Trino-on-Hive data warehouse via a Presto/Trino SQL query against the hive.canonical.repositories_current, hive.canonical.accounts_current, hive.suez.readme_current, and delta.snapshots.github_collab_commit_contributions tables. The query was executed against data current through April 15, 2025 (USA presample; EU readme extraction August 15, 2025). The full SQL is in Appendix 10.

3.1.0.1 Inclusion criteria.

A repository is included in the source pool if it satisfies all of the following:

• Public, non-fork, non-spammy-owner GitHub repository (is_public = TRUE, is_fork = FALSE, NOT is_spammy_owner).

• At least one star (num_stars\(\geq 1\)).

• Declares a top-level README.md (case-insensitive) of at least 750 bytes.

• Disk usage greater than zero, and non-empty repository description.

• At least 6 total commits and at least 2 distinct commit contributors (HAVING SUM(commit_count) > 5 and COUNT(DISTINCT user_id)\(\geq 2\)).

• Owner-account country code in the per-region list (§3.1.0.2).

3.1.0.2 Jurisdiction split.

Repositories are assigned to a region by the country_account field of the owner account in accounts_current, joined via owner_dotcom_id (not by top-contributor location or geo-IP):

• USA: ’US’ (510,380 repositories).

• EU (27 codes, EU-27 excluding the UK): AT, BE, BG, CY, CZ, DE, DK, EE, ES, FI, FR, GR, HR, HU, IE, IT, LT, LU, LV, MT, NL, PL, PT, RO, SE, SI, SK (530,898 repositories).

• AU: ’AU’ (331,211 repositories).

3.1.0.3 Total source pool.

1,372,489 repositories across the three jurisdictions. Each repository carries its owner-and-name identifier (nwo), description, topic tags, README content, SPDX license identifier, and activity metadata (stars, commits, contributors, issue counts, last-commit timestamp).

3.1.0.4 Per-region processing, country-blind release.

The source SQL emits three jurisdictional extractions, and each subsequent generation stage (embedding, retrieval, LLM verification, score filtering) runs per region with an identical configuration; the three score-filtered outputs are then concatenated into the released corpus (§3.4). No released artifact retains a per-row country_code column or any other per-row jurisdictional attribute: rows are shuffled at assembly so regional provenance is not recoverable from row position, and jurisdiction appears in this paper only as aggregate statistics. To enable external reproduction without access to jurisdiction-tagged data, we additionally publish a jurisdiction-blind variant of the pipeline that operates on the concatenation of the three extractions as a single corpus (§8).

3.2 NAICS sector taxonomy↩︎

We use a pre-built JSON taxonomy file (naics_titles_by_group_6digit_clean.json) with 20 entries, one per 2-digit NAICS sector code. Each entry is a single string of subindustry titles separated by semicolons, sourced from the 6-digit NAICS hierarchy and grouped by their parent 2-digit code. We consume the file as-is; no 6-to-2-digit collapsing is performed by our pipeline.

Across the 20 sectors there are 1,029 distinct subindustry phrases in total. The count per sector varies widely: Manufacturing (31–33) has 346 subindustries, the most by far; Utilities (22), Accommodation/Food (72), and Educational Services (61) have the fewest (14, 15, and 17 respectively). The full taxonomy is in Appendix 11.

The taxonomy contributes to the pipeline in two distinct ways:

• Semantic retrieval (§3.3) uses the individual subindustry phrases as queries, one query per phrase.

• LLM verification (§3.4) uses the full concatenated string for each sector as the rubric anchor presented to GPT-4.1 alongside the candidate README.

3.3 Semantic retrieval↩︎

3.3.0.1 Embedding the source corpus.

For each region, we encode the first 1,000 characters of each repository’s raw README using BAAI/bge-large-en [15], with the asymmetric BGE prefixes: "Represent this document for retrieval: " for documents and "Represent this query for retrieval: " for queries. (Repository description and topic tags enter the pipeline at the verification stage, §3.4, where the LLM receives the full concatenated text.) Embeddings are 1024-dimensional and L2-normalized; inference uses half-precision (FP16) on GPU with a batch size of 1024. We build an exact inner-product FAISS index (IndexFlatIP) [16], which under normalization computes cosine similarity.

3.3.0.2 Per-subindustry query loop.

Retrieval is driven by the NAICS taxonomy loaded in §3.2. For each of the 20 sectors we split the sector’s semicolon-separated string into its constituent subindustry phrases and issue one FAISS query per phrase with the template "Repositories about {subindustry}". The number of nearest neighbors retrieved per query, effective_k, is set adaptively:

\[\mathtt{effective\_k} \;=\; \begin{cases} 20 & \text{if } n > 20 \\ \max\!\left(20,\;\lceil 400 / n \rceil\right) & \text{if } n \leq 20 \end{cases}\]

where \(n\) is the number of subindustry phrases in the current sector. For the 17 sectors with \(n > 20\) this gives a uniform \(k = 20\); for the three sectors with narrow taxonomies the formula boosts \(k\) so that each sector retrieves on the order of 400 candidates regardless of \(n\): effective_k \(= 29\) for Utilities (\(n=14\)), \(27\) for Accommodation/Food (\(n=15\)), and \(24\) for Educational Services (\(n=17\)).

3.3.0.3 Tagging and deduplication.

Each returned row is annotated with the sector code, the specific subindustry phrase that produced the match, and the BGE cosine similarity. The loop across all 1,029 subindustry queries retrieves exactly 20,879 rows per region (\(\sum_s n_s \times k_s\), overlap allowed); deduplicating on (repository, sector) within each region yields 31,178 unique candidate (repository, sector) pairs (10,983 USA + 10,531 EU + 9,664 AU).

3.3.0.4 Why this design.

The retrieval is intentionally over-inclusive — the BGE cosine similarities of the returned candidates concentrate in a narrow \([0.78, 0.89]\) band, so the score alone cannot distinguish true matches from near-matches. The expensive verification step (§3.4) handles that discrimination. Querying per subindustry rather than per sector lets us cast a wider net (e.g., Agriculture is queried 64 times rather than once) and exposes downstream code to the specific subindustry that surfaced each candidate, which is useful for error analysis.

3.4 LLM verification↩︎

We score each candidate pair using GPT-4.1 (model snapshot gpt-4.1-2025-04-14) via the GitHub Copilot LLM-lab endpoint (api-model-lab.githubcopilot.com/chat/completions) with a structured prompt (Appendix 12).

3.4.0.1 What the model sees.

For each candidate (repo, sector) pair, the API call sends two messages:

• A system message identifying the model as “a domain expert in economic classification systems with a focus on NAICS industry {code}.”

• A user message containing the structured prompt. The prompt includes:

  • The repository’s combined text — not the bare README, but the concatenation of the repository’s description, topic tags, and cleaned README, joined with explicit field labels:

    description: {desc}, topics: {topics}, readme: {readme}

    This composite string is what is inserted inside the prompt’s <readme>…</readme> tags. The repository name is not included.

  • Preprocessing: Markdown code blocks, inline code, images, links, HTML tags, and excess whitespace are stripped, and the result is truncated to 3,000 whitespace-separated tokens (i.e., a word-count proxy, not a BPE token count).

  • The candidate NAICS sector code and the full concatenated subindustry definition from the taxonomy (§3.2).

  • A four-criterion rubric (industry-specific software, sector-relevant functionality, industry-domain applications, sector-specific data/research) and a 1–10 scoring guide.

  • Instructions to reply as a JSON object nested under the outer key "NAICS {code}" with string-valued match ("Yes" or "No"), string-valued score ("1"–"10"), and free-form rationale. The downstream pipeline coerces match to a boolean and score to an integer.

3.4.0.2 API parameters.

We call GPT-4.1 with temperature = 0 for near-deterministic outputs, and set max_tokens dynamically as \(\min(3500,\;128000 - \mathrm{input\_tokens})\) based on the remaining context budget (input tokens are counted with tiktoken.encoding_for_model("gpt-4.1"), falling back to cl100k_base). On non-200 API responses we retry up to five times with linear backoff (sleep = 5 \(\times\) attempt seconds). Replies are parsed by extracting the JSON object via regular expression and calling json.loads; a small fallback handles the case when the model wraps its JSON in Markdown fences.

3.4.0.3 Filtering.

We retain repositories whose returned score is at least 8.

3.5 Filtering and final assembly↩︎

We concatenate the three regional score-\(\geq 8\) outputs and apply a minimum-samples filter of 80 repositories per NAICS sector, eliminating Sector 55 (“Management of Companies and Enterprises”), which had only 13 repositories. After deduplication and cleanup we obtain the released NAICS-GH training corpus of 6,588 repositories spanning 19 of the 20 NAICS sectors.

3.5.0.1 Pipeline funnel.

Table 1 summarizes the end-to-end retention from raw source to released training corpus.

Overall, only 0.48% of the source pool ends up labeled in the released corpus, and 21.1% of FAISS-retrieved unique candidates survive GPT-4.1 verification at the score-\(\geq 8\) threshold.

4 Dataset Description↩︎

The released NAICS-GH corpus is a single parquet file with 6,588 rows and 6 columns (train_data_gpt_ab8_score_with_code.parquet, 12 MB on disk). Every cell is non-null. The dataset is published as a single training corpus; downstream code (§6) applies a stratified \(70 / 10 / 20\) split with seed=42 to produce train, validation, and test sets of 4,611, 659, and 1,318 rows respectively.

4.1 Schema↩︎

The columns nwo, match, score, rationale, and repo_url present in upstream intermediate files are intentionally dropped from the release so that the corpus is model-ready (no LLM-provenance, no PII via repo-owner URLs). The mapping between label and code is monotonic in NAICS code order: \(0 \!\to\! 11,\, 1 \!\to\! 21,\, 2 \!\to\! 22,\, 3 \!\to\! 23,\, 4 \!\to\! 31\text{-}33,\, \ldots,\, 18 \!\to\! 92\).

4.2 Size and sector coverage↩︎

The released corpus contains 6,588 repositories spanning 19 of the 20 NAICS sectors. Sector 55, “Management of Companies and Enterprises,” contained only 13 candidates that passed GPT-4.1 verification at score \(\geq 8\) — below our minimum class-size threshold of 80 — and is therefore absent. All 19 labels in \(\{0,\ldots,18\}\) are represented in the file.

4.2.0.1 Sector counts and imbalance.

Per-sector counts are given in Table 3. The distribution is moderately imbalanced: the smallest retained sector is 23 (Construction) with 82 repositories and the largest is 44–45 (Retail Trade) with 641, an imbalance ratio of \(\approx 7.82\times\). The mean per-sector count is 346.7 with standard deviation 150.8; the median is 372. Twelve sectors fall within \(\pm 25\%\) of the mean, and the lower tail (sectors 21 Mining, 23 Construction, 42 Wholesale, 56 Admin/Waste) drives most of the imbalance.

4.3 Text characteristics↩︎

Tables 4 and 5 give character-length and whitespace-token-length statistics, respectively. The README field dominates the input by a wide margin: its median length is 1,912 characters (275 words) and its 99th percentile is roughly 3,500 words — comparable to the 3,000-word truncation limit used by the LLM verification step (§3.4). Only \(\sim\)​1.5% of rows (100 of 6,588) have a combined text length that would have been truncated.

4.3.0.1 README size distribution.

The distribution of README character lengths in the released corpus is roughly log-normal:

About 76% of READMEs are in the 500–5,000-character range; only \(\approx 5\%\) exceed 10,000 characters. The lower bound (READMEs \(<\) 500 characters, \(\approx 5\%\) of the file) reflects the source SQL’s \(\geq 750\)-byte threshold combined with downstream Markdown cleanup (code blocks, HTML, URLs stripped) which removes some bytes before the field is stored.

4.3.0.2 Topics field.

The topics field is empty in 4,533 of 6,588 rows (\(\approx 68.8\%\)). Among the 2,055 rows that have any tags declared, the median is 5 tags per row (mean 6.0, p75 = 7, p99 = 20, max = 20). The field is stored as a semicolon-joined string (e.g., "oscibio; lifewatch; biologging; r-package"). Downstream code should treat empty strings as “no tags declared,” not as missing data, and should not assume the field is consistently populated.

4.4 Known duplicates↩︎

Because the canonical (owner/repo) identifier (nwo) is dropped from the release, some short-name collisions appear in the file:

• 6,141 unique name_repo values across 6,588 rows — i.e., 447 rows share their name_repo with at least one other row.

• Of those, 113 (name_repo, code) pairs are duplicated: two or more distinct repositories sharing the same short name independently passed verification into the same NAICS sector.

These are not accidental duplicates of the same repository — they are distinct repositories with the same short name (e.g., two different organizations both maintaining a repository called api, docs, or awesome). Users joining external metadata by name_repo alone should expect ambiguity; the upstream intermediate files produced in §3 retain nwo as a globally unique identifier for users who need an unambiguous join.

4.5 Score distribution↩︎

5 Validation↩︎

5.1 Gold-set construction↩︎

Research assistants reviewed a random sample of 2,421 repositories drawn from the GPT-4.1-labeled output. For each repository they inspected the GitHub page and judged whether the GPT-assigned NAICS sector was Correct or Incorrect given the criteria of Section 3.4. Each row was reviewed by a single annotator; no inter-annotator agreement coefficient was computed (this is acknowledged as a limitation in §7).

5.2 Headline result↩︎

GPT-4.1 label precision: 96.98% (\(2{,}348\) of \(2{,}421\) gold rows judged correct; 95% Wilson confidence interval \([96.23\%, 97.59\%]\)).

5.3 Precision by sector↩︎

Table 6 reports per-sector precision in the gold sample with 95% Wilson intervals. Six sectors (21 Mining, 48–49 Transportation, 51 Information, 53 Real Estate, 71 Arts & Entertainment, 92 Public Administration) reach 100% precision. Two sectors fall noticeably below the overall precision: 31–33 Manufacturing (73.0%) and 42 Wholesale Trade (73.0%), indicating that the LLM rubric struggles to separate truly sector-specific manufacturing or wholesale software from generic operations and supply-chain tooling.

5.4 Precision rises monotonically with GPT score↩︎

The score returned by GPT-4.1 correlates with human-judged correctness in the expected direction: precision rises from 90.76% at score \(8\) to 97.81% at score \(9\) and 99.30% at score \(10\) (Table 7). This validates the rubric: the 1–10 score carries genuine information about label confidence and is not merely an artifact of the prompt template. Practitioners requiring higher precision than the corpus-level 96.98% can simply raise the score threshold; the released corpus uses the inclusive \(\geq 8\) cut to maximize coverage.

5.5 Stratified error analysis↩︎

The error rate is concentrated in two sectors. In Manufacturing (31–33), repositories at score 8 are correct only \(20\%\) of the time (4 of 20 gold rows), but rise to \(80.5\%\) at score 9 and \(100\%\) at score 10. Wholesale Trade (42) shows the same pattern. Practitioners building on NAICS-GH who require high precision in these two sectors should raise the score threshold to \(\geq 9\). A qualitative analysis of the 73 incorrect labels — which NAICS sector RAs would have assigned instead of the model’s prediction — is in Appendix 16.

5.6 What this validation does not cover↩︎

5.6.0.1 Single-annotator protocol.

Each gold-set row was reviewed by exactly one research assistant, so we cannot report Cohen’s \(\kappa\) or another inter-annotator agreement metric. A future double-labeling pass on \(\sim 200\) rows will let an agreement coefficient accompany the headline precision.

5.6.0.2 Precision, not recall.

The validation tells us what fraction of retained labels are correct, but not what fraction of true sector-\(X\) repositories in the source pool the pipeline missed during retrieval. A recall analysis would require an independently constructed reference list of sector-labeled repositories from a source other than the BGE/FAISS retrieval used here, and is left to future work.

6 Benchmark: Fine-Tuned NAICS Classifiers↩︎

To demonstrate the downstream utility of NAICS-GH, we fine-tune six pretrained encoders on the released 6,588-row training corpus and compare them on a held-out test set. All six runs share identical data, splits, and hyperparameters — only the base model differs.

6.1 Input construction↩︎

Each repository is serialized as

and then passed through a cleaner that strips badges, license headers, Markdown formatting, code blocks (which are replaced by a code-{lang} placeholder), excess punctuation, installation commands (npm install, pip install, git clone), and collapses URLs to their domain. The cleaned string is truncated by the tokenizer to a uniform 512 WordPiece tokens across all six models (we choose this max length for direct comparability, even for the ModernBERT variants that support longer contexts).

6.2 Splits and hyperparameters↩︎

We use a stratified 70 / 10 / 20 train / validation / test split with random_state = 42, yielding sizes 4,611 / 659 / 1,318. All 19 NAICS classes are present in every split. Training uses AdamW (adamw_torch_fused) with learning rate \(1.5\times 10^{-5}\), polynomial schedule with \(15\%\) warmup, weight decay \(0.02\), per-device batch size 8, gradient-accumulation steps 2 (effective batch size 16), gradient clipping at 1.0, BF16 mixed precision, 8 epochs, and early stopping with patience 2 and threshold \(0.001\) on weighted F1. Evaluation and checkpoint saving fire every 100 steps, with the best checkpoint by F1 loaded at the end of training. The reported metrics are weighted F1, accuracy, precision, and recall on the held-out test set.

6.3 Models and results↩︎

Table 8 reports test-set performance for six pretrained encoders spanning three families (RoBERTa [17], ModernBERT [18], DeBERTa-v3 [19]) and two parameter scales (base and large).

6.3.0.1 Findings.

• RoBERTa-large is the strongest baseline at F1 \(=86.45\%\), followed by DeBERTa-v3-base (\(85.68\%\)) and DeBERTa-v3-large (\(85.07\%\)).

• The encoder family matters more than parameter count at this corpus size. DeBERTa-v3-base (183M) outperforms ModernBERT-large (395M), suggesting that the original RoBERTa/DeBERTa pretraining objectives transfer better to short-document classification than ModernBERT’s long-context-oriented pretraining, at least for a corpus on the order of \(5\,000\) training examples.

• Larger variants do not uniformly outperform their base counterparts. ModernBERT-large is worse than ModernBERT-base, and DeBERTa-v3-large is worse than DeBERTa-v3-base. RoBERTa is the only family where scaling monotonically improves performance, which is consistent with a relatively small training corpus (4.6k examples, 19 classes) where larger models risk overfitting.

The fine-tuned RoBERTa-large checkpoint is available at https://huggingface.co/alexanderquispe/naics-github-classifier.

7 Limitations and Ethical Considerations↩︎

7.0.0.1 English-only retrieval.

BGE-large-en is trained on English text. Repositories whose READMEs are in other languages are under-represented in the candidate pool and therefore in the released corpus. We make no claim of coverage for non-English software ecosystems.

7.0.0.2 NAICS is a North American taxonomy.

Applying NAICS to European or Australian repositories assumes that economic activities map cleanly across jurisdictions. Sector definitions sometimes diverge (for example, NAICS Sector 22 “Utilities” is structured around the US regulated-utility model). We note this whenever it materially affects interpretation.

7.0.0.3 Label noise is not uniform.

As Section 5 shows, two sectors (Manufacturing and Wholesale Trade) carry substantially higher label error than the rest at the score-\(\geq 8\) threshold. Downstream users should treat the GPT score as a usable confidence signal (raise to \(\geq 9\) for stricter applications) rather than treat all labels as a uniform gold standard.

7.0.0.4 Repository content licensing.

NAICS-GH releases labels about public repositories, plus excerpts of their READMEs for reproducibility. The underlying repositories remain governed by their own licenses (preserved as spdx_license in the released file).

7.0.0.5 Dual use.

Industry-classified repository data could plausibly be used for competitive intelligence or surveillance of open-source contributors. We release the dataset under CC-BY-4.0 with a preferred-use statement encouraging academic and policy use; we do not believe sector-level industry tagging of public repositories raises additional risks beyond what the underlying public metadata already permits.

8 Release and Reproducibility↩︎

The dataset is released at https://huggingface.co/datasets/aquiro1994/naics-gh under CC-BY-4.0, with stratified train/validation/test splits (\(4{,}611 / 659 / 1{,}318\)) ready for direct use via datasets.load_dataset("aquiro1994/naics-gh"). Croissant metadata is generated automatically by Hugging Face from the dataset card’s YAML front matter, satisfying the NeurIPS Datasets & Benchmarks 2025 machine-readable-metadata requirement. The labeling pipeline code is at https://github.com/alexanderquispe/naics-github-classifier (MIT License); the training code and fine-tuned RoBERTa-large checkpoint are at https://github.com/alexanderquispe/naics-github-train and https://huggingface.co/alexanderquispe/naics-github-classifier. A Zenodo DOI for the dataset will be issued at camera-ready.

8.0.0.1 End-to-end replication of the labeling pipeline.

To quantify the reproducibility of the dataset-generation process, we re-ran the full retrieval pipeline from the frozen source extractions on different hardware (consumer RTX 3080 vs.the original A100). The raw retrieval depth is exactly reproduced by construction (20,879 rows per region), and the deduplicated candidate sets match to within \(\pm 0.1\%\) per region — 10,985 vs.983 (USA), 10,529 vs.531 (EU), and 9,674 vs.664 (AU); 31,188 vs.178 in total (\(+0.03\%\)). The residual differences are attributable to FP16 floating-point variation across GPU architectures shifting a handful of borderline nearest neighbors.

8.0.0.2 Jurisdiction-blind public replication package.

The released corpus was generated by the per-region pipeline described in §3.1.0.4, which requires access to jurisdiction-tagged source extractions that we do not publish. To let external researchers reproduce the methodology end-to-end, the pipeline repository additionally provides a jurisdiction-blind variant that concatenates the three extractions into a single shuffled pool before embedding and retrieves with a proportionally scaled depth (base \(k\) tripled). Because a merged pool deduplicates the cross-region overlap that separate regional pools structurally retain, this variant yields a corpus of comparable — not bit-identical — size and composition; it exists to verify the method, while the released corpus remains the canonical artifact.

9 Conclusion↩︎

NAICS-GH supplies a missing resource: a publicly available mapping from GitHub repositories to standardized industry sectors, GPT-4.1-labeled over a multi-jurisdictional source pool and validated against a human-labeled random subsample at 96.98% precision. We hope it enables empirical work on the industrial composition of open-source production — over time and in response to the rise of AI coding assistants. An inter-annotator-agreement coefficient on a doubly labeled subsample and a recall estimate against an independent reference list of sector-labeled repositories are concrete next steps.

10 Source-extraction SQL↩︎

The canonical Presto/Trino query against GitHub’s internal warehouse that produced the source parquets. The query is executed three times, varying only the accounts_current.country_account IN (…) clause (see §3.1.0.2); each output is processed per region by an identical pipeline configuration (§3.1.0.4), and the country_code column is dropped at that point.

SELECT
    nwo, num_stars, is_fork, is_archived, has_readme,
    content_size AS readme_content_size, disk_usage_bytes,
    spdx_license,
    repositories_current.id AS repository_id,
    from_utf8(repositories_current.description) AS description,
    repositories_current.topics,
    from_utf8(readme_current.content) AS readme_content,
    accounts_current.country_account AS country_code,
    COUNT(DISTINCT issues_current.id) AS num_issues,
    COUNT(DISTINCT issues_current.user_dotcom_id) AS num_issue_authors,
    COUNT(DISTINCT github_collab_commit_contributions.user_id)
        AS num_commit_contributors,
    SUM(github_collab_commit_contributions.commit_count) AS num_commits,
    MAX(github_collab_commit_contributions.created_at) AS last_commit_at
FROM hive.canonical.repositories_current
JOIN hive.canonical.accounts_current
    ON repositories_current.owner_dotcom_id = accounts_current.dotcom_id
LEFT JOIN hive.canonical.issues_current
    ON issues_current.repository_id = repositories_current.id
    AND NOT issues_current.is_user_hidden
JOIN hive.suez.readme_current
    ON readme_current.repository_id = repositories_current.id
    AND LOWER(path) = 'readme.md'
    AND LOWER(filename) = 'readme.md'
LEFT JOIN hive.reference.licenses
    ON licenses.id = repositories_current.license_id
LEFT JOIN delta.snapshots.github_collab_commit_contributions
    ON github_collab_commit_contributions.repository_id
       = repositories_current.id
WHERE NOT is_spammy_owner
    AND is_public = TRUE
    AND has_readme = TRUE
    AND num_stars >= 1
    AND is_fork = FALSE
    AND accounts_current.country_account IN ( /* per-region list */ )
    AND content_size >= 750
    AND disk_usage_bytes > 0
    AND repositories_current.description IS NOT NULL
    AND TRIM(from_utf8(repositories_current.description)) != ''
GROUP BY 1,2,3,4,5,6,7,8,9,10,11,12
HAVING SUM(github_collab_commit_contributions.commit_count) > 5
   AND COUNT(DISTINCT github_collab_commit_contributions.user_id) >= 2
ORDER BY repository_id

The warehouse data was current through April 15, 2025 (USA presample); the EU README extraction snapshot was dated August 15, 2025.

11 NAICS sector definitions↩︎

We use the 20-sector top-level NAICS 2022 hierarchy as compiled by the U.S. Census Bureau [20]. The pipeline consumes a flat JSON taxonomy file (naics_titles_by_group_6digit_clean.json, released with the code) keyed by 2-digit sector code; each value is a semicolon-separated string of 6-digit subindustry titles drawn from the 2022 hierarchy. Across the 20 sectors there are 1,029 distinct subindustry phrases.

Table 1 reports, for each 2-digit sector, the canonical Census name, the number of subindustry phrases in our taxonomy file, and the first three subindustry phrases as they appear in the JSON (verbatim, including any minor whitespace artifacts). The full per-sector listings are in the JSON file rather than this appendix in order to keep the paper a reasonable length.

12 LLM verification prompt↩︎

This appendix reproduces verbatim the prompt sent to GPT-4.1 for every candidate (repository, NAICS sector) pair. The three placeholders {naics_code}, {readme_text.strip()}, and {sector_industries.strip()} are substituted per row before the request is sent. Source: 3_eu_repo_gpt_classif.ipynb cell 22 (build_prompt function) and cell 26 (system message and API call). The corresponding Python mirror is 3_eu_repo_gpt_classif.py.

12.0.0.1 System message.

You are a domain expert in economic classification systems with a focus on NAICS industry {naics_code}.

12.0.0.2 User message.

TASK: GitHub Repository NAICS Sector Classification

You are a domain expert tasked with classifying GitHub repositories into NAICS industry sectors for GitHub's repository categorization system. Your goal is to determine with high precision whether this repository belongs to NAICS Sector {naics_code}.

REPOSITORY README CONTENT:
<readme>
{readme_text.strip()}
</readme>

TARGET NAICS SECTOR:
<naics_sector>
Sector {naics_code}: {sector_industries.strip()}
</naics_sector>

CLASSIFICATION FRAMEWORK:

A repository should be classified as "Yes" for this sector if it demonstrates CLEAR ALIGNMENT with one or more of these criteria:

1. **Industry-Specific Software**: Applications, tools, or systems designed specifically for use within this industry sector
   - Example: Farm management software for Agriculture (Sector 11)
   - Example: Church management systems for Religious Organizations (Sector 81)

2. **Sector-Relevant Functionality**: Code that implements processes, calculations, or workflows specific to this industry
   - Example: Prayer time calculators for Religious Organizations
   - Example: Crop yield prediction models for Agriculture

3. **Industry Domain Applications**: Software that directly serves businesses, organizations, or activities within this sector
   - Example: Restaurant POS systems for Food Services (Sector 72)
   - Example: Educational platforms for Educational Services (Sector 61)

4. **Sector-Specific Data/Research**: Datasets, analysis tools, or research implementations focused on this industry
   - Example: Agricultural sensor data analysis
   - Example: Healthcare outcome prediction models

CLASSIFICATION STANDARDS:

**INCLUDE ("Yes") when:**
- The repository's primary purpose aligns with the sector
- The software would be used by businesses/organizations in this sector
- The code implements sector-specific functionality or processes
- The project addresses sector-specific problems or use cases

**EXCLUDE ("No") when:**
- The repository serves multiple sectors equally (generic tools)
- Industry connection is only tangential or in examples
- The primary use case is outside this sector
- No clear business or operational relevance to the sector

SCORING GUIDE:
- 9-10: Core industry software with direct sector application
- 7-8: Strong sector relevance with clear industry use cases
- 5-6: Moderate sector connection with identifiable applications
- 3-4: Weak sector relevance, mostly tangential
- 1-2: No meaningful sector connection

ANALYSIS REQUIREMENTS:
1. Identify the repository's primary purpose and functionality
2. Assess alignment with the target NAICS sector
3. Determine the most applicable classification criterion
4. Consider practical usage within the sector

Provide your response in this exact JSON format:
{
    "NAICS {naics_code}": {
        "rationale": "Concise explanation of classification decision, including primary repository purpose, specific sector alignment criteria met, and justification for inclusion/exclusion",
        "score": "1-10",
        "match": "Yes" or "No"
    }
}

IMPORTANT: Base your decision on the repository's PRIMARY purpose and DIRECT applicability to the sector. Be precise and consistent in your classifications.

12.0.0.3 API parameters.

• Endpoint: https://api-model-lab.githubcopilot.com/chat/completions

• Model alias: gpt-4.1 (resolves to snapshot gpt-4.1-2025-04-14 at the GitHub Copilot LLM-lab endpoint).

• temperature: 0

• max_tokens (output): \(\min(3500,\; 128000 - n_{\text{input}})\), where \(n_{\text{input}}\) is the token count of the system plus user message under tiktoken.encoding_for_model(g̈pt-4.1)̈.

• Retry policy: up to 5 attempts on any non-200 response, with linear back-off sleep = 5\(\times\)attempt seconds.

• Response parsing: the model’s reply is searched with re.search(r’\{[\s\S]*\}’, reply) and the matched substring passed to json.loads. The expected output is nested under the outer key N̈AICS {code}̈ with string-valued match (Ÿes̈ or N̈ö), string-valued score (1̈̈–1̈0̈), and free-form rationale.

13 Schema of the released file↩︎

The released file train_data_gpt_ab8_score_with_code.parquet contains 6,588 rows and 6 columns. Every cell is non-null. Table 10 gives the per-column schema; Table 11 gives character-length and uniqueness statistics for the four text columns; the listing at the end shows one full row (with the readme_content field truncated for readability).

13.0.0.1 Example row.

One representative row from the released file (the readme_content field is truncated here to 240 characters for readability; the actual stored value may be much longer).

name_repo:      etn-occurrences
description:    Acoustic telemetry data
topics:         lifewatch; oscibio; animal-tracking; data-publication; dataset; biologging; animal-movement; rstats; fish; r
readme_content: # Acoustic telemetry datasets This repository contains scripts to publish fish tracking data from the [European Tracking Network (ETN)](lifewatch.be (specifically from the [Permanent Belgian Acoustic Receiver Network](lifewatch.be on [GBIF...
label:          0
code:           11

14 Stratification of the gold sample↩︎

This appendix tabulates the per-sector distribution of the validation gold sample (§5; \(n = 2,421\) rows) against the released NAICS-GH training corpus (\(n = 6,588\) rows). The largest signed delta is \(\pm 1.94\) percentage points; 16 of 19 sectors fall within \(\pm 1\) pp.

Reasonable agreement supports our claim that the headline 96.98% precision is not biased by an over- or under-representation of any single NAICS sector in the gold sample.

15 Training details↩︎

This appendix documents the full benchmark setup behind Section 6: the input cleaning, the six base models and their parameter counts, the complete list of training hyperparameters (identical across all six runs), and reproducibility notes. The canonical source is naics_training_gptdata.ipynb in the Colab naics_github folder; a re-runnable local refactor is naics-github-train/scripts/train.py.

F.1Input construction and cleaning↩︎

Each row is serialized as Repository: {name_repo} | Description: {description} | Topics: {topics} | README: {readme_content} and then passed through the helper clean_repository_text(text) which performs nine normalization steps:

1. Strip Markdown badges and shields (![…]([url]) and [![…](…)](…));

2. strip license/copyright headers (MIT License, Apache License, GPL, BSD, Copyright …);

3. collapse URLs to the bare domain (https?://(domain)/path?... \(\rightarrow\) domain);

4. strip Markdown headers (#…######) and bold/italic/code markers (*, _, ~, );

5. replace fenced code blocks with a code-{lang} placeholder; strip inline backticks but keep the content;

6. normalize tech-stack mentions (js \(\rightarrow\) javascript, py \(\rightarrow\) python, reactjs \(\rightarrow\) react, nodejs \(\rightarrow\) nodejs);

7. normalize excessive punctuation (!! \(\rightarrow\) !, ?? \(\rightarrow\) ?, .... \(\rightarrow\) …);

8. normalize whitespace (collapse multiple newlines and spaces);

9. strip installation-command noise (npm install, pip install, git clone, and the rest of those lines).

The cleaned string is the text column passed to the tokenizer.

F.2Base models and parameter counts↩︎

All six runs share data, splits, and hyperparameters; only the base model differs.

F.3Training hyperparameters↩︎

All values below are set in setup_training_arguments() and were applied identically to every model run.

F.4Splits and evaluation metric↩︎

The 6,588-row corpus is divided into train / validation / test sets of 4,611 / 659 / 1,318 via sklearn.model_selection.train_test_split with test_size = 0.2, val_size = 0.1, random_state = 42, and stratification on the label column. All 19 NAICS classes appear in every split. The evaluation metric for both metric_for_best_model and the headline test-set result is the weighted F1 over the 19 classes; accuracy, weighted precision, and weighted recall are reported alongside.

F.5Hardware and runtime↩︎

All six runs were performed on a Colab Pro+ instance with a single NVIDIA A100 (40 GB). BF16 mixed precision and the fused adamw_torch_fused optimizer keep memory use comfortable for the largest model (DeBERTa-v3-large, 400 M parameters) at the per-device batch size of 8. End-to-end runtime per model was on the order of 8–15 minutes including checkpointing, with early stopping commonly firing between epochs 4 and 7.

F.6Reproducibility↩︎

The published RoBERTa-large checkpoint at https://huggingface.co/alexanderquispe/naics-github-classifier is the artifact produced by the above pipeline with model_id = r̈oberta-largë. To reproduce, clone naics-github-train, place the released training file train_data_gpt_ab8_score_with_code.parquet under data/raw/, and run python scripts/train.py --model roberta-large --batch-size 32 --epochs 8 (the script’s defaults match the hyperparameters listed in Table 14). With the same seed = 42, the F1 figure of 86.45% should reproduce within \(\pm 0.5\) pp of stochastic variation.

16 Qualitative error analysis↩︎

17 Datasheet for Datasets↩︎

A “Datasheet for Datasets” following Gebru et al. (2021). This document accompanies the NAICS-GH dataset (Industry Classification of GitHub Repositories Using the North American Industry Classification System).

• Dataset version: v1.0

• Last updated: 2026-05-22

• Authors: Kevin Xu (GitHub), Alexander Quispe (GitHub)

• Contact: alexanderquispe@github.com (corresponding)

• License: CC-BY-4.0 (labels and metadata); MIT (pipeline code)

For what purpose was the dataset created? NAICS-GH was created to enable empirical work on the industrial composition of open-source software production. GitHub hosts millions of public repositories but provides no native indication of which industry a repository serves. NAICS-GH maps a representative subset of repositories from the USA, the European Union, and Australia onto the 2-digit North American Industry Classification System (NAICS), which is the industry-classification standard used by US, Canadian, and Mexican statistical agencies. The dataset is also the training data for a downstream RoBERTa-large classifier that propagates these labels to arbitrary repositories at inference time.

Who created the dataset and on behalf of which entity? The dataset was created by Kevin Xu and Alexander Quispe at GitHub.

Who funded the creation of the dataset? GitHub. Compute for LLM inference was paid through GitHub’s internal LLM access; no external grants were used.

Any other comments? The dataset is one half of a two-repository system. This file documents the labeled corpus; the downstream classifier and its training code live in a sibling repository, naics-github-train.

What do the instances represent? Each instance is a public GitHub repository, identified by its owner/name (nwo) string. The labels attached to each instance indicate which 2-digit NAICS industry sector the repository serves, along with the LLM-generated rationale and confidence score.

How many instances are there in total? The released corpus contains 6,588 repositories in the parquet file train_data_gpt_ab8_score_with_code.parquet. This is the file that produces the published RoBERTa-large baseline.

Does the dataset contain all possible instances, or is it a sample from a larger set? It is a sample. The pipeline begins with three regional dumps of public GitHub repositories (totaling 1,372,489 rows): - USA: 510,380 - European Union: 530,898 - Australia: 331,211

For each NAICS subindustry (1,000+ across the 20 top-level sectors), the top 20 most semantically similar repositories were retrieved using BGE embeddings + FAISS, then LLM-scored. Only repositories scoring at least 8 on the 1–10 rubric were retained.

What data does each instance consist of? Each instance has six columns in the released file (train_data_gpt_ab8_score_with_code.parquet):

The columns nwo, match, score, rationale, and repo_url present in intermediate files are intentionally dropped from the public release so that the corpus is model-ready (no LLM-provenance, no PII via repo-owner URLs).

Is there a label or target associated with each instance? Yes. The label is the 2-digit NAICS sector code in the code column (string form), or equivalently the integer label column (0–18, monotonic in NAICS code order).

Is any information missing from individual instances? - All four text columns and both label columns are fully populated (zero NaN values). - topics is empty for ~69% of rows (4,533 of 6,588): not all GitHub repositories declare topic tags. - readme_content is preprocessed (code blocks stripped, HTML removed, URLs collapsed) and truncated to 3,000 whitespace-separated tokens before LLM scoring; the released field reflects the preprocessed text, not the original.

Are there recommended data splits? For the RoBERTa-large baseline we used 70% train / 10% validation / 20% test (\(n = 4{,}611 / 659 / 1{,}318\)) with seed=42. Splitting was random and stratified by NAICS sector. Practitioners building their own classifiers are free to re-split.

Are there any errors, sources of noise, or redundancies? - Label noise (USA scope only). Human re-validation of 2,421 USA repositories found 96.98% GPT-4.1 label precision overall, but two sectors (31–33 Manufacturing and 42 Wholesale Trade) had only ~73% precision at the score ≥ 8 threshold. EU and AU portions are GPT-4.1-labeled by the same pipeline but have not yet been included in a manual gold sample. - Duplicate name_repo. Repository short names are not globally unique; the released file has 447 rows sharing a name_repo with another row, of which 113 share both (name_repo,code). These are different repositories with identical short names that independently passed verification. The upstream intermediate files retain nwo as a globally unique identifier. - READMEs. Some READMEs are predominantly HTML badges, build configs, or templates that carry little semantic signal. The preprocessing step removes most of these artifacts but cannot recover content that wasn’t there.

Is the dataset self-contained, or does it link to or otherwise rely on external resources? The released parquet is self-contained. The pipeline that generated it relies on (a) BAAI/bge-large-en (released model, Hugging Face), (b) FAISS (open source), and (c) GPT-4.1 (snapshot gpt-4.1-2025-04-14) accessed via the GitHub Copilot LLM-lab endpoint. The repository content (READMEs, descriptions) is preserved in the file at the time of capture; we do not re-fetch live GitHub content.

Does the dataset contain data that might be considered confidential or that includes content protected by attorney-client privilege or similar? No. All repositories are public on GitHub.

Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? Possibly, since READMEs contain free-form user-submitted text. We do not apply explicit content filtering. Anyone re-using the README content should be aware that public repository content can include offensive language.

Does the dataset identify any subpopulations (e.g., by age, gender)? No. The dataset captures organizational and industrial-economic attributes of repositories, not demographic attributes of contributors.

Is it possible to identify individuals from the data? Repository owner names (nwo prefix) may identify individual GitHub account holders, since public repositories often live under personal accounts. This is the same level of identifiability already present on GitHub itself; the dataset does not add information beyond what GitHub publishes.

Does the dataset contain data that might be considered sensitive in any way? The dataset records that a given GitHub user’s public repository serves a given industry sector. We do not consider this sensitive beyond the existing public nature of GitHub, but downstream uses should respect the principle that aggregating labels about individuals can amplify identifiability.

How was the data associated with each instance acquired? Source repositories were extracted from GitHub’s internal Trino-on-Hive data warehouse via a Presto/Trino SQL query against the hive.canonical.repositories_current, hive.canonical.accounts_current, hive.suez.readme_current, and delta.snapshots.github_collab_commit_contributions tables. The query yielded three regional dumps (USA: 510,380; EU: 530,898; AU: 331,211 — totaling 1,372,489 repositories). Each repository’s NAICS label was then derived via a two-stage pipeline (see Section 4). The label is not directly observed but is the output of an algorithm that combines semantic retrieval with an LLM-scored rubric.

What mechanisms or procedures were used to collect the data? - The source SQL filters on is_public=TRUE, is_fork=FALSE, NOTis_spammy_owner, num_stars>=, README size ≥ 750 bytes, ≥ 6 commits, ≥ 2 distinct commit contributors, and non-empty description. Jurisdiction is assigned by the owner account’s country_account field. - Embeddings were computed with BAAI/bge-large-en (1024-dim, L2-normalized, FP16 on GPU). - FAISS IndexFlatIP was used for exact inner-product search, equivalent to cosine similarity on the normalized vectors. - LLM scoring was issued through the GitHub Copilot LLM-lab endpoint (api-model-lab.githubcopilot.com/chat/completions) using GPT-4.1 (alias resolved to snapshot gpt-4.1-2025-04-14), with temperature= and dynamic max_tokens capped at 3,500.

If the dataset is a sample from a larger set, what was the sampling strategy? For each of the 1,029 NAICS subindustry phrases across 20 sectors, the top-k nearest-neighbor repositories under BGE cosine similarity were retrieved per region. The default k = 20; for the three sectors with fewer than 20 subindustries the formula max(20,ceil(400/n)) boosts k to 24/27/29 so each sector retrieves roughly 400 candidates. After deduplication and GPT-4.1 filtering at score ≥ 8, we kept all surviving repositories. The sampling strategy is therefore “top-k semantic retrieval per industry query,” not uniform random sampling.

Who was involved in the data collection process? The pipeline was designed and operated by the authors. Research assistants conducted the manual gold-set re-check (n=2,421 USA repositories). Each row was reviewed by exactly one annotator; no inter-annotator agreement coefficient was computed.

Over what timeframe was the data collected? The USA source dump was extracted on April 15, 2025; the EU README extraction snapshot is dated August 15, 2025. LLM scoring was conducted between mid-2025 and early-2026. The dataset v1.0 was finalized in early 2026.

Were any ethical review processes conducted? No formal IRB review, since the dataset uses only public data and contains no human subjects. The release was approved internally at GitHub.

Did you collect the data directly from the individuals in question, or obtain it via third parties or other sources? The data was derived from publicly available GitHub content. We did not contact repository owners individually.

Were the individuals in question notified about the data collection? No. The data is public; we considered notification infeasible and unnecessary at the scale of millions of source repositories.

Did the individuals in question consent to the collection and use of their data? GitHub users consent to public visibility of their public repositories under GitHub’s Terms of Service. They do not specifically consent to industry-sector labeling, but the labels are derived computationally from public content.

If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future? We will provide a takedown mechanism: any repository owner can request removal of their repository from the released dataset by emailing the corresponding author. We will issue a v1.x with the requested removal and document the change.

Was any preprocessing/cleaning/labeling of the data done? Yes, extensively. The pipeline is:

1. README preprocessing. For embedding input, the raw README is truncated to its first 1,000 characters (no markup stripping at this stage). For LLM scoring, code blocks, HTML tags, badge URLs, and image markdown are stripped and the text is truncated to 3,000 whitespace-separated tokens (word-count proxy, not BPE tokens).

2. Composite text construction. LLM scoring uses the combined string "description:{desc},topics:{topics},readme:{readme}", constructed at row time and inserted into the <readme>…</readme> block of the prompt. Embedding uses the truncated README only; description and topics enter the pipeline at the scoring stage.

3. Embedding. BGE-large-en with the asymmetric BGE prefixes: "Representthisdocumentforretrieval:" for documents and "Representthisqueryforretrieval:" for queries.

4. Retrieval. For each NAICS subindustry, the query "Repositoriesabout{subindustry}" is embedded and the top-k nearest repositories are retrieved (default k = 20; the three sectors with fewer than 20 subindustries use a boosted k = max(20,ceil(400/n)) = 24, 27, or 29).

5. LLM scoring. Each (repository, sector) candidate is presented to GPT-4.1 with the rubric prompt (see the paper appendix). The model returns a JSON object nested under the outer key "NAICS{code}", with string-valued match ("Yes"/"No"), string-valued score ("1"-"10"), and free-form rationale. Downstream code coerces match to a boolean and score to an integer.

6. Filtering. Only repositories scoring at least 8 are retained.

7. Per-region merge. The three regional outputs are concatenated.

8. Minimum-class filter (training derivative only). Sectors with fewer than 80 repositories are dropped, removing sector 55.

Was the raw data saved in addition to the preprocessed data? Yes. The original parquets of unfiltered GitHub repositories were preserved (USA, EU, AU dumps), as well as the three intermediate per-region GPT outputs ({usa,eu,au}_2k_gpt_ab8score.parquet). These are not in v1.0 of the public release but will be added to Zenodo at camera-ready.

Is the software used to preprocess/clean/label the data available? Yes, in this repository (MIT License). The relevant entry points are scripts/01_generate_embeddings.py through scripts/05_filter_results.py, with prompts in src/classification/prompt_builder.py.

Has the dataset been used for any tasks already? Yes: - Fine-tuning six pretrained encoders (RoBERTa, ModernBERT, and DeBERTa-v3 in base and large variants) on the released corpus. RoBERTa-large is strongest at 86.45% F1 / 86.35% accuracy on a held-out 20% test set; the fine-tuned checkpoint is available on Hugging Face at alexanderquispe/naics-github-classifier. - Internal industry-mix analyses of open-source production.

Is there a repository that links to any or all papers or systems that use the dataset? At release time, the paper repository will track downstream uses. The released Hugging Face dataset card will also link to known references.

What (other) tasks could the dataset be used for? - Studying the industrial composition of open-source contributions across regions or over time. - Building classifiers that map new repositories to NAICS sectors. - Measuring AI-coding-assistant adoption by industry (the downstream classifier is already being used for this). - Comparative open-source ecosystem analysis vs. economic classification statistics.

Is there anything about the composition of the dataset or the way it was collected and preprocessed that might impact future uses? - NAICS is a North American taxonomy. Applying it to EU and AU repositories assumes mapping consistency that may not hold for some sectors (notably Sector 22, “Utilities,” which has US-specific regulatory structure). - English-only embeddings. Repositories with non-English READMEs are under-represented. - Score-conditional reliability. Manufacturing and Wholesale Trade labels are only ~73% precise at score = 8 and should be filtered at score ≥ 9 for high-precision applications.

Are there tasks for which the dataset should not be used? - Targeting individual contributors. The dataset is intended for aggregate analysis; using it to profile individual developers is outside its intended use. - Inferring repository quality, popularity, or value. The dataset encodes industry classification, not quality. - Treating labels as ground truth without conditioning on score. Two sectors have substantially higher error rates, and the LLM score should be used as a confidence signal.

Will the dataset be distributed to third parties outside of the entity on behalf of which the dataset was created? Yes. The dataset is publicly released.

How will the dataset be distributed? - Hugging Face Datasets: aquiro1994/naics-gh — public, CC-BY-4.0, with stratified train/validation/test splits ready for datasets.load_dataset. Croissant metadata generated automatically by HF. - Zenodo: DOI-citable archive (DOI to be reserved at release). - GitHub: Code, prompts, and reproduction scripts at the two repositories listed in §1.

When will the dataset be distributed? At paper acceptance / camera-ready, expected late 2026 or 2027.

Will the dataset be distributed under a copyright or other intellectual-property license? Yes: - Labels and metadata: CC-BY-4.0. - Pipeline code: MIT. - Repository content (READMEs and descriptions) excerpted in the dataset remains governed by each repository’s own license; we preserve spdx_license in the released file so users can filter by license type.

Have any third parties imposed IP-based or other restrictions on the data associated with the instances? Each source repository’s content is subject to its own license. The dataset preserves SPDX license identifiers so users can comply with upstream license terms (e.g., excluding non-commercial-licensed repositories from commercial uses).

Do any export controls or other regulatory restrictions apply to the dataset? None known.

Who will be supporting/hosting/maintaining the dataset? Alexander Quispe (corresponding author) and Kevin Xu, with GitHub’s support.

How can the owner/curator/manager be contacted? alexanderquispe@github.com.

Is there an erratum? A CHANGELOG.md will track all post-release errata at the Hugging Face dataset page and in the GitHub paper repository.

Will the dataset be updated? - v1.x patches for take-down requests, typo fixes, and metadata corrections. - v2.0 planned to add: India (1M repos), finer-grained 3- to 6-digit NAICS labels, and multilingual README support. Timeline to be confirmed.

If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances? We retain the labels indefinitely subject to take-down requests.

Will older versions of the dataset continue to be supported/hosted/maintained? Yes, on Zenodo. Each tagged version has its own DOI; older versions remain citable but will not receive further updates.

If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? Yes. Pull requests to the GitHub paper repository, issues on the Hugging Face dataset page, and email to the corresponding author are all accepted. Substantive contributions (e.g., new regions, multilingual labels) will be co-credited in the next dataset release.

References↩︎