1 Introduction↩︎

The Internet of Things (IoT) has experienced significant growth in recent years, with estimates suggesting that over 75 billion IoT devices will be in use by 2025 [1]. This rapid expansion reflects IoT’s integration across diverse domains, such as workplace environments [2], home management [3], healthcare delivery [4], sports applications [5], and educational settings [6]. This proliferation has created a critical challenge that devices from different manufacturers often struggle to communicate seamlessly. The resulting interoperability gaps have generated inefficiencies, increased deployment costs, and widespread user frustration [7]. Understanding how practitioners navigate these challenges is important for advancing IoT adoption and reliability.

In response to these interoperability obstacles, the Matter standard (formerly Project CHIP) was announced in 2019, with the first official specification released in 2022 [8]. Matter is an open-source, application-layer connectivity standard intended to support communication among smart home devices from different manufacturers, and its development is coordinated by the Connectivity Standards Alliance (CSA) [9]. According to CSA reports, the alliance includes over 550 companies and organizations as of 2025, e.g., Apple, Amazon, Google, IKEA, and Huawei [10]–[12]. Earlier consortia, such as the Zigbee Alliance and other home-automation efforts, also sought to improve interoperability but were limited by fragmented ecosystems, varying levels of engagement, and constrained device support [13]–[15]. As of recent CSA and industry reports, more than 1,200 devices are Matter-certified, and additional products can participate via bridge mechanisms that connect non-Matter devices into Matter-based systems [16]–[18]. This paper examines how developers experience this emerging standard in practice, rather than evaluating its adoption claims.

Despite Matter’s growing deployment and substantial industry backing, the practical realities of implementing this emerging standard remain poorly understood. Prior research has demonstrated the value of systematically analyzing developer-reported issues in related technical domains, surfacing recurring pain points that inform the evolution of frameworks and tools [19]–[21]. For emerging standards like Matter, such empirical analysis of developers’ lived experiences implementing the specification directly is particularly essential. These insights can inform CSA working groups and member companies that already participate in the Project CHIP repository, helping refine specification documentation, prioritize test and tooling improvements, and adjust guidance so that the standard is easier and safer to deploy.

To address this empirical gap, we conducted a systematic analysis of issues reported on the official GitHub repository for Matter (Project CHIP) [22]. GitHub Issues serve as a valuable data source because the discussions capture organic developer engagement across the full spectrum of use cases. Unlike formal bug reports or vendor surveys, these threads reflect real-world challenges encountered by both professional developers and hobbyists working across diverse application domains [23], including integration challenges and specification gaps. By analyzing and categorizing these discussions, we identify the most prevalent technical and usability issues, reveal temporal patterns in problem persistence and resolution, and highlight areas where the Matter specification or its sup-porting ecosystem requires refinement.

Our work aims to make the following contributions:

• We present, to the best of our knowledge, the first empirical analysis of developer and practitioner-reported issues related to Matter standard implementation, grounded in systematic analysis of GitHub issue discussions.

• We identify and categorize the most common themes emerging from reported issues, analyze distribution and resolution timelines across issue types, and uncover patterns in community engagement and communication dynamics.

• We provide actionable recommendations for the CSA, developers, and the broader IoT community based on empirical evidence, informing future improvements to the Matter specification, developer resources, and best practices in IoT interoperability.

2 Background↩︎

2.1 The Matter Standard↩︎

To address interoperability issues in smart homes and IoT, major industry stakeholders launched Project Connected Home over IP (CHIP) in 2019, later rebranded as Matter and now developed under the Connectivity Standards Alliance (CSA) and maintained in the official Project CHIP connectedhomeip repository [22]. Matter is an open-source, application-layer protocol intended to unify smart device connectivity across brands and platforms. A notable feature is its support for bridge devices, which allow existing non-Matter devices (e.g., Zigbee or Z-Wave) to participate in Matter-based ecosystems [24]. This design extends interoperability while preserving legacy deployments, but also introduces additional configuration and reliability challenges for implementers [25], [26]. Matter further incorporates secure commissioning and certificate-based authentication to protect device integrity and user data [27], and has evolved through successive versions (e.g., Matter 1.5) as vendors add new device classes and features [28], [29].

2.2 GitHub and Issue Tracking↩︎

Modern software engineering relies heavily on collaborative platforms that integrate version control, continuous integration and deployment (CI/CD), and project management [30]–[32]. GitHub has become a central platform of this kind, combining Git-based version control with tooling for code review, automated testing, and workflow automation [33]–[35]. With over 100 million developers and 370 million repositories, it is a key locus for open-source infrastructure and standards development [36], [37].

A core GitHub feature is its issue tracking system. GitHub Issues allow contributors to report bugs, request features, document tasks, and discuss design decisions in an organized and searchable form [38], [39]. Labels and issue types help categorize and prioritize reports, while pull requests provide a structured workflow for proposing and reviewing code changes before merging them into the main branch. Together, these mechanisms support asynchronous coordination and integrate closely with CI/CD pipelines, enabling projects to evolve through iterative, peer-reviewed changes.

Prior work shows that GitHub Issues discussions often capture both technical and socio-technical concerns, such as documentation gaps, environment error, and development bugs  [39]–[41]. For large and evolving projects, the issue tracker becomes a record of recurring pain points and solutions that may not appear in formal documentation or release notes [42]. In the case of Matter, the official Project CHIP repository on GitHub hosts the reference implementation, documentation, and support channel, and is used to raise questions, report bugs, and review proposed changes with other users including CSA members and participants  [43]/. The presence of issue and pull-request guidelines provides a structured path for feeding developer feedback into the codebase, which can also be discussed further in CSA working groups[44]. This study uses GitHub Issues from the Project CHIP repository as an empirical window into how developers and practitioners experience Matter in practice, focusing on challenges, such as technical, interoperability, and network-related problems, and how the community works to resolve them over time.

3 Related Work↩︎

Understanding the challenges that software engineers and technology practitioners encounter is essential for guiding both standard evolution and tooling improvements. While traditional qualitative methods such as interviews and surveys provide valuable depth, they face constraints in emerging and technical domains. Recruitment is often difficult due to the niche expertise required, and non-disclosure agreements frequently limit candid sharing of implementation experiences [45]–[47]. Moreover, interviews and surveys typically capture a temporal snapshot, making it harder to trace how problems emerge, evolve, and resolve over time [48]. As standards like Matter mature within fast-moving ecosystems, methods that can observe longitudinal practitioner discourse become critical for capturing authentic, evolving challenges.

3.1 Mining Developer Discussions on GitHub↩︎

Open collaboration platforms such as GitHub provide a unique perspective into the collective problem-solving practices of software engineers. Issue tracking preserves both technical details and the social dynamics of development, in particular, documenting the lifecycle of bugs, feature requests, and design debates [23], [49]. This publicly accessible data enables large-scale empirical analysis of real-world development challenges, complementing traditional qualitative methods with longitudinal, naturally occurring records of developers’ experience.

Prior research demonstrates how GitHub Issues can reveal patterns in developer behavior and project health. Zhou et al. found that affective communication, such as emoji use, improves participation and accelerates issue resolution, underscoring how community engagement shapes technical outcomes [50]. Liao et al. used visual analytics to map engagement and responsiveness within open-source projects, showing how community coordination influences project dynamics and maintenance burdens [51]. Bühlmann and Ghafari examined how developers respond to security vulnerabilities, identifying recurring tensions between openness and risk management in collaborative environments [19]. Other studies have analyzed sentiment, language complexity, and responsiveness to trace collaboration bottlenecks and communication norms [52], [53]. In adjacent domains, Zhang et al. applied large-scale opinion mining to MLOps discussions, uncovering systematic practitioner pain points and building taxonomies of recurring issues [54]. Collectively, this body of work establishes that mining open developer communities provides a scalable lens on technical, organizational, and usability challenges.

Analyzing developer discussions offers insight into the iterative improvement of standards and their real-world adoption. By examining technical problems and issues, researchers can identify specification ambiguities and limitations, or ecosystem dependencies impact progress. Such insights can feed back into standardization bodies and technical teams, promoting better alignment between specification design and implementation feasibility [55]–[57]. For emerging standards like Matter, where both technical maturity and ecosystem support are still evolving, this feedback mechanism becomes especially important.

3.2 Developer Perspectives on Matter Standard↩︎

Matter standard for the Internet of Things aims to reduce fragmentation and enable seamless device communication, yet its real-world adoption depends heavily on developer experience. Prior work on Matter standard has largely focused on architectural evaluations, security analyses, or user-centric studies of smart home adoption, leaving the practitioner’s implementation perspective underexplored. Zegeye et al. developed a hardware testbed to evaluate Matter’s technical capabilities, demonstrating that integrating heterogeneous devices remains challenging even under controlled conditions [58]. Mangar et al. designed an experimental Matter deployment to assess user experience, revealing usability frictions that emerge in practice [59]. These studies highlight Matter’s potential but also underscore that specification design and implementation reality can vary.

More recent work has identified privacy and security risks introduced by Matter’s interoperability features, such as vulnerabilities in device pairing and delegation that enable hidden hub eavesdropping attacks [60], [61]. These findings suggest that the same mechanisms enabling flexible cross-device control can create security pitfalls. However, such analyses typically focus on protocol design rather than on the experiences of developers encountering these issues in production code. Understanding the practitioners and developers’ challenges essential for informing both specification refinements and tooling support. By systematically examining how developers themselves report Issues from the Project CHIP repository and resolve such challenges in the Matter codebase, this study complements prior research providing empirical insights into the technical, usability, and process challenges that shape Matter’s ongoing development and adoption.

4 Methodology↩︎

4.1 Data Collection and Preparation↩︎

This study analyzes publicly available issue discussions from the official Matter specification repository, project-chip/connectedhomeip, hosted on GitHub. The dataset was collected using the GitHub REST API and includes all public issues from the repository’s creation in 2020 through August 6, 2025. This time window spans multiple Matter specification releases, including version 1.4, and captures the ecosystem’s evolution over several development cycles. For each issue, we retrieved the title, body, status (open/closed), creation and closure timestamps, labels, and author identifiers.

We then preprocessed the issue text to prepare it for topic modeling and qualitative analysis. Issue titles and bodies were concatenated into a single document per issue and tokenized using NLTK [62]. Tokens were lowercased and cleaned by removing:

• Code blocks enclosed in triple backticks and inline code spans (which primarily contained stack traces, build logs, and command-line output),

• Numeric sequences longer than three digits,

• Non-alphabetic characters and single-character tokens.

Whitespace was normalized and stop words were removed using NLTK’s English stop-word list, extended with frequent repository-specific terms (e.g., “github”, “script”, “commit”, “issue”) to reduce noise and focus on substantive technical content. After preprocessing, the final corpus comprised 13,008 issues related to the Matter specification and its reference implementation. We also verified that removing embedded command output and logs did not materially change the topic structure (further details are provided in Appendix 8.1).

Ethical & Limitations. All data was collected from public GitHub using authenticated API calls and respecting rate limits and the analysis is limited to publicly visible data. Although CSA-affiliated contributors participate in the repository and issues follow structured reporting and review workflows, this dataset does not reveal how often, or how systematically, these reports are incorporated into CSA’s internal working-group decisions or changes to the Matter specification.

4.2 Topic Modeling↩︎

To identify latent themes in developer-reported challenges, we applied Latent Dirichlet Allocation (LDA), a widely used unsupervised topic modeling technique for software repository mining [63]. LDA was implemented in Gensim and trained on a bag-of-words representation of the preprocessed corpus, with each issue treated as a separate document and represented using a learned vocabulary dictionary.

We tuned the model through a systematic grid search over three hyperparameters: the number of topics \(k\), the number of passes \(p\), and the number of training iterations \(i\). Concretely, we varied:

• \(k \in \{5, 15, 25, 35, \ldots, 95\}\),

• \(p \in \{10, 15, 20\}\),

• \(i \in \{100, 200, \ldots, 1000\}\).

For each configuration, we computed the \(C_v\) coherence score, which measures semantic relatedness among high-probability terms within topics and has been shown to correlate with human judgments of topic quality. We selected the model with the highest \(C_v\) coherence for subsequent qualitative analysis. The optimal configuration used \(k = 17\) topics, \(p = 20\) passes, and \(i = 100\) iterations, yielding a coherence score of 0.59. Table ¿tbl:visual:topics95table? summarizes the resulting topics, their top keywords, and the higher-level categories used in our analysis.

4.3 Qualitative Thematic Interpretation↩︎

To move from automatically discovered topics to interpretable developer challenges, we conducted a multi-stage qualitative interpretation process. For each of the 17 topics, we extracted the top 10 terms and sampled two sets of issues: (i) the 15 issues with the highest proportion of that topic, and (ii) a stratified random sample of 15 additional issues associated with the topic. This design balances exposure to prototypical cases with coverage of more diverse instances and served as the primary basis for our qualitative coding and label assignment. In practice, the term lists and 30-issue samples per topic quickly converged on stable, interpretable labels, and additional spot checks beyond the initial samples did not yield new categories or meanings.

Across all topics, this procedure yields 510 closely examined issues on top of the full quantitative analysis of all 13,008 issues, providing both depth and breadth in our interpretation. An initial investigator (Author 1) reviewed the terms and sampled issues for each topic and assigned a descriptive label reflecting the dominant technical, interoperability, or process concern (e.g., build failures, platform setup, cluster conformance). Two independent reviewers (graduate students with prior research experience in cybersecurity and IoT) then examined the assigned labels, topic keywords, and representative issues. They proposed refinements where labels did not adequately capture the observed discussion patterns. Disagreements were resolved through iterative discussion until consensus was reached.

To further validate the thematic structure, three faculty reviewers with backgrounds in IoT systems, security and privacy engineering, and usable security examined the topic labels, supporting examples, and the grouping of topics into broader categories (Testing, Interoperability, Development, and Platform & Network). They assessed whether the topics were coherent, whether the labels aligned with the underlying issue content, and whether the category structure reflected meaningful distinctions in the Matter ecosystem. This multi-stage process, which combines automated topic modeling with expert review, follows established practice in qualitative software repository mining and is intended to increase the credibility and interpretability of the resulting themes [64]–[68].

Using the validated labels and categories, we then computed aggregate statistics for each broad category, including issue counts, closure rates, unique participants, comment activity, and time-to-close (Table 1). These quantitative summaries provide a complementary view of how different classes of challenges manifest and are addressed over time within the Matter development community.

5 Result↩︎

Our dataset comprises 13,008 issues submitted by 1,010 unique contributors to the official Matter repository. Figure 1 shows that as the number of issues reported per user increases, the number of users at that reporting level decreases, indicating that a relatively small subset of contributors files a large share of issues[40], [49] . On average, a contributor reports approximately 13 issues.

Table 1 summarizes activity across the four broad categories derived from topic modeling and qualitative coding. Testing issues constitute the largest group (42.63%), followed by Interoperability (30.18%), Development (16.04%), and Platform & Network (11.15%). Each category captures a distinct dimension of practitioner work, ranging from maintaining build and tooling infrastructure to validating cross-platform behavior and network connectivity. Figure 2 further shows that issue volume grows sharply from late 2022 onward across all categories, corresponding to increased deployment and testing following the initial Matter releases, with Testing and Interoperability dominating the activity over time.

Among all contributors, 288 (about 28%) self-reported a company affiliation, spanning 166 distinct organizations. These include vendors and platform providers such as Amazon, Apple, Samsung, Nordic Semiconductor, Google, Silicon Labs, and the CSA itself, with Google appearing most frequently (31 users). This mix suggests that both alliance participants and other industry actors use the repository as a shared space for coordinating Matter development.

In the following subsections, we describe each broad category and its topics, and relate them to how developers experience Matter’s implementation, integration, and validation in practice. Table 3 in the appendix provides brief explanations of the technical terms used in describing the results.

5.1 Development↩︎

The Development category captures issues related to building the codebase, maintaining tooling, updating cluster implementations, revising documentation, and managing containerized build environments. This category contains 2,087 issues (16.04%) reported by 396 contributors (39.21% of all contributors). Two contributors filed more than 100 Development issues. Most issues have been closed (85.34%), and discussions are active, with an average of 3.39 comments and 3.31 labels per issue. Common labels include review – approved, review – pending, examples, and app. The average time to close is 23 days, suggesting steady but non-trivial effort to resolve build, tooling, and refactoring tasks. The following describe the five topics covered by Development category

5.1.1 Matter Build & Errors↩︎

These issues show that build stability is sensitive to rapid dependency changes and cross-language tooling, especially as the project spans multiple operating systems and architectures. Typical reports include build failures, misconfigured scripts, CI pipelines, and compiler errors, often triggered when new features are integrated or shared components are adapted to additional platforms. Examples include failing example applications (e.g., air-purifier conformance failures) and recurring Android and Java build breakages (Issues #37286, #34881, #31509, #36870).

5.1.2 Build Systems↩︎

These reports reflect the complexity of sustaining a unified build system for a multi-vendor IoT standard and highlight recurring friction when developers synchronize tooling across environments. Build Systems issues concern the underlying build infrastructure, such as configuration files for build tools (e.g., GN and CMake), branch alignment, and setup scripts for macOS, Linux, Android, and embedded targets. Representative issues involve bootstrap failures, missing Android build artifacts, and version mismatches across release branches (e.g., Issues #32966, #31191, #29935, #29936).

5.1.3 Cluster Management↩︎

These reports indicate ongoing evolution of Matter’s data model and the effort required to keep cluster behavior consistent and portable across device types. Cluster Management issues relate to implementing, updating, and refactoring clusters, attributes, and server-side logic. Developers report duplicated callbacks, migrations toward newer abstractions (e.g., ServerClusterInterface), and feature requests for additional cluster types (Issues #37044, #38699, #30234–30235), as well as adjustments to binding behavior and diagnostic reporting.

5.1.4 Documentation Maintenance↩︎

These reports highlight how documentation directly shapes developer understanding and can either prevent or introduce recurring classes of errors. Documentation Maintenance issues include updates to API descriptions, platform guides, example instructions, and CI/CD documentation. Many issues address formatting problems, version drift, or missing explanations that lead to configuration or build mistakes, such as restyling CI/CD documentation (Issue #30333) or updating Linux setup instructions (Issue #32144).

5.1.5 Docker Build Environment↩︎

These issues highlight the importance of reproducible containerized environments for maintaining a stable build pipeline across diverse vendor toolchains. Docker Build Environment issues focus on container images used in CI and vendor workflows, including updating vendor-specific images, refreshing SDK versions, and addressing resource constraints such as disk usage (e.g., Issues #29886, #35569, #31245, #33531).

5.2 Interoperability↩︎

The Interoperability category captures issues that arise when Matter components interact across platforms, applications, third-party dependencies, and device–controller interfaces. It contains 3,926 issues (30.18%) reported by 612 contributors (60.59% of all contributors). Five contributors filed more than 100 issues in this category, indicating sustained focus on cross-platform and integration-related problems. Most issues have been closed (77.43%), while 886 issues (22.57%) remain open. Issue discussions average 3.14 comments and 2.74 labels per issue, with frequent labels including review – approved, needs triage (managing and prioritizing issues), examples, darwin, and bug. The average time to close is 46 days, the longest among all categories, reflecting the complexity of diagnosing and resolving integration problems across heterogeneous environments. The Interoperability category contains four topics, each describing a specific set of integration challenges.

5.2.1 Dependency and Environment Configuration↩︎

These issues underline the difficulty of keeping dependencies aligned across the many platforms that participate in the Matter ecosystem. Dependency and Environment Configuration issues involve setup and maintenance of third-party libraries, build tools, and frameworks. Developers report build failures due to missing or incompatible components, misconfigured Python environments (e.g., SSL errors or missing tools), and version skew in large dependency trees (e.g., Pigweed and Abseil) across Linux, Android, ESP32, NXP, Raspberry Pi, and other platforms (Issues #38984, #29842, #30634, #39229).

5.2.2 Application Interface↩︎

Collectively, these issues show that application-level interoperability depends not only on the specification but also on convergence across platform-specific implementations. Application Interface issues involve how Matter applications behave on different platforms, such as Android, Linux, ESP32, TV casting apps, and camera controllers. Common problems include commissioning-flow failures, missing or inconsistent media features, and mismatched data formats, such as passcode flow differences in TV applications or missing fields in Android JSON encodings (e.g., Issues #32958, #38406, #39152). Reports also highlight gaps in WebRTC-based communication.

5.2.3 Cross Platform↩︎

These reports emphasize the challenge of maintaining consistent Matter behavior as vendors integrate the standard into their own SDKs and hardware. Cross Platform issues focus on platform-dependent defects, API inconsistencies, and missing feature parity across operating systems. Examples include unsupported commands, missing attributes, performance anomalies on specific hardware, questions about 32-bit Linux support, and attribute type mismatches between Android and iOS (e.g., Issues #23206, #22753, #33400).

5.2.4 Fabric State Management↩︎

This topic reflects the complexity of ensuring coherent fabric behavior across controllers and devices, particularly when handling long-lived subscriptions and recovery after network disruptions. Fabric State Management issues address fabric data handling, subscription state, idle or check-in behavior, session resumption, and controller–device synchronization. Many reports arise from interactions between the interaction model, subscription lifecycle, and persistent storage, such as implementing and testing check-in counters, resolving subscription resumption failures, and clarifying error handling and metadata representation (e.g., Issues #30705, #28903, #30965, #17227).

5.3 Platform & Network Category↩︎

The Platform & Network category covers issues related to network configuration, commissioning, and integration with platform-specific networking stacks. It includes 1,450 issues (11.15%) reported by 227 contributors (22.48%). One contributor filed more than 100 issues. This category has the highest closure rate (92.48%), with 109 issues (7.52%) still open. Issues average 3.37 comments and 3.73 labels, with common labels such as platform, review – approved, review – pending, examples, and submodules. The oldest open issue in this category dates to early 2022 and concerns commissioning Ethernet devices over BLE (Bluetooth Low Energy). Overall, these characteristics suggest continuous platform enablement and iterative refinement of network-related components.

5.3.1 Network & Platform Setup↩︎

These issues illustrate the ongoing work required to maintain network support across diverse hardware vendors. Network & Platform Setup issues involve Wi-Fi, Thread, BLE, and commissioning workflows on specific boards and SDKs. Many reports describe platform bring-up and configuration tasks, such as addressing low-power or sleep behavior, enabling support for particular development boards, updating vendor SDKs, or fixing Thread initialization on embedded devices (e.g., Issues #38078, #29070, #37810).

5.3.2 Connectivity↩︎

Connectivity issues show how bugs in communication or state management can disrupt device behavior across heterogeneous runtime environments. These issues focus on how devices exchange messages and maintain state, including OTA updates, command handling, and subscription behavior. Developers report unintended commands emitted by bridge applications, timeouts during remote updates, firmware update failures on specific boards, problems with subscription resumption, and initialization bugs, as well as runtime issues such as JNI reference leaks (e.g., Issues #38086, #28517, #27460).

5.3.3 Third-Party Dependency↩︎

Third-Party Dependency issues reflect continuous effort to keep Matter aligned with upstream network stacks and utility libraries while avoiding regressions. These issues capture updates to networking and cryptographic components such as OpenThread, OT-BR-Posix, Pigweed, mbedTLS, nanopb, and tracing libraries. Many issues are triggered by upstream version changes but are critical for platform stability because Matter implementations rely heavily on these dependencies (e.g., Issues #28774, #40221, #29523).

5.4 Testing Category↩︎

The Testing category constitutes the largest portion of the dataset, with 5,545 issues (42.63%) reported by 501 contributors (49.60%). Eleven contributors filed more than 100 issues. Most Testing issues are closed (85.83%), while 786 (14.17%) remain open. Discussions average 3.53 comments and 3.50 labels per issue, with common labels including review – approved, tests, app, and review – pending. The average time to close is 26 days. The oldest open issue ( #4397) dates to early 2021 and concerns separating build components to support independent unit testing. These statistics highlight the centrality of testing in Matter’s development workflow.

5.4.1 Specification Testing↩︎

Specification Testing issues illustrate continuous alignment work between the evolving Matter specification and the test suite that validates conformance. These issues relate to test-case definitions, XML specifications, cluster attributes, and Python-based test scripts. Many reports document updates required after specification or test-plan revisions, such as adjusting attribute checks, updating XML definitions, or adding new tariff or cluster tests (e.g., Issues #37988, #38404, #34517, #32937), as well as fixing gaps between YAML/XML specifications and implementations.

5.4.2 Hardware-based Security↩︎

Hardware-based Security issues show how security-related behavior is incrementally tested and stabilized across platforms. These reports concern unit tests, API refactoring, and behavior related to security and hardware-backed cryptography. Examples include adding or refining security support on specific platforms, restructuring interfaces to improve testability, strengthening integration tests for critical subsystems, and fixing low-level correctness problems such as iterator invalidation or brittle test fixtures (e.g., Issues #37977, #36809, #38008).

5.4.3 Testing Maintenance↩︎

Testing Maintenance tasks collectively ensure that the testing environment remains consistent with current tooling and code structure. These issues cover upkeep of the test infrastructure, such as fixing compiler warnings, updating ZAP-generated files, regenerating cluster definitions, adjusting CI behavior, and correcting file permissions (e.g., Issues #39900, #38820, #37360).

5.4.4 Testing Tools↩︎

Testing Tools issues highlight how limitations and bugs in testing tools interact with device behavior to surface failures that might not appear in simpler scenarios. These focus on test execution tools and harnesses, failure traces, and CHIPTool behavior. Many reports describe certification test failures, timeouts, and commissioning reliability problems, particularly under stress tests, as well as failures in semi-automated workflows, trusted-root mismatches, and cluster-specific conformance breakdowns (e.g., Issues #31467, #37655, #33994).

5.4.5 Cluster and Testing Conformance.↩︎

This topic indicates that application examples often require refinement to match current cluster specifications and conformance rules. These issues arise when example applications fail conformance tests or do not satisfy cluster requirements. Reports describe systematic failures across example apps, such as all-clusters, air-quality-sensor, lock, lighting, and thermostat applications (e.g., Issues #37269, #35680, #37295, #36824, #30321, #30323).

6 Discussion↩︎

6.1 Security- and Privacy-Relevant Issues↩︎

Security and privacy concerns appear throughout the repository. We identified 725 issues (5.57% of all issues) containing security or privacy-related keywords, of which 166 remain open. The earliest unresolved issue dates to February 2021 and discusses unsafe API usage. Most of these issues fall within the Testing category, reflecting the central role of conformance suites and test harnesses in surfacing security-relevant behavior. The issues cover incorrect privilege checks, insufficient validation of message fields, inconsistent error handling, and missing boundary conditions.

The persistence of unresolved security-related issues over multiple years suggests that it is an ongoing and non-trivial task in a Matter standard as it spans multiple dependencies, programming stacks, and device classes. These results motivate continued effort towards automated fuzzing, continuous conformance testing, and, where feasible, more formal validation of critical paths, particularly around commissioning, access control, and long-lived device subscriptions [19], [66]

6.2 Limits of Developer-Centric Data↩︎

The Matter repository functions primarily as a coordination space for developers and engineers. Issues are used to track design and development, platform integration, network configuration, test infrastructure, interoperability failures, and dependency maintenance, and to negotiate clarifications when specification text and implementation diverge. This provides a view of implementation-level challenges and the engineering effort required to sustain a large, cross-vendor codebase, however, the dataset does not show how often, or how systematically, these reports feed into CSA working‑group meeting or changes to the Matter specification updates.

Similarly, the repository only partially reflects end-user experience. Pain points such as home deployment difficulties, device onboarding failures, or multi-vendor configuration problems often surface instead in public discussion forums and consumer support communities. As an extension of this work, we plan to combine our corpus with data from user-focused venues, such as Matter Protocol and IoT discussion forums (e.g., Reddit ). This would connect developer-facing issues with the end-user challenges and can help identify where changes to tooling, documentation, or platform defaults would most directly benefit both practitioners and end users.

6.3 Implications for Matter and IoT Standards↩︎

This study provides an empirical view of how developers experience the Matter standard in practice. To the best of our knowledge, it is the first systematic analysis of Matter-related issues based on a comprehensive examination of the official repository. The results show how contributors use issues to report implementation problems, track issues, and coordinate fixes for the standard.

Our four broad categories, Testing, Interoperability, Development, and Platform & Network, reveal distinct patterns in both their numbers and resolution behavior. Testing and Interoperability dominate issue counts, and Interoperability issues remain open for longer on average, which suggests that cross-vendor and cross-platform behavior is harder to reproduce, attribute, and resolve than localized build or tooling defects. This pattern is consistent with Matter’s goal of unifying heterogeneous ecosystems and illustrates some of the practical effort required to achieve that goal.

These challenges also point to concrete directions for improvement. The large number of test-related issues indicates that strengthening test frameworks, tooling, and documentation could reduce recurring failures and improve the efficiency of day-to-day development. The amount and duration of interoperability issues suggest a need for clearer cross-vendor guidelines, richer diagnostics, and more comprehensive reference examples for common deployment scenarios. The consistent security issues in testing and commissioning processes highlight the need for automated and formal validation methods for critical protocol behaviors. Together, these findings show that using developer reported GitHub issues as a valuable feedback source for the Connectivity Standards Alliance, implementers and IoT vendors seeking to refine Matter and related interoperability standards.

7 Conclusion↩︎

This study offers an empirical view of how developers engage with the Matter standard through the official Project CHIP repository. Most reported issues concern testing and interoperability, with integration and network-related bugs often proving difficult to resolve. Development and platform support also demand ongoing maintenance, and security-relevant problems continue to appear across versions. Taken together, these findings indicate that improving Matter standard requires continued effort in testing infrastructure, clearer guidance for cross-vendor deployments, and more automated validation of critical protocol workflows as the standard continues to evolve.

Acknowledgment↩︎

The authors would like to thank Ramazan Yener and Nicholas Yeung for their help and feedback in analysis.

8 Appendix↩︎

8.1 Impact of Command Output Removal↩︎

While logs and stack traces often contain concrete failure details, we found that including them primarily inflated vocabulary size and emphasized low-level error messages rather than higher-level themes. To assess whether removing command output and logs affected topic structure, we trained LDA models on corpora with and without this content. The best models differed only slightly in topic count (18 vs.), and their topic–term distributions mapped to the same higher-level categories in qualitative analysis. This suggests that excluding verbose diagnostic log improves semantic coherence without losing important themes. We therefore use the filtered corpus in the main analysis to improve semantic coherence without losing important themes.

References↩︎