1 Introduction↩︎

Ontologies are a key enabler for scalable energy services in buildings [1]. They provide a semantic layer that supports interoperability and automation across projects, tools, and stakeholders. Ontologies have been investigated and compared in terms of coverage, expressiveness, interoperability, and transferability [2]–[9].

Project Haystack Version 4¹ (further referred to only as Project Haystack in this work) is a promising ontology that has achieved industrial adoption due to its intuitive tagging-based semantic model, openness, flexibility, and extensibility [2]. It defines a semantic tagging model for building data[2], [10]. To use this model in operational systems, a Haystack-compliant server implementation such as Haxall or the commercial SkySpark platform is required to store, manage, and expose the tagged data via standardized APIs [10], [11]. Client libraries and formats, such as the Phable Python client or the JSON-based Hayson representation, are then used to query, exchange, and integrate Haystack data with external applications.

Project Haystack’s flexibility, included query/filter language that is easy to use, organization structure and steering board comprised of established companies in the field of building technologies and control [12], as well as the availability of reference software implementations including a full ontology server including a database system explains its acceptance and spread.

Despite these strengths, Haystack exhibits several limitations, some related to the ontology itself, some related to the ways in which the ontology is defined and specified, and some related to the software ecosystem around Project Haystack. This results in ambiguities and inconsistencies in tag usage, and the lack of systematic, automated mechanisms to validate models against the standard, and hardships to implement and add such validation mechanisms [2], [5]–[7], [13]–[17].

Although Project Haystack is formally open, its reliance on custom file formats and domain-specific languages (e.g., Trio, Zinc, Xeto; Fantom, Axon) that originate from the Haxall ecosystem creates a de facto barrier to integration. Haxall is the reference implementation of a Haystack server [18], [19]. The company behind Haxall is Skyfoundry, offering their proprietary Skyspark architecture that builds upon Haxall and offers scalability, sensor data recording, and analytics services.

To illustriate how intertwined Project Haystack is with the Haxall ecosystem: The source code of the schemantic tagging models core definitions is provided in the form of Trio files, "a semi-subset of YAML" [20]. Building the actual ontology / applicable definitions from these sources requires the DefCompiler. The reference Parser for Project Haystacks custom Trio file format, as well as the DefCompiler - are part of Haxall. Haxall and all of its components are implemented in the Fantom [21] programming language.

Beyond forum and mailing list posts there is no discussion on the software engineering perspective Interfacing Haystack with mainstream programming languages and data pipelines remains cumbersome, and available parsers and compilers for Haystack definition formats are largely confined to the Fantom ecosystem.

Alongside missing validators, there is a lack of software supporting creation and construction of schemantic models of buildings using the Project Haystack ontology. This lack of validation together with the flexible tagging approach creates an interoperability and transferability problem. While the Xeto specification language is intended to fill this role in the future, development is still ongoing (part of Haystack 5), and Project Haystack specifications in the form of Xeto specifications is not yet complete. Once finished Xeto may be used to validate Haystack models; however, it is again implemented in the Fantom programming language.

1.1 Main contribution↩︎

In order to increase acceptance of the Project Haystack ontology, we argue that it has to become more open in a technical sense, by providing parsers, tools, validators in general purpose programming languages other than Fantom - beyond interfaces and bindings to Fantom implemented tools. Endeavors like J2inn’s Haystack definitions to typescript interfaces creator, the phable python client for Haxall servers, the JSON schema definitions of Haystack definitions, and a multitude of other smaller projects in (java, c++, etc) supports our thought that there is a real need for using / processing Haystack models in programming languages and frameworks outside the Fantom/Haxall ecosystem.

In this paper, we address these limitations by introducing a Python-based toolchain for Haystack. We present (i) a parser for Haystack definition files written in Trio, and (ii) a code generator that derives Pydantic [22] models and JSON Schema definitions from these parsed specifications. The resulting models enable static type checking and enable structural validation of Haystack grids within Python, as well as schema-based validation of JSON representations outside the Python ecosystem.

Applying this approach, we identify inconsistencies and missing definitions in selected Haystack releases, demonstrating the practical value of automated validation.

By enabling type checking and partial validation of Haystack models using widely adopted technologies, our work improves interoperability, lowers the entry barrier for new users, and facilitates integration with modern software stacks. All tools, generated models, and schemas are released publicly under an open-source license, with the goal of strengthening the Haystack ecosystem and opening a practical pathway beyond its current technical boundaries. All code and data, including the Pydantic models and JSON schemas are made publicly available under an open source license on github.²

2 Related Work↩︎

Quinn et al. [2] conduct a qualitative comparison of the Brick Schema and Project Haystack ontologies. They observe comparable completeness and expressiveness for both, though Haystack under-performs across six key ontology attributes: flexibility, portability, readability, extensibility, interoperability, and queryability.

Bergmann et al. [13] proposes a pathway to advance semantic interoperability for smart buildings. They discuss and compare various building metadata schemas, highlighting brick and project Haystack as the predominant operational metadata schemas, while criticizing Haystack for unverifiable correctness and extensive customizability, eroding confidence in its implementations.

Fierro et al. [14] perform a qualitative analysis of Project Haystack and identifies interpretability and consistency issues caused by its lack of formal semantics. They introduce Brick+, a formally defined ontology that enables reliable inference of Brick models from Haystack data, demonstrated on five models.

Qiang et al. [4] perform an in-depth systematic evaluation and comparison four building ontologies (Brick Schema, RealEstateCore, Project Haystack, and Digital Buildings) based on axiomatic design and assertions in a use case. Further, they investigate the compatibility between ontologies and propose paths towards alignments and translations between these different ontologies.

Luo et al. [5] reviews 24 building data tools across the building lifecycle, categorizing them as data dictionaries/terminologies, ontologies/schemas, and platforms. They discuss brick and project Haystack within their ontology category.

Pen et al. [6] investigate the modeling capabilities of six major building ontologies (Brick, RealEstateCore, Project Haystack, SAREF, Flow Systems Ontology, and ASHRAE 223P) for 4th–5th generation district heating and cooling (45GDHC) systems. They show that while existing ontologies handle basic building and sensor data well, they fall short in representing district-level energy exchanges and prosumer relationships.

Yefi et al. [16] investigate Brick and Project Haystack ontologies capabilities to model building systems beyond hvac systems. They identify seven key issues with the existing ontologies, amongst them lack of validation mechanisms, and semantic issues relevant to this work, and propose their own object oriented meta model.

A recurring theme in the related work on building ontologies is the project Haystack ontology being more widely adopted, performing en-par with other building ontologies when it comes to completeness and expressiveness, while suffering from its lack of formality, machine readability, and verifiability [2], [5]–[7], [13]–[17].

Several of the works mentioned above base their investigations on Project Haystack Version 3, explicitly noting that Version 4 was under development and expected to address some of the identified limitations [5], [13].

This work, is based in Project Haystack Version 4. In the following section, we demonstrate that many of the issues observed in earlier versions persist, although possibly to a lesser extent.

3 Design Challenges↩︎

This section presents some of the design challenges we faced when trying to implement basic means of automated validation for haystack models. Some are general issues with Project Haystack ontology definitions, others arise from our target to achieve this without relying on the Haxall/Fantom ecosystem.

3.1 Integration Challenges↩︎

Trio The Trio file format [20] is the primary format for project haystack definitions [23]. While Trio was based on YAML, it is not valid YAML, and YAML parsers can not be used to read Trio files. There is no open source Python implementation of a Trio parser available.

Build process The process of building the project haystack ontology is called normalization [24]. The reference implementation of normalization process is the DefCompiler, implemented in the Fantom programming language and contained in the Haxall project [25]. This build process is rather complex (e.g. the defx system [24] for late binding).

Broken versions The tagged release of project haystack version 3.9.8 is invalid due to missing type definitions. There seems to be a lack of testing and validation during and after the normalization process for versions before 3.9.10. The validation process described in [24] is minimal and rather simplistic, and does not capture many of the ontological challenges described below.

Serialization of Haystack models Haystack models of buildings and technical equipment within them can be serialized and stored in JSON format [26], aside from the Project Haystack specific formats (Trio and Zinc).

J2inn [27] provide a publicly available JSON Schema to validate this so called "Hayson" schema that is of substantial value for software developers working outside the Haxall/Fantom ecosystem. However, this schema validates only the general serialization format, it does not validate the contained models adherence to the Project Haystack ontological definitions.

3.2 Ontological Challenges↩︎

Definitions not machine readable Some definitions are not provided in machine readable form, with information provided only in prose text form in the Project Haystacks documentation, not being reflected in the corresponding Trio files.

Notable examples can be found in the point definitions:

"All points must define exactly one of the following pointFunction marker tags"[28] Which is not reflected in the Trio definitions of the type point or pointFunction. The same pattern reoccurs with the kind parameter of a point. Even the currently under development Xeto definitions of points do not reflect the documentation. While kind is indeed mandatory in the Xeto definitions, the pointFunction is not - contradicting the prose text documentation.

Being only one of many examples, the arising situation requires a developer of a haystack model for a building or system to have in-depth knowledge of the haystack standard and its documentation, and will inevitably lead to a broader spectrum of what different applicants of the haystack ontology deem a valid haystack model.

Further, such lapses in lack of machine readable ontology definition lead to inconsistent and even contradictory ontology definitions. For example points connected to weatherStation [29] contradict the definition of point [28] and its mandatory fields.

Missing, or incomplete machine readable definitions prevent automated validation and checking of the ontological definitions and models based on this ontology. Previous investigations have already highlighted this issue [13], [14] in previous haystack versions, however, this problem still persists in Project Haystack 4, and in the current state of Project Haystack 5 (Xeto).

Loose definitions Some haystack types are defined very loosely, complicating type checking and validation. Notable example of this is the enum type, which can be applied to def as well as point, and can contain dict, str, Xeto ref, as values. Furthermore, if the value of an enum is of type string, it can be a comma separated, newline separated, or markdown formatted list according to its prose text documentation [30].

While versions before 3.9.15 defined enum with is:[^str,^dict], newer versions simply define it as its own type using val for which one must read the prose text documentation to determine the allowed data types. Finally, depending on whether enum is applied to a def or a point results in very different handling when modeling a building grid despite being of the same type underneath.

Ambiguous Cardinality The Trio format and the defs are not expressive enough to define cardinality of relationships/references. There is no way to distinguish one-to-one and one-to-many relationships in Project Haystack definitions. For example: The project haystack documentation states that the basic containment relations (siteRef, equipRef, or spaceRef) should always be unary [31], while systemRef is explicitly allowed to contain a list of refs [32]. The Trio source definitions of systemRef and equipRef are both defined as is:[^ref]. In contrast, siteRef is defined as is: ^ref definition. While at first glance, the equipRef seems to be incorrect, allowing for multiple ref despite the documentation allowing only unary ref, the issue is more complex: During normalization all is: ^ref are transformed to is:[^ref]. In the normalized ontology, is:[^ref] is to be interpreted as being either list of ref, or ref. Finally, [] are not only used to imply cardinality, but also to denote a def having multiple supertypes (e.g. battery: is: [^equip, ^elec-output])

Haystack 5 introduces Xeto to strengthen the ontology. For example, the situation described above can now be solved with the Xeto type MultiRef. However, Xeto is still in development, comes with yet another new custom .xeto file format, and corresponding Xeto programming language that itself is implemented in Fantom.

4 Implementation↩︎

In order to create type checking information in the form of JSON and Pydantic schemas the Project Haystack definitions have to be read. From version 3.9.10 on, normalized definitions are included in the Project Haystack releases. Normalized definitions are available in Trio, Zinc, JSON, JSONld and ttl formats. For versions before 3.9.10 only the Trio source files are available. We therefore implement two paths for Pydantic and JSON Schema creation:

• A rudimentary re-implementation of the normalization process described in Project Haystack.

• A transformation of the normalized definitions obtained in JSON format.

The normalization process as defined in Project Haystack
documentation [24]:

1. Scan: traverse input libs to find Trio files

2. Parse: parse each Trio file to discover def and defx dicts

3. Resolve: ensure every symbol tag maps to a def

4. Taxonify: compute taxonomy tree of supertype/subtypes

5. Defx: add defx tags to each declared def

6. Normalize Tags: normalize def tags

7. Inherit: recursively apply supertype tags into subtypes

8. Validation: perform additional sanity checks

9. Generate: output Namespace

Using our implementations we create schemas from both sources (if available) and compare them to ensure that our normalization process and Trio parser implementation are aligned with the normalization process implemented in DefCompiler in the Haxall repository.

Step 1(a) Trio source We implement a Trio parser using python lark [33]. Using this parser we read all Trio files in the Haystack definitions. We only consider tag definitions that are generally applicable or applicable to rows in Haystack model. We apply the def extension (defx) to perform the late binding step, updating the parsed definitions.

Step 1(b) Normalized Json source The normalized Haystack ontology is loaded from a release (3.9.10 and above).

Step 2 Resolve inheritance We calculate the inheritance tree for all defs to deduce the base datatype for each tag. The intermediate results are stored in a JSON file.

Step 3 Filter The resulting definitions are filtered based on tagOn field, origin library and namespace, and the inheritance tree of each definition. Only definitions that are applicable to rows in a Haystack model are kept.

Step 4 Schema creation We implement Pydantic [22] schemas for each base type and created a mapping from Project Haystack types to those schemas. This mapping is applied to the inheritance trees of the filtered defs to create a Pydantic schema of a row in a Haystack model. This row schema now contains all tags/definitions that are allowed to occur in a Haystack models row together with the correct datatype. This row model is wrapped in another schema representing a Haystack grid.

An excerpt from the resulting Pydantic schema:

class HsGridRow(BaseModel):
  id: HsRef
  dis: str = Field(..., title='Dis')
  ahu: Optional[HsMarker] = None
  equip: Optional[HsMarker] = None
  siteRef: Optional[Union[HsRef, HsRefList]] = Field(None, title='Siteref')
  geoAddr: Optional[str] = Field(None)
  geoCity: Optional[str] = Field(None)
  site: Optional[HsMarker] = None
  area: Optional[Union[HsNumber, float]] = Field(None, title='Area')
  ...

This Pydantic schema can now be used to type check a Haystack model in the form or a JSON formatted Haystack grid. An exemplary application:

with open(path) as fd:
    grid = json.load(fd)
grid = HsGrid.model_validate(grid)

Pydantic will try to apply the schema to the provided data, if type checking fails, a detailed error message will be provided. If model validation succeeds, a Pydantic model is returned. This Pydantic model behaves like a data-class in python. For example, a call like grid.rows[0].ahu will either return a marker if the first row in the grid has this marker set, or pythons None otherwise.

Aside from type checking, using the Pydantic model instead of the python representation of the JSON data structure enables auto-completion capabilities of the IDE, as well as continuous type checking. Setting or accessing an undefined tag will result in an error. Pydantic models can be serialized back to JSON with ease.

Furthermore, Pydantic allows for serializing the schemas themselves to produce JSON Schema, enabling type checking using JSON Schema, or integration of the schema into many other programming languages and services aside from python.

5 Conclusion↩︎

Project Haystack has emerged as a practical and well adopted ontology within the building automation industry, primarily due to its intuitive tagging-based semantic model, open design, and high degree of flexibility and extensibility [2]. Underneath, project haystack consists of a semantic tagging model and filter language for building-related data [2], [10]

However, despite its strengths, the ecosystem’s close coupling with the Haxall framework and Fantom programming language results in integration challenges. In this work, we address this limitation by introducing a Python-based toolbox that enables users to perform data type checking and partial validation of Project Haystack models independently of the Haxall ecosystem and Fantom programming language. Our approach includes the generation of both JSON Schemas and Pydantic models, allowing developers to seamlessly process, validate, and interact with Haystack data across multiple programming environments.

To achieve this, we implemented a Python parser for the Haystack specific Trio format, re-engineered the normalization and def compilation processes to create an operational Project Haystack ontology, and developed a transformation pipeline producing the corresponding Pydantic and JSON schemas. All code, along with the generated artifacts, is freely available on GitHub³ to promote openness, transparency, and community collaboration.

Future work: We plan to release our implementation as an installable Python package on PyPI, making it easier for developers to access and apply the Haystack schemas in their projects. Further research will focus on the automatic generation of Pydantic validator rules [34] derived from the machine-readable elements of the Haystack definitions. Additionally, we will develop a dedicated parser for the Xeto format to support automated rule creation and explore the use of SHACL [35]–[37] as a formal mechanism for validating both normalized RDF releases and Project Haystack model instances.

Acknowledgments↩︎

The research leading to these results was conducted within the project Self2B (project number 920143), funded by the Austrian Research Promotion Agency (FFG).

References↩︎