1 Introduction↩︎

1.1 Background and Motivation↩︎

Maternal and child health (MCH) remains a critical public health challenge in the United States, with persistent disparities in access to information and care. South Carolina, in particular, faces significant challenges, with pregnancy-related mortality reaching 47.2 per 100,000 births in 2021, representing a 46% increase from the previous year, and disproportionately affecting Black women [1]. Broader structural factors across the Southern United States, including provider shortages, rural hospital closures, and systemic inequities, further limit access to timely and reliable health resources [2].

Although a wide range of MCH resources are available through government agencies, healthcare providers, and non-profit organizations, these resources are often fragmented across websites and documents, making them difficult for families to navigate. Traditional directory-based systems require users to formulate queries and interpret complex information structures, which can be especially challenging in urgent or stressful situations.

Recent advances in large language models (LLMs) have enabled conversational interfaces that provide more natural access to information. In healthcare, such systems have been explored for patient support, education, and administrative assistance [3]. However, deploying LLM-based systems in public health contexts introduces significant challenges related to safety, trust, and information quality. LLMs can generate fluent but potentially incorrect or misleading responses, raising concerns about hallucinations, outdated information, and inappropriate medical guidance [4]. These risks are particularly consequential in healthcare settings, where incorrect information may lead to harmful decisions.

1.2 Challenges↩︎

In this work, we focus on three key challenges in designing LLM-based systems for public health information access:

Safety and liability. Healthcare applications require strict safeguards to prevent systems from providing medical advice, diagnoses, or treatment recommendations. Prior studies have shown that automated systems can produce unsafe or inaccurate triage recommendations [5], [6], and clinicians express concern about patient misuse of AI-generated information [7]. Ensuring that LLM-based systems operate within well-defined boundaries is therefore essential.

Trust and credibility. Users are more likely to trust health information when it originates from recognized institutions [8], [9]. However, many conversational AI systems lack clear institutional grounding, making it difficult for users to assess the reliability of the information provided.

Information quality and control. While LLMs can synthesize responses from diverse knowledge sources, they may also introduce hallucinated or unverifiable content. In public health contexts, it is critical to ensure that responses are grounded in vetted, up-to-date resources rather than generated from the model’s pre-trained knowledge.

1.3 Approach and Contributions↩︎

To address these challenges, we present the design and implementation of a safety-constrained LLM-based system for MCH resource navigation. The system is designed to provide conversational access to curated public health resources while enforcing strict boundaries on medical content and maintaining user privacy.

Our approach centers on a multi-layered architecture that combines domain-restricted retrieval-augmented generation (RAG), explicit safety enforcement mechanisms, and comprehensive monitoring infrastructure. Rather than relying on unconstrained language generation, the system grounds responses in a curated knowledge base and enforces source attribution, ensuring that all outputs can be traced to authoritative resources.

The main contributions of this work are:

• A safety-constrained architecture for LLM-based public health systems that integrates input filtering, domain validation, response verification, and audit logging.

• A controlled data pipeline for domain-restricted RAG that limits responses to curated resources and reduces reliance on the model’s pre-trained medical knowledge.

• A privacy-preserving multi-user session framework that supports anonymous interactions without collecting personally identifiable information.

• An end-to-end system implementation and deployment framework, including monitoring and audit logging infrastructure to support evaluation and accountability.

• Scenario-based validation demonstrating consistent enforcement of safety constraints and reliable system performance across diverse query types.

Beyond the specific system, this work provides practical insights into the design trade-offs involved in deploying LLM-based systems in safety-critical domains. These insights are relevant to a broader class of applications where strict information boundaries, transparency, and accountability are required.

The remainder of this paper is organized as follows: Section 2 reviews related work in healthcare chatbots and retrieval-augmented generation. Section 3 presents the system architecture. Section 4 describes the implementation details. Section 5 presents validation and testing results. Section 6 discusses implications and limitations, and Section 7 concludes with future directions.

2 Related Work↩︎

2.1 Healthcare Conversational Systems↩︎

Conversational AI systems have been increasingly applied in healthcare for tasks such as symptom checking, patient education, and mental health support [3], [10]. Commercial platforms, including Babylon Health and Ada Health, provide automated triage and diagnostic guidance through structured or AI-driven interactions. While these systems demonstrate the potential of conversational interfaces, prior evaluations highlight important limitations. Studies report variability in diagnostic accuracy and instances of unsafe or inappropriate recommendations, raising concerns about reliability and clinical risk [5], [6].

Recent research has explored domain-specific conversational systems, including applications in maternal and perinatal health. For example, the “Rosie” chatbot demonstrated feasibility and user engagement in postpartum support contexts [11], and systematic reviews report positive outcomes in user satisfaction and behavior change [12]. However, many existing systems either provide diagnostic-style interactions or operate with limited constraints on generated content. This creates challenges in ensuring safety, particularly in settings where incorrect or misinterpreted information may have significant consequences.

In contrast, our work focuses on resource navigation rather than diagnosis, and emphasizes strict boundary enforcement to prevent the generation of medical advice. This positioning aligns with emerging perspectives that conversational systems in healthcare should complement, rather than replace, clinical decision-making [7].

2.2 Retrieval-Augmented Generation in Healthcare↩︎

Retrieval-augmented generation (RAG) has become a widely adopted approach for improving the factual grounding of language model outputs by combining parametric models with external knowledge sources [13]. In healthcare applications, RAG has demonstrated improved accuracy and reduced hallucination compared to standalone language models. For example, recent systems have shown substantial gains in medical question answering and fact-checking tasks when grounded in curated datasets [14], [15].

Despite these advances, most RAG-based systems are designed to enhance performance rather than enforce strict information boundaries. Retrieved documents are often used to guide generation, but models may still introduce unsupported or synthesized content. In safety-critical domains such as public health, this lack of strict control can be problematic.

Our work builds on RAG by adopting a domain-constrained approach in which responses are explicitly restricted to curated resources, with mandatory source attribution. This design prioritizes controllability and transparency over generative flexibility, reflecting the requirements of public health information systems.

2.3 Safety and Governance in Healthcare AI↩︎

Ensuring safety and accountability in healthcare AI systems has been an active area of research, particularly in clinical decision support and regulatory frameworks. Prior work emphasizes the importance of validation, monitoring, and fail-safe mechanisms in mitigating risks associated with automated decision-making [16]. Concerns about large language models include the generation of plausible but incorrect information and the potential for misuse in self-diagnosis contexts [4], [17].

Guidelines from regulatory and research communities highlight several key principles for safe deployment, including clear scope definition, transparency of information sources, and continuous post-deployment monitoring [16], [18]. However, there remains a gap between these high-level recommendations and concrete system designs that operationalize them in practice.

This work contributes to this gap by presenting an end-to-end system architecture that integrates safety mechanisms directly into the LLM interaction pipeline, including input filtering, domain validation, response verification, and audit logging. Our approach demonstrates how safety and governance considerations can be implemented at the system level for public health applications.

2.4 Summary↩︎

Prior work highlights both the potential and risks of LLM-based systems in healthcare. While conversational interfaces and RAG techniques improve accessibility and grounding, challenges remain in ensuring safety, trust, and controllability. Existing approaches often prioritize performance or user engagement, with less emphasis on strict boundary enforcement and system-level governance.

In this context, our work focuses on the design of a safety-constrained LLM system that prioritizes controlled information access, transparency, and accountability, addressing practical requirements for deployment in public health settings.

3 System Architecture and Design↩︎

The system is designed to provide conversational access to maternal and child health (MCH) resources while enforcing strict safety and information constraints. Our design is guided by four principles: (1) safety-first operation, with layered mechanisms to prevent inappropriate medical guidance; (2) domain-constrained information access, ensuring responses are grounded in curated resources; (3) privacy-preserving interaction, enabling anonymous use without personal data collection; and (4) system transparency, supported through logging and traceability.

3.1 System Overview↩︎

As illustrated in Fig. 1, the system follows a three-tier architecture: System Architecture

User Interface Layer. A lightweight web-based interface provides a conversational entry point for users. The interface is designed for accessibility across mobile and desktop devices and presents responses with explicit source attribution.

Application Layer. A Node.js/Express backend manages request handling, session tracking, safety enforcement, and logging. This layer coordinates all interactions between users and AI services, acting as a control point for enforcing system constraints.

AI Services Layer. The system integrates a large language model (GPT-4o) with a vector-based retrieval system to enable retrieval-augmented generation (RAG). This layer is responsible for generating responses grounded in the curated knowledge base.

This modular design separates user interaction, control logic, and model inference, allowing safety and governance mechanisms to be enforced independently of the underlying model.

3.2 Safety and Boundary Enforcement↩︎

A central design goal is to constrain the system to resource navigation while preventing the generation of medical advice. To achieve this, we implement a multi-layered safety pipeline integrated into the request lifecycle:

• Input filtering and emergency detection. User queries are analyzed for emergency-related keywords and patterns. When triggered, the system bypasses standard processing and returns predefined emergency guidance.

• Domain scope validation. Queries are classified as in-scope or out-of-scope based on relevance to MCH topics. Out-of-scope queries result in controlled fallback responses.

• Constrained response generation. The LLM is prompted to generate responses only from retrieved content, with explicit instructions to avoid diagnostic or treatment-related language.

• Post-generation validation. Responses are checked for compliance with safety rules, including the absence of medical advice and the presence of source attribution.

• Audit logging. All interactions are recorded for monitoring, analysis, and post-deployment evaluation.

This layered approach ensures that safety constraints are enforced at multiple points, reducing reliance on any single mechanism.

3.3 Multi-User Session Management↩︎

The system supports concurrent interactions through a privacy-preserving session framework. Each user is assigned an anonymous session identifier stored on the client side, which is used to maintain conversational context without requiring authentication or collecting personally identifiable information.

On the server side, session identifiers are mapped to conversation threads managed by the AI service. The backend remains stateless with respect to user identity, enabling horizontal scalability and reducing privacy risks. This design demonstrates that conversational continuity can be achieved without persistent user profiles.

3.4 Data Architecture and Domain-Constrained RAG↩︎

The system’s knowledge base consists of a curated collection of MCH resources and medical term definitions, organized within a vector store. Each entry is enriched with structured metadata to support accurate retrieval and categorization.

To enforce strict information control, we adopt a domain-constrained RAG approach in which responses must be grounded in retrieved resources. Unlike general-purpose RAG systems, which may blend retrieved content with model-generated knowledge, our design prioritizes traceability by requiring that responses reference specific sources from the curated dataset.

A key design element is the integration of medical terms as resource-linked entities rather than standalone knowledge. During preprocessing, terms are associated with relevant resources, enabling queries involving medical terminology to resolve to actionable information (e.g., local services) rather than generalized explanations.

3.5 Design Rationale and Trade-offs↩︎

Several design decisions reflect trade-offs between safety, flexibility, and usability. Strict boundary enforcement reduces the risk of inappropriate responses but limits the system’s ability to answer open-ended medical questions. Constraining generation to curated resources improves reliability and transparency, but requires ongoing maintenance of the knowledge base.

Similarly, the decision to support anonymous sessions enhances privacy but limits personalization capabilities. These trade-offs are intentional, reflecting the priorities of public health applications where safety, trust, and accountability outweigh generative flexibility.

Overall, the architecture emphasizes controlled information access and system-level safeguards, providing a foundation for deploying LLM-based systems in safety-critical domains.

4 Implementation↩︎

We implement the system as a modular, cloud-ready application that integrates a web-based interface, a backend control layer, and LLM-based retrieval and generation services. The implementation emphasizes controlled interaction with the language model, traceability of outputs, and compatibility with deployment and monitoring infrastructure.

4.1 System Pipeline↩︎

User queries are processed through a structured pipeline that enforces safety and grounding at each stage. Given an input query, the system executes the following steps:

1. Session handling. The request is associated with an anonymous session identifier and routed through the backend service.

2. Safety preprocessing. The query is analyzed for emergency conditions and domain relevance. Emergency queries trigger predefined responses, while out-of-scope queries are handled through controlled fallback mechanisms.

3. Retrieval. Relevant entries are retrieved from the vector store based on semantic similarity, using embedded representations of curated MCH resources and medical terms.

4. Constrained generation. The retrieved content is provided to the LLM with structured prompts that enforce domain boundaries, prohibit medical advice, and require source attribution.

5. Post-processing and validation. Generated responses are checked for compliance with safety constraints and formatted to include references to the underlying resources.

6. Logging and response delivery. The final response is returned to the user, and the interaction is recorded in the audit log for monitoring and analysis.

This pipeline ensures that all responses are mediated by retrieval and validation steps, reducing the risk of unsupported or unsafe outputs.

4.2 Backend and AI Integration↩︎

The backend is implemented using Node.js and Express, providing an API layer that manages request handling, session mapping, and orchestration of AI services. The system integrates a GPT-4o-based language model through Azure OpenAI services, combined with a vector search component for retrieval.

The language model is configured with conservative generation parameters (e.g., low temperature) to reduce variability and encourage consistent, grounded responses. Prompt templates are designed to explicitly constrain outputs, including instructions to avoid diagnostic or treatment-related content and to rely exclusively on retrieved resources.

4.3 Knowledge Base and Retrieval↩︎

The knowledge base is constructed from a curated set of MCH resources and medical term definitions, embedded and indexed within a vector store. Each entry includes structured metadata to support filtering, categorization, and bilingual access.

During retrieval, the system prioritizes relevance to user queries while maintaining domain constraints. Medical term queries are resolved through their associations with resource entries, enabling the system to provide actionable information rather than isolated definitions.

4.4 Emergency Handling↩︎

Emergency scenarios are handled through a dedicated detection and response mechanism. Queries containing predefined emergency indicators trigger immediate, standardized responses that provide guidance and direct users to appropriate services. This process bypasses LLM generation to ensure consistent and reliable behavior in high-risk situations.

4.5 Monitoring and Audit Logging↩︎

To support transparency and post-deployment evaluation, the system implements structured audit logging for all interactions. Each log entry captures request metadata, system responses, processing time, and error conditions. Logs are stored in a JSON format compatible with centralized monitoring systems (e.g., ELK stack), enabling performance analysis, debugging, and longitudinal studies of system usage.

This monitoring-first design allows the system to be evaluated and audited in real-world deployments, supporting accountability and continuous improvement.

5 Evaluation and Initial Validation↩︎

We conduct an initial validation of the system to assess (1) safety constraint enforcement, (2) response grounding in curated resources, and (3) baseline system performance. Given the early stage of deployment, the evaluation focuses on scenario-based testing designed to reflect representative usage patterns in public health information systems.

5.1 Evaluation Setup↩︎

We construct a set of test scenarios covering three categories of user queries:

• In-scope queries: Requests related to maternal and child health (MCH) resources, such as prenatal care, mental health support, and assistance programs.

• Out-of-scope queries: Queries unrelated to MCH or outside the system’s intended domain.

• Emergency queries: Inputs indicating urgent or potentially life-threatening situations requiring immediate intervention.

These scenarios are designed to evaluate system behavior under typical usage, boundary conditions, and safety-critical situations. Testing is conducted across multiple sessions to validate consistent system behavior.

5.2 Metrics↩︎

The evaluation focuses on metrics aligned with the system’s design goals:

• Safety compliance: Whether responses avoid medical advice and adhere to defined system boundaries.

• Response grounding: Whether outputs are supported by and reference curated resources.

• System reliability: Successful response generation without errors or failures.

• Response latency: Time required to generate responses.

5.3 Results↩︎

Table 1 summarizes the results across the evaluated scenarios.

Across all evaluated scenarios, the system consistently enforced safety constraints, with no observed instances of medical advice generation or boundary violations. Emergency queries reliably triggered predefined response protocols, bypassing LLM generation and providing immediate guidance.

For in-scope queries, responses were grounded in the curated resource database and included source attribution. Out-of-scope queries were handled through controlled fallback responses, maintaining domain boundaries without generating unsupported content.

From a performance perspective, the system demonstrated stable operation, with an average response time of 5.3 seconds and no observed system errors during testing. Multi-user session handling functioned as expected, maintaining isolation across concurrent interactions.

5.4 Observations↩︎

The evaluation highlights several practical observations. First, layered safety mechanisms were effective in preventing unsafe outputs across diverse query types. Second, grounding responses in curated resources improved transparency and traceability. Third, separating emergency handling from the LLM pipeline ensured consistent and low-latency behavior in critical scenarios.

5.5 Limitations↩︎

This evaluation represents an initial validation and does not constitute a large-scale user study. The number of test scenarios is limited, and the evaluation does not capture long-term usage patterns, user satisfaction, or real-world behavioral variability. Future work will include expanded testing with larger query sets, expert review, and longitudinal analysis of deployed system usage.

5.6 Summary↩︎

These results demonstrate that the system can reliably enforce safety constraints, provide grounded responses, and operate with stable performance under representative conditions. While preliminary, this validation supports the feasibility of deploying safety-constrained LLM systems for public health information access and motivates further large-scale evaluation.

6 Design Insights and Lessons Learned↩︎

This work focuses on the practical challenges of deploying LLM-based systems in safety-critical public health contexts. Beyond the system itself, we highlight several design insights that emerged during development and initial validation.

6.1 Safety Requires Multi-Layered Enforcement↩︎

A key insight is that safety cannot be achieved through a single mechanism, such as prompt design or retrieval grounding alone. Instead, effective safety requires layered enforcement across the entire system pipeline, including input filtering, domain validation, constrained generation, and post-generation checks.

In practice, we found that relying solely on prompt-level constraints is insufficient, as language models may still produce undesirable outputs under edge cases. Combining multiple safeguards provides redundancy and improves robustness, particularly in scenarios involving ambiguous or adversarial queries.

6.2 Constrained RAG Improves Control but Limits Flexibility↩︎

Restricting responses to curated resources significantly improves traceability and reduces the risk of hallucinated or unsupported content. This approach aligns well with the requirements of public health systems, where accountability and source attribution are critical.

However, this design introduces a trade-off: strict grounding can limit the system’s ability to handle open-ended or complex user queries. In some cases, the system must decline to answer rather than provide incomplete or potentially misleading information. This highlights a fundamental tension between flexibility and safety in LLM-based systems.

6.3 Separation of Emergency Handling is Critical↩︎

We found that emergency scenarios require fundamentally different handling compared to standard queries. Routing such cases through the LLM introduces unnecessary variability and latency. By separating emergency detection and using predefined response protocols, the system achieves consistent and immediate behavior in high-risk situations.

This design pattern may be broadly applicable to other safety-critical applications, where certain classes of inputs should bypass generative components entirely.

6.4 Privacy-Preserving Interaction is Feasible but Constraining↩︎

The use of anonymous session management demonstrates that conversational continuity can be maintained without collecting personally identifiable information. This approach reduces privacy risks and simplifies compliance considerations.

At the same time, the absence of user profiles limits personalization and long-term adaptation. This trade-off reflects a broader design choice: prioritizing privacy and minimal data collection over individualized user experiences.

6.5 Monitoring is Essential for Real-World Deployment↩︎

A monitoring-first design, including comprehensive audit logging, proved essential for validating system behavior and supporting future evaluation. In safety-critical domains, deployment should be accompanied by mechanisms for tracking performance, identifying failure modes, and enabling post hoc analysis.

This insight underscores that system design should account not only for initial functionality but also for ongoing governance and accountability.

6.6 Limitations↩︎

This study presents initial functional validation rather than large-scale or longitudinal evaluation. The current system relies on a curated and manually maintained resource database, which requires ongoing updates to remain accurate and comprehensive. Additionally, the system is currently limited to a specific geographic region and primarily English-language interaction, although it supports bilingual resources.

6.7 Implications for LLM-Based Public Health Systems↩︎

The findings of this work suggest that deploying LLM-based systems in public health contexts requires prioritizing control, transparency, and accountability over generative flexibility. Design patterns such as domain-constrained retrieval, layered safety enforcement, and explicit monitoring infrastructure provide a practical foundation for achieving these goals.

More broadly, these insights extend to other safety-critical domains where incorrect or unverified information may have significant consequences. Future systems should consider integrating these principles at the architectural level rather than relying solely on model-level improvements.

7 Conclusion and Future Work↩︎

This work presents the design and implementation of a safety-constrained LLM-based system for public health information access, with a focus on maternal and child health resource navigation. The system demonstrates how conversational interfaces can be combined with domain-constrained retrieval, layered safety mechanisms, and privacy-preserving session management to support controlled and accountable use of large language models in healthcare-related contexts.

Rather than emphasizing model performance alone, this work highlights the importance of system-level design in managing the risks associated with LLM deployment. By enforcing strict boundaries on medical content, grounding responses in curated resources, and integrating monitoring and audit logging, the proposed architecture provides a practical approach for deploying LLM-based systems in environments where safety and trust are critical.

The initial validation demonstrates that the system can consistently enforce safety constraints, provide grounded responses, and operate with stable performance. While preliminary, these results suggest that safety-constrained LLM systems are a viable approach for improving access to public health information without relying on uncontrolled generative behavior or personal data collection.

Beyond the specific application, this work contributes design insights into the trade-offs between safety, flexibility, and usability in LLM-based systems. These insights are relevant to a broader class of safety-critical applications that require controlled information access, transparency, and accountability.

Future Work. Future work will focus on expanding both evaluation and system capabilities. A key priority is large-scale validation, including structured testing with expanded query sets, expert review, and longitudinal analysis of real-world usage. In addition, user-centered studies will be conducted to assess usability, trust, and the effectiveness of resource navigation in practice.

On the system side, planned extensions include expanding geographic coverage beyond South Carolina, improving multilingual support, and integrating dynamic data sources to reflect real-time resource availability. Further work will also explore mechanisms for adaptive retrieval and improved query understanding while maintaining strict safety constraints.

As LLM-based systems continue to be adopted in healthcare and other high-stakes domains, approaches that prioritize safety, transparency, and governance at the system level will be essential. This work provides a step toward that goal by demonstrating how such principles can be operationalized in a practical deployment setting.

Code and Data Availability

A live version of the system is available at https://empower-assistant.sc.edu/. Due to the sensitive nature of health-related data and associated security/privacy considerations, the source code is not publicly released. The curated MCH resource database can be accessed through collaboration with the Institute for Families in Society at the University of South Carolina. The code may be made available by the corresponding author upon reasonable request.

References↩︎