PriEval-Protect: A Unified Framework for Privacy Evaluation and Protection in Healthcare Systems
July 15, 2026
Abstract
Safeguarding patient privacy while enabling meaningful healthcare data use remains critical under GDPR and HIPAA. Existing compliance methods are manual, error-prone, and separate policy audits from data-level assessments. This paper presents PriEval-Protect, a two-phase framework for unified privacy risk evaluation and mitigation. The evaluation phase combines regulatory compliance scoring using a fine-tuned legal LLM with RAG, and technical analysis via encryption type, data architecture, and metrics including similarity, uncertainty, adversary success, and information gain/loss. A composite risk score uses weighted aggregation via Analytic Hierarchy Process. The protection phase recommends countermeasures including federated learning and differential privacy based on assessed risk. Results on hospital documents and datasets demonstrate regulation-aligned, explainable assessments, bridging legal conformance and data-level risk analysis.
1 Introduction↩︎
Healthcare systems increasingly rely on data-driven infrastructure for clinical decisions and research, yet must operate within strict regulatory boundaries like GDPR1 and HIPAA2. Current privacy evaluation remains fragmented and under-automated, with tools emphasizing either manual policy audits or isolated technical metrics without meaningful integration.
Regulatory frameworks like Digital Health Assessment Framework3 and HITRUST CSF4 offer structured evaluations but lack technical integration. Technical methods like Differential Privacy (DP)—adopted by Apple5—provide mathematical guarantees, while Federated Learning (FL) via frameworks like PySyft6, Flower7, and IBM FL8 enables decentralized training. However, these solutions fail to unify policy compliance with technical mitigation.
1.0.0.1 Our Contributions.
Unified Two-Phase Framework: Combines (i) evaluation via regulatory scoring (fine-tuned legal LLM with RAG) and technical risk assessment (encryption, architecture, similarity, uncertainty, adversary success, information gain/loss), and (ii) protection recommending context-specific countermeasures (masking, Federated Learning FL, Differential Privacy DP) based on risk level.
Theoretical Integration: Comprehensive privacy taxonomy integrating regulatory, architectural, and technical dimensions with Multi-Criteria Decision Making (MCDM) based risk aggregation using Analytic hierarchy process (AHP)[1], [2].
Empirical Validation: Evaluated on hospital policy documents and medical datasets, demonstrating regulation-aligned compliance scores and tailored protection strategies.
2 Related Work↩︎
Prior research divides into: (i) regulatory compliance, (ii) privacy evaluation (attribute classification and metrics), and (iii) protection techniques.
Regulatory compliance efforts like the Benjumea Privacy Scale [3] provide structured GDPR scoring via 14 weighted items validated by experts, but remain manual and error-prone. Privacy evaluation methods from Wagner and Eckhoff [4] categorize 80+ metrics (uncertainty, information gain/loss, adversary success), yet lack healthcare-specific calibration. Tools like Pycanon [5] implement limited metrics requiring extensive preprocessing.
Attribute classification is essential for privacy metrics. Simões [6] explored autoencoders, custom Deep Neural Networks (DNNs), and LLMs for Quasi-Identifiers/Sensitive Attributes (QID/SA) detection, showing promise but suffering from computational cost and interpretability issues. Two-step rule-based algorithms [7] offer superior performance, transparency, and healthcare adaptability.
Protection techniques like FL [8] and DP [9] provide strong guarantees but lack context-aware application. Table 1 summarizes existing approaches, highlighting gaps in integration between compliance, metrics, and adaptive protection that PriEval-Protect addresses.
| LLM-based [6] | FL, DP [8], [9] | |
|---|---|---|
| Attribute Classification | Protection Techniques | |
| Context-aware | Strong guarantees | |
| Prompt-sensitive | Not adaptive |
3 Background and Preliminaries↩︎
3.0.0.1 Legal Frameworks.
GDPR establishes EU data protection via principles like lawfulness, transparency (Art.5-6,13), purpose limitation, minimization, security (Art.32), impact assessments (Art.35), and breach notification (Art.33-34) [10]. HIPAA governs U.S. PHI privacy through technical, administrative, and physical safeguards [11].
3.0.0.2 Data Classifications.
Healthcare datasets contain: Quasi-Identifiers (QIDs)—indirect identifiers (age, ZIP); Sensitive Attributes (SAs)—directly sensitive fields (disease, income); Non-Sensitive Attributes (NSAs)—negligible risk attributes [12], [13].
3.0.0.3 Privacy Metrics.
Quantify leakage risk [4]: Data Similarity (S)—k-anonymity, l-diversity; Uncertainty (U)—entropy of sensitive attributes; Adversary Success (\(P_{\mathrm{adv}}\))—re-identification probability; Information Gain/Loss (IG/IL)—mutual information or KL divergence changes.
3.0.0.4 AI Components.
LLMs are transformer-based models for language understanding and generation [14], supporting compliance verification [15]. RAG merges LLMs with external retrieval for grounded responses [16]. Vector Databases store embeddings for semantic search [17]. Fine-Tuning adapts pre-trained LLMs via domain pretraining, supervised tuning, or parameter-efficient methods like LoRA with quantization [18]–[21].
3.0.0.5 Privacy-Preserving Techniques.
Differential Privacy (DP) quantifies privacy via \(\Pr[\mathcal{M}(D_1) \in S] \leq e^\epsilon \cdot \Pr[\mathcal{M}(D_2) \in S] + \delta\) for neighboring datasets \(D_1, D_2\), injecting calibrated noise [9], [22]. Federated Learning (FL) trains models across institutions without raw data exchange [8], [23], [24].
3.0.0.6 MCDM.
Multi-Criteria Decision Making provides structured frameworks for evaluating conflicting criteria. AHP translates qualitative judgments into quantitative weights via pairwise comparisons [1], [2].
4 Method↩︎
4.1 Framework Overview↩︎
PriEval-Protect integrates regulatory and technical privacy evaluation in two phases. Phase 1 computes composite risk scores from policies and data; Phase 2 recommends tailored countermeasures. Figure 1 shows the pipeline: policy documents and health data are evaluated through legal LLMs and technical metrics, producing scores that guide protection strategies (DP, FL, or combined).
4.2 Phase 1: Privacy Risk Evaluation↩︎
Phase 1 evaluates privacy via structured taxonomy covering regulatory, architectural, and technical factors (Figure 2). Users upload datasets and policy documents, configuring encryption type (AES-256, RSA, hybrid) and data architecture (central, federated, distributed). The system computes privacy metrics and regulatory compliance scores, fused via weighted formula.
4.2.1 Privacy Metrics Engine↩︎
Custom Python implementations compute formal metrics from literature. After attribute classification via two-step rule-based algorithms [7], the module calculates: (1) Adversary Success Rate and Delta Presence, (2) k-Anonymity, alpha-k-Anonymity, l-Diversity, t-Closeness, (3) Information Gain and Conditional Entropy, (4) Shannon and normalized entropy. Multiprocessing optimizes large-scale processing.
4.2.2 Regulatory Compliance Assessment↩︎
Automated GDPR/HIPAA alignment combines fine-tuned LLM with RAG pipeline using Qdrant for semantic retrieval.
4.2.2.1 Dataset and Training.
We curated: 9,000+ labeled GDPR policy segments, violation reports, and article corpus with Q&A expansions. Preprocessing included normalization, article extraction, harmonization, and stratified oversampling (27:1 imbalance). Final corpus: 11,447 instances, 54% compliant, 200+ indexed articles.
Two-stage fine-tuning: (1) Domain-Adaptive Pretraining on GDPR corpus using Saul-7B-Instruct, (2) Supervised Fine-Tuning on labeled examples with LoRA (6.8M parameters), 4-bit quantization, and gradient checkpointing.
4.2.2.2 RAG Pipeline and Scoring.
Structure-aware chunking preserves legal context. Chunks are embedded (BAAI/bge-base-en-v1.5, L2-normalized) and stored in Qdrant. For each policy chunk, top-k GDPR articles are retrieved via cosine similarity. Compliance score: \[\text{Score} = \sum_{i=1}^{k} s_i \cdot (1 - P_i)\] where \(s_i\) is similarity and \(P_i\) is penalty. Retrieved articles prompt the fine-tuned LLM, producing compliant/non-compliant labels with rationales and references (Figure 3).
4.2.3 Score Aggregation↩︎
Final privacy risk via weighted aggregation: \[\text{Final Score} = \alpha \cdot C_{\text{compliance}} + \beta \cdot M_{\text{technical}} + \gamma \cdot E_{\text{encryption}} + \delta \cdot A_{\text{architecture}}\] Weights determined by AHP where \(\alpha + \beta + \gamma + \delta = 1\). Risk categories: Low (80–100%), Moderate (60–79%), High (0–59%).
4.3 Phase 2: Privacy Protection↩︎
Risk-based protection mapping (Table 3): Low risk applies data masking; Moderate uses DP with noise injection; High activates FL+DP for collaborative training with encrypted updates.
| Risk | Technique | Description |
|---|---|---|
| Low | Data Masking | Redacts sensitive fields |
| Moderate | Differential Privacy | Adds calibrated noise |
| High | FL + DP | Decentralized training with noise |
5 Experimental Setup↩︎
To empirically validate the PriEval-Protect framework, we conducted a multi-faceted evaluation combining real institutional data and expert assessments from healthcare organizations. The setup examines three key dimensions: (i) regulatory compliance scoring, (ii) technical privacy metrics computation, and (iii) unified privacy risk classification.
We utilized two categories of authentic institutional materials:
Healthcare Policy Documents: Internal privacy and data-sharing policies from multiple clinical institutions, used as input for the GDPR compliance assessment module. These documents represent diverse policy styles and governance maturity levels.
Clinical Data Samples: Structured, anonymized patient datasets comprising demographic, clinical, and administrative fields. These were processed through our privacy metrics engine to quantify re-identification and inference risks.
For compliance scoring, we used the Benjumea Privacy Scale [3] as the baseline reference, aligning our LLM-based scoring model with established GDPR compliance indicators. For technical privacy metrics, the framework’s computation engine was benchmarked against Pycanon [5] and ARX [25], two widely recognized tools for evaluating anonymization and re-identification risks in data publishing. For global privacy risk classification, multiple privacy and data protection experts—including compliance officers and healthcare information specialists—were asked to manually assign risk levels (Low, Moderate, High) to various dataset scenarios. These labels were used as ground truth to validate the interpretability and accuracy of the aggregated PriEval-Protect risk scores.
| Component | Evaluation Metrics |
|---|---|
| Compliance Scoring | Mean Absolute Error (MAE), Pearson correlation |
| Privacy Metrics | Absolute deviation from PyCanon/ARX |
| Global Risk Score | Weighted agreement score |
6 Results and Discussion↩︎
To evaluate the effectiveness of PriEval-Protect, we performed a cross-component assessment covering compliance, technical, and integrated privacy risk evaluation. The quantitative and qualitative findings are summarized in Table 5.
6.1 GDPR Compliance Assessment↩︎
The compliance scoring module achieved a Mean Absolute Error (MAE) of 1.32 compared to expert-derived Benjumea Privacy Scale judgments, and a Pearson correlation of 0.78, indicating a strong positive linear relationship between predicted and expert scores. These results confirm that the LLM-based analysis reliably captures regulatory intent and contextual interpretation across diverse policy documents, validating its utility for semi-automated GDPR alignment.
6.2 Technical Privacy Metric Benchmark↩︎
The lightweight privacy metric engine demonstrated high consistency with formal tools. Deviations from Pycanon and ARX outputs were 0.6 and 0.4 respectively, indicating accurate computation of privacy measures such as k-anonymity, l-diversity, and t-closeness. This result highlights the framework’s potential for integration into real-time data governance pipelines without compromising analytical rigor.
6.3 Global Privacy Risk Validation↩︎
Comparing system-generated global risk classes to expert annotations yielded an agreement rate of 82.6%, with a recall of 81.2% in high-risk cases—demonstrating that PriEval-Protect effectively identifies sensitive data configurations requiring stronger safeguards. These results suggest that the unified scoring model generalizes well beyond specific datasets or local contexts.
6.4 Key Insights and Limitations↩︎
The experimental outcomes validate PriEval-Protect as a robust framework capable of unifying legal and technical privacy evaluations. While results indicate strong agreement with human expertise, future studies should extend this validation to unstructured modalities (e.g., clinical narratives and imaging data) and broader institutional datasets. The framework’s modular design makes it adaptable to varying compliance regimes, supporting its scalability across international healthcare ecosystems.
| Component | Metric | Value |
|---|---|---|
| Compliance | MAE (vs. BPS) | 1.32 |
| Pearson Correlation | 0.78 | |
| Privacy Metrics | Deviation (PyCanon) | 0.6 |
| Deviation (ARX) | 0.4 | |
| Global Score | Class Agreement | 82.6% |
| High-Risk Recall | 81.2% |
7 Conclusion↩︎
PriEval-Protect advances privacy-aware AI in healthcare by integrating regulatory compliance, technical metrics, and data architecture analysis. It bridges legal, technical, and operational perspectives through LLM-driven scoring, weighted aggregation, and practical protection strategies (encryption, FL, DP). The framework enables automated, explainable privacy governance beyond manual audits. Future directions include extending to additional data types, enhancing automation, and real-world clinical deployment. PriEval-Protect demonstrates that scalable, automated privacy evaluation is achievable and necessary for responsible digital health innovation.
7.0.1 ↩︎
This research was supported by INSAT, Efrei Research Lab, and LARIA Research Lab at ENSI.
7.0.2 ↩︎
The authors have no competing interests to declare.