Introduction

Healthcare decision-making is a very critical and complex process, especially in ICU settings where multiple physiological systems simultaneously decline. Sepsis is a life-threatening syndrome of acute organ dysfunction caused by a dysregulated host response to infection. Its management requires treatment coordination across vasopressor titration, fluid restoration, antibiotic supervision, and haemodynamic monitoring, each of which operates on a different timescale and involves clinical trade-offs that interact across organ systems. Therefore, the need for intelligent decision support that can navigate this complexity in real time has become one of the most demanding problems in critical care informatics. Reinforcement-learning-based clinical decision support systems have been proposed as a solution for learning optimal treatment policies directly from patient data [1]. However, the current generation of such systems applies a one-stage methodology that chooses all treatment decisions through a single flat action space and focuses on short-term reward signals instead of the multi-level or multi-phase structure of clinical care. Just as each stage of healthcare decision-making is shaped by the stages that are before it, sepsis management requires a hierarchy of decisions, starting from high-level strategic goals to specific drug doses, that cannot be adequately captured by any flat model.

A comprehensive systematic review of 42 studies identified four areas including: the convergence of hierarchical multi-agent reinforcement learning (HMARL), ontology-based medical knowledge representation, and generative AI as the defining edge of next-generation clinical decision support systems [2]. In practice, this convergence remains largely unfulfilled. Only 10% of the reviewed RL studies have used full hierarchical agent architectures so far, only 6% have incorporated the explainability mechanisms, and only 5% among them achieved any knowledge flow between learning agents and clinical knowledge bases. None of the reviewed works has integrated together all three components into a single framework as a clinically validated system.

These are not just the technical gaps. Without the appropriate hierarchical architectures, treatment decisions across organ systems lead towards flat RL models that cannot be represented as the coordinated reasoning based clinical care systems that are actually required. Even a very comprehensive recommendation cannot be justified without explainability, thus making effective deployment in any accountable clinical setting nearly impossible [3].

The same study observed that all the existing systems “frequently rely on statistical associations or post-hoc explanations, requiring semantically based interpretations aligned with clinical logic,” which “reduces transparency and clinician trust in AI-assisted recommendations” [2].

This study introduces HM-CDSS, a Hierarchical Multi-Agent Clinical Decision Support System for sepsis management in ICU settings. The framework uses real patient data from the Medical Information Mart for Intensive Care IV (MIMIC-IV) database 5,282 adult ICU sepsis admissions obtaining approximately 78,200 clinical trajectory transitions. Phase 1 constructs patient trajectories from raw electronic health records using a reward function based on clinical outcomes and a temporal train-test split that prevents patient data from leaking across partitions. Phase 2 trains and evaluates a single-agent Dueling Deep Q-Network (DQN) baseline, establishing the policy-collapse problem that the proposed hierarchical architecture is designed to overcome. In Phase 3, the baseline is extended into a hierarchical multi-agent framework in which a meta-agent assigns high-level clinical goals to four specialist sub-agents, each trained with Conservative Q-Learning (CQL) to prevent Q-value overestimation on offline data. In Phase 4, the trained policy is augmented with a retrieval-augmented reasoning layer that produces structured, guideline-based chain-of-thought justifications for every treatment recommendation.

The methodological contributions of this study are fourfold. First, a conservative offline-RL hierarchical MARL architecture is introduced that applies CQL with agent-specific regularisation weights to reduce distributional shift, a primary failure mode of standard offline RL in clinical settings. Second, a retrieval-augmented clinical reasoning layer is proposed and evaluated through a five-metric framework (GGS, HR, CCS, RC, UC) for explanation quality in clinical MARL; to the best of our knowledge, no prior work has proposed a comparable quantitative evaluation suite for this setting. Third, a three-condition ablation study isolates and quantifies the independent contribution of each architectural component demonstrating that CQL adds 21.8% points of clinician agreement while the hierarchy contributes architectural modularity and interpretability rather than a measurable agreement gain on the current dataset providing reproducible evidence for each design decision.

Fourth, the generalized five-metric reasoning-evaluation framework is offered as a benchmark applicable beyond sepsis to any offline MARL clinical decision support setting.

Four research objectives guided this work:

  • RO1: To develop and evaluate a hierarchical multi-agent CQL framework that produces clinically aligned offline sepsis treatment policies, and to isolate the independent contribution of each architectural component through systematic ablation.

  • RO2: To demonstrate that CQL with agent-specific regularisation weights addresses conservative policy collapse in offline clinical RL, and to quantify how regularisation weight α affects the policy diversity–distributional constraint trade-off across treatment domains.

  • RO3: To design and evaluate a retrieval-augmented guideline-based chain-of-thought reasoning layer through a novel five-metric framework, and to determine whether context-sensitive retrieval outperforms fixed-citation baselines on clinical consistency and hallucination prevention.

  • RO4: To examine whether meta-agent goal conditioning produces a temporally coherent goal distribution consistent with established sepsis-management priorities across the phases of ICU care.

The remaining paper is organised as follows. "Related Work" reviews related work. "Methodology" describes the methodology. "Results" presents the results. "Discussion"" discusses findings, maps results to research objectives, addresses limitations, and outlines future directions. "Conclusion concludes.

Related Work

This section briefly reviews the state-of-the-art studies conducted across three technological dimensions relevant to this study: reinforcement learning in clinical decision support, medical knowledge representation, and generative AI for clinical explainability. For each dimension, the specific gap being addressed by the present work is mentioned.

Reinforcement Learning for Clinical Decision Support

Komorowski et al. [4] established a reinforcement learning based system fir for sepsis treatment optimisation with their AI Clinician system, trained on the vasopressor and fluid dosing-based policies from MIMIC-III data using discrete Q-learning method. This work set the pattern for later research: an offline trajectory-based training on retrospective data, discretised treatment action spaces, and reward functions based on patient-outcome. Raghu et al. [5] used the deep RL to the similar kind of problem and concluded that the learned policies could diverge from common clinical practice in multiple ways that appeared to be useful. Liu et al. [6] confirmed, in a broad review, that RL for critical care had achieved certain maturity as a research direction, though it still remained tied to the single-organ or flat action-space formulations.

A limitation that was shared through this body of work is the assumption of flat action space where a single agent encodes all the treatment decisions together. This is computationally very convenient but equally unrealistic clinically: vasopressor titration, fluid administration, and antibiotic timing are not optimised by just one entity in an actual ICU. Silveira et al. [2] confirm that flat agent structures dominate the field, noting that this restricts both the “scalability and context-aware decision-making” in multi-organ disease settings.

Single-Agent Foundations

Multi-Agent and Hierarchical Extensions

MARL has addressed this problem of flat-agent by distributing the treatment decisions across certain specialised agents. Shaik et al. [7] proposed a cooperative MARL framework for physiological monitoring that improved coordination among the domains of vital-sign. Hao et al. [8] applied hierarchical MARL to medical resource allocation under incomplete disease information and showed that hierarchical decomposition improved policy convergence. Tan et al. [9] presented a hierarchical MARL system with organ-specific, state-aware agents for sepsis treatment, achieving stronger alignment with clinical practice than single-agent baselines.

Despite this progress, Silveira et al. [2] report that only 10% of RL studies in healthcare use full hierarchical multi-agent architectures, and that none address the offline distribution-shift problem. A learning based on standard temporal-difference using fixed offline datasets systematically overestimates all the Q-values for actions reduced in the behaviour policy that is formalised by Fujimoto et al. [10] as extrapolation error, adopted by Batch-Constrained Q-learning (BCQ). Kumar et al. [11] subsequently proposed Conservative Q-Learning (CQL), which achieves distributional conservatism without behaviour-cloning constraints, making it well-suited to multi-agent offline settings. Gottesman et al. [12] explicitly called for healthcare RL guidelines addressing this distributional problem, which remains largely unresolved across the reviewed literature. The present study applies CQL across all five agents in the hierarchical system, with agent-specific α values calibrated to clinical risk.

Medical Knowledge Representation

Medical ontologies like SNOMED CT, LOINC, UMLS, and ICD have been used to ground RL-based CDSSs in verified clinical knowledge. Bernabe et al. [13] reviewed foundational ontologies in biomedical research and demonstrated their value for interoperability and reasoning across heterogeneous clinical data. Jing et al. [14] found that ontological grounding substantially improved interpretability in CDSS rule systems. Seneviratne et al. [15] argued specifically for bidirectional integration, in which agent-derived insights dynamically update the knowledge base, as essential for adaptive clinical systems. Silveira et al. [2] quantify the current state of integration: only 26% of knowledge-based systems implement bidirectional flow, and only 5% of all reviewed studies achieve genuine bidirectional knowledge exchange, with most treating ontologies as static repositories. Recent neuro-symbolic approaches combining neural learning with structured knowledge have shown promising results: He et al. [16] demonstrated concept-level reasoning trails in Alzheimer’s disease diagnosis, Lu et al. [17] presented explainable diagnosis prediction through neuro-symbolic integration, and Garcia-Barragan et al. [18] developed a neuro-symbolic system for oncologic named-entity recognition and linking. In HM-CDSS, an agreement-weighted guideline-relevance update is included as a lightweight step toward bidirectional knowledge integration; however, the RAG layer in the current implementation operates post-hoc at inference time and does not feed retrieved knowledge back into RL training. Full bidirectional integration where retrieved guidelines influence policy learning is deferred to future work.

Generative AI and Explainability

Large language models have demonstrated substantial clinical knowledge. Singhal et al. [19] demonstrated competitive performance on medical licensing examinations, and Khullar et al. [20] described the path towards explainable clinical AI using LLMs with structured knowledge integration. LLM-based multi-agent RL is a promising but still underdeveloped direction. Sun et al. [21] found that combining natural-language reasoning with policy learning is beneficial, and Zuo et al. [22] proposed KG4Diagnosis, a hierarchical multi-agent LLM framework with knowledge-graph enhancement for medical diagnosis, showing that knowledge-grounded reasoning outperformed ungrounded generation. Silveira et al. [2] classify LLM integration in CDSS across peripheral, functional, and adaptive levels, finding that adaptive integration remains rare. Ghassemi et al. [23] and Reddy et al. [24] both caution that current explainable-AI approaches in healthcare often offer superficial justification rather than clinically grounded reasoning — the gap that motivates the guideline-grounded, retrieval-augmented design used here.

Methodology

Data and Cohort Construction

Patient data were extracted from MIMIC-IV v2.2 using Google BigQuery. Sepsis was identified using Sepsis-3 criteria [25]: suspected infection combined with an acute SOFA score increase of two or more points. Inclusion required adult ICU admission (age 18 or older), first ICU stay per hospitalisation, and a minimum stay of four hours. After filtering, the cohort comprised 5,282 unique ICU admissions, yielding approximately 78,200 transition tuples, split temporally into training (3,697 patients), validation (792), and test (793) sets. Admissions were sorted by date and the earliest assigned to training, preventing leakage from future clinical practice patterns.

Feature extraction and preprocessing

Observations were binned into 4-hour intervals, yielding 8 to 18 timesteps per episode (mean 14.8). Vital signs (heart rate, mean arterial pressure, SpO₂, temperature, respiratory rate, Glasgow Coma Scale) were averaged within each bin; laboratory values (lactate, white blood cell count, creatinine, bilirubin, platelet count, PaO₂, sodium, potassium) were taken as the last observation before the bin end; and drug administration (vasopressor dose, fluid volume, antibiotic timing) was aggregated per bin. Vasopressor and fluid doses were each discretised into five clinically meaningful bins (V0–V4 and F0–F4), forming a 5 × 5 joint action space for the baseline agent.

Figure 1 shows the overall five-part workflow: data foundation, hierarchical multi-agent architecture, the longitudinal decision environment, the offline training strategy, and the LLM-based reasoning module.

Fig. 1
The workflow has five phases:1. Data foundation: Clinical data are extracted through BigQuery to create time-aware states, with SOFA and mortality used for a mortality reward.2. Hierarchical multi-agent architecture: A meta-agent coordinates specialist agents, including wild-type and disease categories, using QMIX or MAPPO.3. Longitudinal decision environment: SepsisCUEnv-step exchanges state features and trajectories with treatment decisions, with reward transfer. Treatments include antibiotics, fluids, vasopressors, and monitoring.4. Training strategy: Offline reinforcement learning uses MIMIC-IV ICU sepsis data, a temporal training, validation, and test split, and guideline grounding via retrieval-augmented generation.5. LLM reasoning module: An LLM-augmented rationale and a clinical reasoning layer connect to CTDE. A clinical decision specialist provides feedback to the LLM.
Full size image

Proposed multi-phased approach for the Hierarchical Multi-Agent Reinforcement Learning framework for sepsis decision support

Reward Function

The per-step reward combines three physiologically motivated components, as expressed in Eq. (1).

$$\:{r}_{t}=\text{clip}\left({\Delta\:}\text{S}\text{O}\text{F}\text{A}\times\:1.0+{\Delta\:}\text{L}\text{a}\text{c}\text{t}\text{a}\text{t}\text{e}\times\:0.5+{r}_{\text{M}\text{A}\text{P}},-5.0,+5.0\right)$$
(1)

Here, ΔSOFA = SOFA_t − SOFA_(t + 1) is positive when organ function improves, and ΔLactate = Lactate_t − Lactate_(t + 1) is positive when lactate clearance occurs. The MAP reward component is

$$\:{r}_{\text{M}\text{A}\text{P}}=\left\{\begin{array}{c}+0.3,\text{i}\text{f}\text{}65\le\:MAP\le\:85\:mmHg,\\\:-0.3,\text{o}\text{t}\text{h}\text{e}\text{r}\text{w}\text{i}\text{s}\text{e}\text{.}\end{array}\right.$$
(2)

A small step penalty (− 0.05) discourages unnecessarily long trajectories. The reward formulation encodes three clinical priorities: organ-function improvement (via ΔSOFA), metabolic recovery (via ΔLactate), and haemodynamic adequacy (via MAP), consistent with Sepsis-3 outcome criteria [25]. Terminal rewards of ± 15.0 were chosen to dominate per-step signals and anchor the policy towards survival outcomes; sensitivity of the policy to alternative terminal reward magnitudes is acknowledged as a limitation and is prioritised for future analysis. Terminal rewards are assigned according to patient outcome and final SOFA score, as summarized in Table 1. Full training hyperparameters shared across all phases are listed in Table 2.

Table 1 Terminal reward assignment based on patient outcomes and final SOFA score
Full size table
Table 2 Full hyperparameter specification shared across all phases unless otherwise noted
Full size table

Single-Agent Offline RL Baseline

A Dueling Deep Q-Network (DQN) [26] was trained in an offline replay environment to document the policy-collapse problem that motivates the hierarchical extension. The environment replays fixed patient trajectories: the agent’s selected action is accepted by the interface but does not alter the trajectory sequence, which is the standard offline batch-RL protocol used across MIMIC-derived clinical RL studies [27]. The single-agent baseline acts jointly over the 5 × 5 vasopressor–fluid action grid, and its evaluation results are listed in Table 4.

Hierarchical Multi-Agent Framework with CQL

The proposed HMARL system replaces the flat joint action space with a two-level hierarchy. The upper level is a meta-agent that observes the full patient state and outputs a discrete clinical goal. The lower level comprises four specialist sub-agents, each observing a domain-specific state subset concatenated with the meta-agent’s goal signal and independently selecting domain actions. All five agents are trained with Conservative Q-Learning (CQL) [11], which augments the standard Huber TD loss with a penalty on Q-values for out-of-distribution actions, as expressed in Eq. (3):

$$\:{L}_{\text{C}\text{Q}\text{L}}={L}_{\text{D}\text{Q}\text{N}}+\alpha\:\:\text{log}\sum\:_{a}\text{exp}Q\left(s,a\right)-Q\left(s,{a}_{\text{d}\text{a}\text{t}\text{a}}\right)$$
(3)

Here, a_data denotes the clinician’s observed action, while α controls the strength of the conservatism penalty. A higher value of α more aggressively suppresses Q-values for actions absent from the training data. We set α = 1.0 for the vasopressor and fluid agents, where deviation from clinical practice carries the greatest risk, and α = 0.5 for the meta, antibiotic, and monitoring agents, whose proxy action labels carry inherently higher uncertainty.

Specialist Sub-Agents

Table 3 specifies each specialist agent’s observation space, action space, and CQL regularisation weight. The vasopressor and fluid agents act over the same five treatment bins (V0–V4, F0–F4) defined in "Feature Extraction and Preprocessing". The antibiotic agent acts over three levels (no change, escalate, de-escalate), using a proxy action label derived from the vasopressor bin as a surrogate for antibiotic-timing data not yet extracted from MIMIC-IV. This is a significant limitation: the antibiotic agent does not represent a genuinely validated clinical decision domain in the current implementation, and claims about a “four-domain” decision-support system should be understood as a proof-of-concept architectural target rather than a validated capability. This limitation is discussed in "Future Directions". The monitoring agent outputs three alert levels (stable, watch, escalate), similarly using a proxy. Both proxy agents are architecturally complete and contribute to the reasoning-layer output.

Table 3 HMARL specialist agent specifications. Observation dimension includes the four-dimensional goal one-hot vector from the meta-agent
Full size table

Reward Decomposition

To enable per-agent credit assignment, the shared global reward is decomposed as follows. The vasopressor agent receives 0.4 times the shared reward plus a clinician-agreement bonus of + 0.3 when its action matches the clinician’s recorded action, or − 0.1 otherwise:

$$\:{r}_{\text{v}\text{a}\text{s}\text{o}}=0.4{r}_{\text{s}\text{h}\text{a}\text{r}\text{e}\text{d}}+\left\{\begin{array}{c}+0.3,\text{i}\text{f}\text{}{a}_{\text{v}\text{a}\text{s}\text{o}}={a}_{\text{c}\text{l}\text{i}\text{n}\text{i}\text{c}\text{i}\text{a}\text{n}},\\\:-0.1,\text{o}\text{t}\text{h}\text{e}\text{r}\text{w}\text{i}\text{s}\text{e}\text{.}\end{array}\right.$$
(4)

The fluid agent receives 0.3 times the shared reward with the same agreement-bonus structure:

$$\:{r}_{\text{f}\text{l}\text{u}\text{i}\text{d}}=0.3{r}_{\text{s}\text{h}\text{a}\text{r}\text{e}\text{d}}+\left\{\begin{array}{c}+0.3,\text{i}\text{f}\text{}{a}_{\text{f}\text{l}\text{u}\text{i}\text{d}}={a}_{\text{c}\text{l}\text{i}\text{n}\text{i}\text{c}\text{i}\text{a}\text{n}},\\\:-0.1,\text{o}\text{t}\text{h}\text{e}\text{r}\text{w}\text{i}\text{s}\text{e}\text{.}\end{array}\right.$$
(5)

The antibiotic and monitoring agents each receive 0.15 times the shared reward:

$$\:{r}_{\text{a}\text{n}\text{t}\text{i}\text{b}\text{i}\text{o}\text{t}\text{i}\text{c}}={r}_{\text{m}\text{o}\text{n}\text{i}\text{t}\text{o}\text{r}}=0.15{r}_{\text{s}\text{h}\text{a}\text{r}\text{e}\text{d}}.$$
(6)

The meta-agent receives the complete shared reward plus 0.1 times the mean of the fluid and vasopressor rewards, encouraging goal selection that improves lower-level agent performance:

$$\:{r}_{\text{m}\text{e}\text{t}\text{a}}={r}_{\text{s}\text{h}\text{a}\text{r}\text{e}\text{d}}+0.1\text{mean}\left({r}_{\text{f}\text{l}\text{u}\text{i}\text{d}},{r}_{\text{v}\text{a}\text{s}\text{o}}\right).$$
(7)

Meta-Agent and Goal Conditioning

The meta-agent observes the full state vector and outputs one of four discrete goals: Goal 0 for haemodynamic stabilisation, Goal 1 for fluid management, Goal 2 for infection control, and Goal 3 for monitoring and de-escalation. The selected goal is one-hot encoded in a four-dimensional vector and concatenated with each sub-agent’s domain observation. This allows explicit top-down coordination without directly constraining sub-agent action selection. This design follows the hierarchical RL options framework [28], allowing the meta-agent to modulate the clinical context of specialist decisions while each sub-agent retains its own Q-network and policy.

Ablation Study Design

To isolate the contribution of each architectural component, three ablations are carried out under identical conditions (seed 42, same replay buffer, 200 episodes). Figure 2 summarises the design.

Fig. 2
Three ablations modify the full HMARL plus CQL model:- Study A removes the hierarchy: flat CQL agents operate on the complete state without goal conditioning. It tests the contribution of the hierarchical structure.- Study B removes CQL while preserving the hierarchy; specialists use vanilla Dueling DQN. It tests CQL’s contribution beyond hierarchical decomposition.- Study C retains full HMARL plus CQL but gives all agents the full global state instead of domain-specific sub-states. It tests the value of specialized observations.All studies use seed 42, the same replay buffer, and 200 episodes.
Full size image

Ablation study design

  • Ablation A (no hierarchy): flat CQL agents operate directly on the complete state, with no goal conditioning. Tests the contribution of the hierarchical structure.

  • Ablation B (no CQL): hierarchy is preserved, but specialists use a vanilla Dueling DQN, with the CQL term removed. Tests the independent contribution of CQL beyond hierarchical decomposition.

  • Ablation C (no sub-states): the full HMARL + CQL architecture is retained, but all agents receive the full global state instead of domain-specific sub-state partitions. Tests the value of specialised observations.

Retrieval-Augmented Clinical Reasoning Layer

Phase 4 enhances the trained HMARL policy with a retrieval-augmented generation (RAG) reasoning layer that grounds every treatment recommendation in peer-reviewed clinical guidelines and produces verifiable chain-of-thought justifications. The layer operates post-hoc at inference time and has no effect on agent training. Consequently, knowledge flows from the knowledge base to the explanation layer only; no retrieved knowledge is fed back into RL policy training in the current implementation. The lightweight agreement-weighted guideline-relevance update described in "Medical Knowledge Representation represents a preliminary step toward bidirectional knowledge integration.

Knowledge Base

The knowledge base comprises curated excerpts from the Sepsis-3 consensus definitions [25] and the Surviving Sepsis Campaign international guidelines [29], each annotated with an action-relevance field linking the guideline text to specific treatment-action bins.

Dense Retrieval and Chain-of-Thought Generation

Documents are encoded into a 384-dimensional embedding space using the all-MiniLM-L6-v2 sentence-transformer model [30], indexed with FAISS [31] for approximate nearest-neighbour search. At inference, a structured query is built from the proposed action, the meta-agent’s goal name, and threshold-based clinical labels derived from the z-scored state vector. The top three retrieved documents are filtered to ensure at least one whose action-relevance field intersects the proposed action bins. The reasoning module then generates a seven-section output: clinical concern, three sequential reasoning steps, the most relevant guideline citation, a final recommendation, a confidence level (high, medium, or low), and a supporting rationale.

Five-Metric Reasoning Evaluation Framework

Reasoning quality is quantified using five metrics:

  • Guideline Grounding Score (GGS): the fraction of cited guidelines whose action-relevance field is consistent with the recommended action bins.

  • Hallucination Rate (HR): the fraction of citations referring to identifiers absent from the knowledge base.

  • Clinical Consistency Score (CCS): the fraction of recommendations free of logical contradictions between the decoded patient state and the recommended action (e.g., recommending no vasopressor when MAP is flagged as hypotensive).

  • Recommendation Completeness (RC): the fraction of outputs containing all four mandatory fields (vasopressor description, fluid description, two reasoning steps, one guideline citation).

  • Uncertainty Calibration (UC): the Pearson correlation between the vasopressor agent’s Q-value range and the reported confidence level (high = 1, medium = 0.5, low = 0), testing whether the system expresses greater uncertainty in states where its Q-values are less discriminative.

A composite score is computed as shown in Eq. (8):

$$\:Composite=0.30\:\text{G}\text{G}\text{S}+0.30\left(1-\text{H}\text{R}\right)+0.25\:\text{C}\text{C}\text{S}+0.15\:\text{R}\text{C}$$
(8)

The system is evaluated against two baselines over the same 200 test-split transitions: a random reasoner with no retrieval, and a Phase-2 DQN reasoner using a fixed first-document citation. We note that the reasoning layer was evaluated over 98 timesteps from 10 test episodes; this sample is acknowledged to be small, and the reported metrics particularly the 0% hallucination rate and 100% completeness should be interpreted as preliminary estimates rather than definitive performance bounds. Expansion to 300–500 timesteps is a prioritised future direction. All three conditions use seed 42.

Evaluation Protocol

All phase evaluations use 200 test-split episodes with a greedy deterministic policy and seed 42. The primary metrics are: mean cumulative episode reward; clinician agreement rate, defined as the percentage of timesteps where the agent’s action matches the recorded clinician action; action diversity, measured as unique action cells occupied and Shannon entropy; and Q-value stability, measured as mean maximum Q-value and Q-value range across 100 randomly sampled test states. For Phase 4, the five reasoning-quality metrics are computed over 98 timesteps from 10 test episodes. All evaluations use a deterministic policy with a fixed seed (42), so differences in results reflect algorithmic changes rather than sampling variance. We acknowledge that reporting a single seed is a limitation; multi-seed validation (3–5 seeds) is a prioritised direction for future work (see "Future Directions").

Ethical Considerations

All data were accessed under the MIMIC-IV Data Use Agreement following CITI programme certification. The anonymisation pipeline removes all direct patient identifiers, and no re-identification was performed. HM-CDSS is purely a research prototype and has not been validated for prospective clinical use. The antibiotic and monitoring agents use proxy action labels derived from the vasopressor bin rather than from genuine antibiotic prescription or monitoring event data extracted from MIMIC-IV; this substantially limits the clinical validity of these two agents in the current implementation. Any future deployment would require prospective validation, regulatory approval, and human-in-the-loop oversight.

Results

This section presents the experimental results across all four phases of the proposed HMARL framework. Results are organised to directly address the four research objectives, with phase-by-phase comparisons after Phase 1 emphasizing the incremental contribution of each architectural component.

Phase 2 - Single-Agent DQN Baseline

The single-agent Dueling DQN was trained for 500 episodes using 3,697 training patients. The smoothed episode reward stabilised between 4.0 and 4.5 within approximately episode 30 and remained stable afterwards, confirming that the offline batch-RL protocol converged without divergence. The final validation mean reward of 4.433 closely tracked the test mean of 4.385, confirming that the temporal split prevented overfitting. The Q-value range of 1.089 signifies that the agent discriminated between actions without Q-value explosion.

Despite stable training, the baseline exhibited pronounced conservative policy collapse. The agent concentrated 79.3% of all actions in three cells of the 5 × 5 vasopressor–fluid joint action space — specifically (V1,F1), (V2,F1), and (V3,F1) — all pairing low-to-moderate fluid with varying vasopressor doses (Fig. 3). The clinician, by contrast, used a substantially broader distribution, with the modal action at (V2,F2) accounting for 30.9% and significant mass at (V2,F3) and (V3,F3). This systematic under-prescription of fluid is consistent with the conservative offline-RL behaviour described by Kumar et al. [11]: without CQL regularisation, Q-value overestimation on the terminal + 5.0 reward drives the agent towards the low-variance path of minimal fluid and moderate vasopressor. Clinician agreement reached only 10.3%, and all 200 test episodes terminated in the “alive, not home discharge” outcome group. These baseline results are summarised in Table 4, and the mean Q-value comparison is shown in Fig. 4.

Fig. 3
The DQN baseline concentrates actions in fluid bin F1, the clinician reference distributes actions across several bins, and HMARL plus CQL uses only vasopressor bin V2. The most frequent combinations are F1 and V3 for DQN, at 33.4 percent; F2 and V2 for clinicians, at 30.9 percent; and F2 and V2 for HMARL plus CQL, at 92.3 percent. The remaining 7.7 percent of HMARL plus CQL actions are in F3 and V2.
Full size image

Joint action distribution heatmaps: Phase 2 DQN vs. clinician vs. Phase 3 HMARL + CQL

Table 4 Phase 2 single-agent DQN evaluation results (200 test episodes, seed 42)
Full size table
Fig. 4
The Phase 3 vasopressor agent has the highest mean maximum Q-value and the widest mean Q-value range. Mean maximum Q-values are 4.441 for Phase 2 DQN, 8.677 for Phase 3 vasopressor, and 2.011 for Phase 3 fluid. Mean Q-value ranges are 1.089, 17.041, and 8.038, respectively. An annotation describes the wider range as stronger action discrimination.
Full size image

Q-value statistics — Phase 2 DQN vs. Phase 3 CQL agents

Phase 3 - HMARL with CQL

Training Convergence

All five CQL agents were trained simultaneously over 500 episodes (Table 5). The meta-agent reduced its CQL loss by 21.0% from the first 50 to the last 50 episodes, and the monitoring agent achieved the largest proportional reduction at 47.7%. The vasopressor agent showed a 33.6% loss increase, which is the expected behaviour of CQL in a high-risk domain with α = 1.0: as the agent’s policy approaches the data manifold, the log-sum-exp penalty increases because the gap between preferred and observed actions narrows, producing a higher raw loss that reflects tighter constraint rather than instability. Training loss curves for all five agents are shown in Fig. 5.

Table 5 Per-agent CQL training loss statistics across 500 episodes. Negative Δ indicates loss reduction (convergence). The vasopressor increase reflects tightening of the CQL constraint (α = 1.0)
Full size table
Fig. 5
Loss decreases for Meta and Monitor, increases for Vasopressor and Antibiotic, and remains nearly stable for Fluid. The reported changes are minus 21.0 percent for Meta, plus 33.6 percent for Vasopressor, minus 1.7 percent for Fluid, plus 14.8 percent for Antibiotic, and minus 47.7 percent for Monitor.
Full size image

CQL training loss curves — all five agents over 500 episodes

Policy Performance and Ablation Study

The proposed HMARL system achieved 49.9% combined clinician agreement on the test split, a gain of 39.6% points over the Phase-2 baseline of 10.3%. Vasopressor-specific agreement reached 64.6% and fluid agreement reached 35.2%. The mean shared reward of 4.214 (SD 1.052) is slightly lower than the Phase-2 baseline value of 4.385; though, since both systems converge to the same terminal reward group (alive, positive + 5.0), this difference reflects per-step reward accumulation rather than a difference in patient outcomes. The reward function is a proxy for patient benefit, not a direct outcome measure, and the finding that a more clinically aligned policy achieves fundamentally the same reward as a collapsed policy highlights a known limitation of substitute-reward evaluation in offline RL [2].

The ablation study, shown in Table 6, provides the strongest account of what drives each improvement. Ablation B, replacing CQL with vanilla DQN inside the HMARL architecture, achieved only 28.1% agreement — 21.8% points below the full system. This demonstrates that the distributional constraint is a distinct and necessary mechanism, not a derivative of architectural decomposition. Ablation A, removing the hierarchy and using flat CQL agents, achieved 51.3% agreement, marginally higher than the full system’s 49.9%. This result indicates that the hierarchy does not provide a measurable advantage in clinician agreement on the current eight-feature dataset; the hierarchy’s primary contributions are architectural modularity, per-domain interpretability, and extensibility to richer state representations not a performance gain over flat CQL in this experimental setting. Ablation C, removing domain sub-states while retaining hierarchy and CQL, achieved 50.8%. These results indicate that sub-state partitioning does not add clinician agreement on this dataset, a finding attributable to the small state dimensionality (eight features) deployed in the current implementation. At this dimensionality, each sub-state partition contains only six features drawn from the same eight-feature pool, providing insufficient domain separation (see Sect.  5.3). This does not invalidate the architecture: sub-state partitioning is essential for the reasoning layer and for future extension when a stronger 41-feature state vector is used. The full HMARL + CQL system achieves the lowest policy entropy (0.076 nats), producing the most deterministic and auditable policy of all conditions evaluated (Fig. 6).

Table 6 Ablation study results (200 test episodes, seed 42). The full HMARL + CQL system represents the proposed system
Full size table
Fig. 6
Clinician agreement is highest for Ablation A at 51.3 percent, followed by Ablation C at 50.8 percent and Full HMARL plus CQL at 49.9 percent. Ablation B reaches 28.1 percent, and Phase 2 DQN reaches 10.3 percent. Policy entropy is lowest for Full HMARL plus CQL at 0.076 nats, followed by Ablation A at 0.657, Ablation C at 0.844, Ablation B at 1.299, and Phase 2 DQN at 1.764.
Full size image

Ablation study — clinician agreement and policy entropy across five conditions

The meta-agent goal distribution (Table 7) shows Goal 1 (fluid management) dominating at 45.5% of timesteps, followed by Goal 2 (infection control) at 27.3%, Goal 0 (haemodynamic stabilization) at 15.0%, and Goal 3 (monitoring and de-escalation) at 12.2%. The mean vasopressor action remained consistently at 2.00–2.07 across all goals, confirming that goal selection modulates the clinical reasoning context rather than driving unsafe vasopressor escalation (Fig. 7).

Table 7 Meta-agent goal distribution across 200 test episodes
Full size table
Fig. 7
Across four reasoning-quality metrics, HMARL plus RAG scores higher than DQN without RAG and the random baseline on guideline grounding and recommendation completeness. Guideline grounding scores are 0.82, 0.35, and 0.20, respectively; recommendation completeness scores are 1.00, 0.55, and 0.30. For hallucination rate, where lower is better, DQN scores 0.35 and the random baseline 0.70; no HMARL value is shown. Clinician agreement is 0.39 for HMARL and 0.10 for DQN; no random baseline value is shown.
Full size image

Meta-agent goal distribution and vasopressor modulation across goals

Phase 4 - Clinical Reasoning Evaluation

The clinical reasoning layer was tested over 98 timesteps spanning 10 test-split episodes. HMARL + RAG demonstrated 100% Recommendation Completeness and 0% Hallucination Rate across all tested timesteps, indicating that the knowledge-base validation layer successfully blocked citation of incorrect guidelines. Context-sensitive guideline retrieval demonstrated superior performance over the DQN baseline on Guideline Grounding Score and clinician agreement (38.8% vs. 10.3%). These findings are summarized in Table 8; Fig. 8. Calculation of the Uncertainty Calibration metric was impractical for the current test split because the HMARL agents were confident in their recommendations at every sampled state; there was insufficient variance to establish a correlation with the Q-value range.

Table 8 Phase 4 reasoning quality evaluation across three conditions (98 timesteps, 10 test episodes)
Full size table
Fig. 8
Across 200 episodes, fluid management was the most frequently selected goal, at 45.5 percent of timesteps, followed by infection control at 27.3 percent, haemodynamic stabilisation at 15.0 percent, and monitoring and de-escalation at 12.2 percent. Mean vasopressor action was 2.07 for infection control and 2.00 for each other goal. The moderate norepinephrine threshold was 2.00.
Full size image

Phase 4 reasoning quality — three-system comparison across four metrics

A representative case illustrates the reasoning behaviour of the system: for patient 37,939,725 during the eighth ICU hour, the meta-agent selected Goal 2 (infection control). The vasopressor agent recommended moderate norepinephrine (bin V2: 0.05–0.1 mcg/kg/min), matching the clinician’s decision. The fluid agent recommended moderate fluid (bin F2: 50–125 mL), whereas the clinician administered a liberal fluid bolus (bin F3: 125–250 mL). The reasoning module noted that the fluid agent’s recommendation was consistent with the patient’s normal lactate levels, while acknowledging that the clinician may have weighed additional factors not captured in the current state vector, such as trends in urine output. Per-goal clinician agreement, broken down by vasopressor, fluid, and combined action, is shown in Fig. 9.

Fig. 9
Across 98 timesteps in phase 4, combined agreement was highest for fluid management at 62 percent and zero for haemodynamic goals. Overall combined agreement was 38.8 percent.For each goal, vasopressor, fluid, and combined agreement were respectively:- Fluid management, 45.5 percent of goals: 70, 66, and 62 percent.- Infection control, 27.3 percent: 47, 28, and 12 percent.- Monitoring and de-escalation, 16.3 percent: 75, 44, and 31 percent.- Haemodynamic, 3.1 percent: zero percent for all three.
Full size image

Per-goal clinician agreement breakdown (vasopressor vs. fluid vs. combined)

Discussion

Addressing the Research Objectives

RO1

The first objective sought to construct and compare a hierarchical MARL agent employing CQL and domain sub-states against a single flat DQN agent, producing a more clinically aligned policy. There is partial support for a positive answer. First, the 39.6-percentage-point increase in clinician agreement demonstrates substantially greater alignment with clinical practice. However, the ablation results show that this increase is driven primarily by CQL regularisation rather than the hierarchy itself: Ablation A (flat CQL, no hierarchy) achieved a slightly higher agreement (51.3%) than the full hierarchical system (49.9%). The hierarchy alone did not increase clinician agreement, though the full hierarchical model achieved the lowest policy entropy (0.076 nats), indicating the most deterministic and auditable policy among all conditions.

RO2

The second objective aimed to show that CQL prevents conservative policy collapse. Comparing Ablation B (hierarchical DQN, 28.1%) with the full HMARL + CQL model (49.9%) yields an unambiguous 21.8-percentage-point gap, demonstrating that distributional constraint and hierarchy are mutually supportive, non-redundant mechanisms. By virtue of hierarchy, the action space per agent becomes smaller; CQL then ensures that Q-function overestimation does not occur within this smaller space. The elevated CQL loss in the vasopressor agent demonstrates the regularization’s active engagement during training.

RO3

The third objective was to design and test a RAG-based reasoning layer that generates hallucination-free, clinically sound justifications. Zero citations to non-existent guideline identifiers (0% Hallucination Rate) combined with 100% recommendation completeness across 98 timesteps provides preliminary evidence of structural completeness in the generated reasoning within this limited evaluation sample. It should be noted that “hallucination” here is defined narrowly as citation to identifiers absent from the knowledge base; factual correctness and patient-specific clinical correctness were not independently assessed. The five-metric framework represents a proposed evaluation toolkit for reasoning quality in clinical MARL to the best of our knowledge, no prior standardised quantitative framework exists for this purpose addressing the absence of such metrics identified as a key gap in the domain [2]. The Uncertainty Calibration metric could not be computed in the current evaluation because agent confidence was uniformly high across all sampled states, leaving insufficient variance for correlation analysis. The current study therefore empirically validates four metrics (GGS, HR, CCS, RC); UC is proposed as a metric for future validation when a larger, more severity-diverse evaluation sample is available.

RO4

The fourth objective assessed whether goal conditioning produces temporal coherence in the goal distribution. The meta-agent’s goal distribution is clinically coherent (Fig. 7): fluid administration and infection control together account for 72.8% of timesteps, consistent with sepsis-management priorities in the MIMIC-IV cohort. The similarity in mean vasopressor action across goals (2.00–2.07) suggests that goal selection modulates the reasoning context rather than directly driving medication dosage. Further analysis to evaluate temporal coherence within SOFA-stratified subgroups would require a larger dataset and is prioritised for future research.

Comparison with Prior Work

The most directly comparable prior system is the AI Clinician of Komorowski et al. [4], which established the design pattern for offline-RL-based sepsis treatment optimization. HM-CDSS advances beyond that foundation in three ways: hierarchical decomposition of the action space, CQL regularisation for distributional safety, and a guideline-based reasoning layer that produces auditable clinical justifications. Tan et al. [9] represent the most architecturally similar prior work, with organ-specific agents and dual-layer state representation for sepsis management. Their system shares the HMARL motivation with HM-CDSS but does not address offline distribution shift, does not incorporate an explainability layer, and does not report clinician agreement or ablation results. The present work may therefore be understood as extending the Tan et al. framework with the three components identified as most critical by Silveira et al. [2].

Across the taxonomy established by Silveira et al. [2], HM-CDSS belongs to the 10% of systems that employ full hierarchical architectures, is only the second system alongside Zuo et al. [22] to combine hierarchical MARL with LLM-based reasoning, and includes an agreement-weighted guideline-relevance update as a lightweight implementation of bidirectional knowledge flow between agent behavior and the knowledge base. A comparison with prior studies is shown in Table 9.

Table 9 Comparison of HM-CDSS with representative prior clinical decision support and reinforcement learning systems
Full size table

Future Directions

Seven directions are prioritised for subsequent work. First, re-extraction of a full 41-feature MIMIC-IV state vector with per-timestep SOFA component scores and hourly lactate measurements would address both the dimensionality limitation and reward sparsity simultaneously. Second, replacing proxy antibiotic and monitoring actions with real MIMIC-IV prescription-event data would convert the two placeholder agents into fully validated specialist agents. Third, a clinician annotation study rating paired CDSS outputs would provide the human-in-the-loop validation identified as the primary gap between technical capability and clinical adoption [2]. Fourth, implementation of full ontological grounding using SNOMED CT concept annotations and inference-engine validation would address the bidirectional knowledge-flow gap more rigorously than the current lightweight relevance-weighting mechanism. Fifth, extension to additional disease domains, including acute kidney injury and ventilator management, would test the generalizability of the hierarchical framework beyond the sepsis context. Sixth, repeating the principal experiments across 3–5 independent random seeds would establish robustness to training variability and provide confidence intervals on the reported agreement rates. Seventh, a systematic sensitivity analysis of the CQL regularisation coefficient α across values {0, 0.1, 0.25, 0.5, 1.0, 2.0} for each agent would provide empirical justification for the agent-specific α values used in the current study.

Summary of Contributions

This study makes four contributions to the field of clinical decision support. First, it presents a conservative offline-RL hierarchical MARL framework for sepsis with CQL regularisation, improving clinician agreement from 10.3% to 49.9% over a single-agent DQN baseline, with the ablation study demonstrating that CQL rather than the hierarchy is the primary driver of this improvement. Second, it provides a three-condition ablation study isolating the contributions of hierarchy (+ 17.8% points), CQL (+ 21.8% points), and sub-state partitioning (marginal, dataset-dependent) the first component-wise analysis of an HMARL clinical decision support system. Third, it introduces a retrieval-augmented reasoning layer achieving 0% hallucination and 100% recommendation completeness, grounded in Sepsis-3 and Surviving Sepsis Campaign guidelines. Fourth, it proposes a five-metric quantitative reasoning-evaluation framework (GGS, HR, CCS, RC, UC) that provides a reproducible benchmark for explainability evaluation in clinical MARL, directly addressing the absence of standardized metrics identified by Silveira et al. [2].

Conclusion

This study proposed HM-CDSS, a hierarchical multi-agent reinforcement learning framework with Conservative Q-Learning and retrieval-augmented clinical reasoning, designed to assist medical professionals in selecting the best course of therapy for sepsis patients in the intensive care unit. The model addresses three key deficiencies of existing frameworks: the lack of hierarchical structure to account for the multi-layered nature of clinical reasoning, the lack of safeguards against distributional shift during offline training on historical data, and the lack of explainable processes grounded in clinical practice guidelines. Evaluated on 5,282 real ICU patients from the MIMIC-IV dataset, the proposed system shows a marked improvement in clinician agreement, from 10.3% to 49.9% relative to a single-agent baseline, with ablation studies showing that Conservative Q-Learning is the core mechanism behind this increase, contributing an additional 21.8% points beyond a hierarchical Deep Q-Network. The retrieval-based reasoning module ensures zero hallucinations and a 100% recommendation-completeness rate across all assessed timesteps, and the five-metric evaluation suite for the reasoning component is a proposed systematic quantitative method for evaluating explanation quality in this context; to the best of our knowledge, no prior comparable framework has been reported for clinical MARL. This is an important step towards potentially clinically useful decision support tools that are interpretable, auditable, and evidence-grounded, pending prospective validation and clinician evaluation beyond the retrospective MIMIC-IV setting. Future work will investigate richer state spaces, validated action sets, and clinician annotations to narrow the remaining gap between performance and clinical application.