Advertisement

Performance of an Electronic Decision Support System as a Therapeutic Intervention During a Multicenter PICU Clinical Trial

Heart and Lung Failure-Pediatric Insulin Titration Trial (HALF-PINT)
Open AccessPublished:April 28, 2021DOI:https://doi.org/10.1016/j.chest.2021.04.049

      Background

      The use of electronic clinical decision support (CDS) systems for pediatric critical care trials is rare. We sought to describe in detail the use of a CDS tool (Children’s Hospital Euglycemia for Kids Spreadsheet [CHECKS]), for the management of hyperglycemia during the 32 multicenter Heart And Lung Failure-Pediatric Insulin Titration trial.

      Research Question

      In critically ill pediatric patients who were treated with CHECKS, how was user compliance associated with outcomes; and what patient and clinician factors might account for the observed differences in CHECKS compliance?

      Study Design and Methods

      During an observational retrospective study of compliance with a CDS tool used during a prospective randomized controlled trial, we compared patients with high and low CHECKS compliance. We investigated the association between compliance and blood glucose metrics. We describe CHECKS and use a computer interface analysis framework (the user, function, representation, and task analysis framework) to categorize user interactions. We discuss implications for future randomized controlled trials.

      Results

      Over a 4.5-year period, 658 of 698 children were treated with the CHECKS protocol for ≥24 hours with a median of 119 recommendations per patient. Compliance per patient was high (median, 99.5%), with only 30 patients having low compliance (<90%). Patients with low compliance were from 16 of 32 sites, younger (P = .02), and less likely to be on inotropic support (P = .04). They were more likely to be have been assigned randomly to the lower blood glucose target (80% vs 48%; P < .001) and to have spent a shorter time (53% vs 75%; P < .001) at the blood glucose target. Overrides (classified by the user, function, representation, and task analysis framework), were largely (89%) due to the user with patient factors contributing 29% of the time.

      Interpretation

      The use of CHECKS for the Heart And Lung Failure-Pediatric Insulin Titration trial resulted in a highly reproducible and explicit method for the management of hyperglycemia in critically ill children across varied environments. CDS systems represent an important mechanism for conducting explicit complex pediatric critical care trials.

      Clinical Trial Registration

      ClinicalTrials.gov Identifier: NCT01565941, registered March 29 2012; https://clinicaltrials.gov/ct2/show/NCT01565941?term=HALF-PINT&draw=2&rank=1

      Key Words

      Abbreviations:

      CDS (clinical decision support), CHECKS (Children’s Hospital Euglycemia for Kids Spreadsheet), HALF-PINT (Heart and Lung Failure-Pediatric Insulin Titration), UFuRT (user, function, representation, and task analysis framework)
      Electronic clinical decision support (CDS) systems or tools are infrequently used in the PICU, despite widespread data supporting their use in adult ICUs.
      • Williams C.N.
      • Bratton S.L.
      • Hirshberg E.L.
      Computerized decision support in adult and pediatric critical care.
      Additionally, the use of CDS systems for multicenter research trials are often underutilized.
      • Thompson B.T.
      • Orme J.F.
      • Zheng H.
      • et al.
      Multicenter validation of a computer-based clinical decision support tool for glucose control in adult and pediatric intensive care units.
      • Morris A.H.
      • Orme Jr., J.
      • Truwit J.D.
      • et al.
      A replicable method for blood glucose control in critically ill patients.
      • Hirshberg E.L.
      • Lanspa M.J.
      • Wilson E.L.
      • et al.
      A pediatric intensive care unit bedside computer clinical decision support protocol for hyperglycemia is feasible, safe and offers advantages.
      With the use of an electronic CDS system, Children’s Hospital Euglycemia for Kids Spreadsheet (CHECKS), the National Institutes of Health-funded Heart and Lung Failure-Pediatric Insulin Titration (HALF-PINT) trial demonstrated that blood glucose control to a target of 80 to 110 mg/dL with IV insulin was associated with increased hypoglycemia and conferred no advantage compared with a blood glucose target of 150 to 180 mg/dL.
      • Agus M.S.
      • Wypij D.
      • Hirshberg E.L.
      • et al.
      Tight glycemic control in critically ill children.
      Large multicenter trials provide the highest quality evidence that indicates the optimal approach to blood glucose control with IV insulin.
      • Agus M.S.
      • Wypij D.
      • Hirshberg E.L.
      • et al.
      Tight glycemic control in critically ill children.
      • Hirshberg E.L.
      • Sward K.A.
      • Faustino E.V.
      • et al.
      Clinical equipoise regarding glycemic control: a survey of pediatric intensivist perceptions.
      • Macrae D.
      • Grieve R.
      • Allen E.
      • et al.
      A randomized trial of hyperglycemic control in pediatric intensive care.
      • Macrae D.
      • Tasker R.C.
      • Elbourne D.
      A trial of hyperglycemic control in pediatric intensive care.
      Furthermore, successful dissemination of such research results requires demonstration of a safe, feasible, and replicable intervention of blood glucose control that is generalizable and minimizes the cognitive or work burden of bedside nurses.
      • Macrae D.
      • Pappachan J.
      • Grieve R.
      • et al.
      Control of hyperglycaemia in paediatric intensive care (CHiP): study protocol.
      • Agus M.S.
      • Hirshberg E.
      • Srinivasan V.
      • et al.
      Design and rationale of Heart and Lung Failure - Pediatric INsulin Titration Trial (HALF-PINT): a randomized clinical trial of tight glycemic control in hyperglycemic critically ill children.
      • Faraon-Pogaceanu C.
      • Banasiak K.J.
      • Hirshberg E.L.
      • Faustino E.V.
      Comparison of the effectiveness and safety of two insulin infusion protocols in the management of hyperglycemia in critically ill children.
      In the adult ICU, the use of electronic CDS systems for blood glucose control in clinical care results in more consistent blood glucose target levels and fewer adverse events than do paper protocols.
      • Eslami S.
      • de Keizer N.F.
      • Dongelmans D.A.
      • de Jonge E.
      • Schultz M.J.
      • Abu-Hanna A.
      Effects of two different levels of computerized decision support on blood glucose regulation in critically ill patients.
      ,
      • Mann E.A.
      • Jones J.A.
      • Wolf S.E.
      • Wade C.E.
      Computer decision support software safely improves glycemic control in the burn intensive care unit: a randomized controlled clinical study.
      Before HALF-PINT, investigators implemented paper protocols or guidelines for IV insulin titration in the PICU,
      • Faraon-Pogaceanu C.
      • Banasiak K.J.
      • Hirshberg E.L.
      • Faustino E.V.
      Comparison of the effectiveness and safety of two insulin infusion protocols in the management of hyperglycemia in critically ill children.
      ,
      • Preissig C.M.
      • Hansen I.
      • Roerig P.L.
      • Rigby M.R.
      A protocolized approach to identify and manage hyperglycemia in a pediatric critical care unit.
      largely because electronic CDS protocols for clinical care in the PICU are not accepted widely.
      • Mack E.H.
      • Wheeler D.S.
      • Embi P.J.
      Clinical decision support systems in the pediatric intensive care unit.
      • Randolph A.G.
      • Clemmer T.P.
      • East T.D.
      • et al.
      Evaluation of compliance with a computerized protocol: weaning from mechanical ventilator support using pressure support.
      • Randolph A.G.
      • Pronovost P.
      Reorganizing the delivery of intensive care could improve efficiency and save lives.
      • Randolph A.G.
      • Wypij D.
      • Venkataraman S.T.
      • et al.
      Effect of mechanical ventilator weaning protocols on respiratory outcomes in infants and children: a randomized controlled trial.
      Historically, clinicians express concern about electronic-based tools due to alienation from bedside decisions and the possible introduction of unseen risk to the patient.
      • Williams C.N.
      • Bratton S.L.
      • Hirshberg E.L.
      Computerized decision support in adult and pediatric critical care.
      ,
      • Hunt D.L.
      • Haynes R.B.
      • Hanna S.E.
      • Smith K.
      Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review.
      ,
      • Tobin M.J.
      • Jubran A.
      Meta-analysis under the spotlight: focused on a meta-analysis of ventilator weaning.
      Excessive time consumption, interruptions of workflow, and interface usability are other stated barriers to bedside electronic CDS system implementation.
      • Williams C.N.
      • Bratton S.L.
      • Hirshberg E.L.
      Computerized decision support in adult and pediatric critical care.
      ,
      • Weber S.
      Clinical decision support systems and how critical care clinicians use them.
      During the HALF-PINT trial, CHECKS served as the research intervention protocol and was exported to 35 PICUs, 32 of which enrolled patients in the study. A novel just-in-time training and monitoring of adherence was facilitated by a web-based system that aggregated data in real time and processed the data on a server located at the clinical coordinating center for the trial. These efforts contributed to an overall compliance with CHECKS recommendations of >98%.
      • Agus M.S.
      • Wypij D.
      • Hirshberg E.L.
      • et al.
      Tight glycemic control in critically ill children.
      We describe in detail the adoption of CHECKS for the management of blood glucose control during the HALF-PINT trial. We compare patient factors, glucose metrics, and outcomes between high and low compliance with recommendations groups. We discuss novel CDS tools and clinician interaction data and perform an electronic CDS tool framework analysis to better understand broad areas of electronic CDS system refinement to enhance the adoption of CHECKS for additional research. We detail the website used to facilitate training. We expand our discussion to encourage the use of electronic CDS systems as explicit protocols in future pediatric critical care clinical trials.

      Study Design and Methods

       Study Site and Subject Selection

      In this secondary analysis of the HALF-PINT trial, patients at 32 sites who were treated with the CHECKS computer blood glucose control protocol were eligible for inclusion.
      • Agus M.S.
      • Wypij D.
      • Hirshberg E.L.
      • et al.
      Tight glycemic control in critically ill children.
      The parent HALF-PINT trial included children aged 2 weeks to 17 years with heart and/or lung failure and hyperglycemia and randomly assigned patients to a blood glucose target of 80 to 110 mg/dL vs 150 to 180 mg/dL. Hyperglycemia and blood glucose levels were managed by the CHECKS tool for all study patients. The Boston Children’s Institutional Review Board approved the study with informed parental consent (IRB # P00002310). We excluded patients in this analysis who were treated on CHECKS for less than 24 hours. We first reviewed overall compliance with the protocol as reported from the parent trial and a priori determined compliance <90% to be suboptimal. We compared baseline characteristics, blood glucose management and insulin therapy, and clinical outcomes between patients with high compliance (≥90%) and low compliance (<90%).

       Data Collection

      Baseline characteristics included demographic information, medical history, reason for ICU admission, Pediatric Risk of Mortality III-12 score,
      • Pollack M.M.
      • Patel K.M.
      • Ruttimann U.E.
      The pediatric risk of mortality III: Acute Physiology Score (PRISM III-APS): a method of assessing physiologic instability for pediatric intensive care unit patients.
      and invasive care therapies at the time of randomization. The CHECKS program template for all patients was maintained as a single version on a Boston Children’s Hospital server. Randomization occurred via a web-application built into the study website, with the treatment group assignment, patient weight, and insulin concentration being verified against a table of accepted values before being passed into a copy of the program template and downloaded to a bedside laptop. Thereafter, data from the individual sites was synchronized hourly with the central server and aggregated into a composite Structured Query Language database. From the CHECKS database, we calculated days of insulin therapy, average daily insulin dose, number of average daily glucose measurements, time to and time in the target range, and time-weighted blood glucose average. Clinical outcomes included ICU-free days through day 28, the primary outcome of the HALF-PINT trial, ventilator- and hospital-free days through day 28, and hospital mortality rate at 28 and 90 days. We also collected site-level data on factors that might affect compliance, which included whether sites used insulin protocols before HALF-PINT and whether nurses and/or resident physicians had the authority to enact therapeutic changes based on CHECKS recommendations.

       Exploratory Granular Analysis of the CHECKS CDS Tool Compliance

      The CHECKS system provided detailed instructions in response to patient-specific data. Instructions included when to enter Continuous Glucose Monitor, which measures glucose concentration in interstitial fluid, and/or bedside glucose meter (Nova StatStrip; Nova Biomedical), which measures whole blood glucose, results into the algorithm. CHECKS then provided detailed explicit instructions on how to adjust the insulin infusion rate to reach target blood glucose range (e-Fig 1) or dextrose rescue doses to avoid impending, or treat active, hypoglycemia. If the CHECKS algorithm (proportional-integral-derivative)
      • Steil G.M.
      • Reifman J.
      Mathematical modeling research to support the development of automated insulin-delivery systems.
      recommended a change in insulin delivery, the user was prompted to verify the value using the bedside glucose meter and to follow the instructions accordingly. Once a recommendation was made, the bedside nurse and/or physician had the option to accept or override recommendation. In the event of an override, the user was provided with a pop-up menu (e-Fig 2) with the option to choose from a predetermined list of possible reasons or a comment box in which to explain the reason the recommendation was not being followed. More detail regarding the CHECKS system is described by Steil and Reifman.
      • Steil G.M.
      • Reifman J.
      Mathematical modeling research to support the development of automated insulin-delivery systems.
      We further conducted an exploratory analysis (both quantitative and qualitative) of the CHECKS CDS tool compliance by blood glucose measurement value and by compliance group to better understand the factors associated with CDS tool compliance. We reviewed user-specified reasons for CHECKS overrides. Due to the inadequate specificity of the embedded responses, we conducted a qualitative review of all overrides in the low-compliance group.

       CDS Tool Framework Analysis

      A random sample of 10 files of patients in the low-compliance group was reviewed by the primary author (E. Hirshberg) who then derived a grouping of themes that consistently emerged as applicable to CHECKS recommendations that were overridden. This list of eight themes was then sent to the entire writing group for refinement and consensus. A total of 12 themes were agreed on by the writing group before clinical review and category assignment. Once the themes were agreed on, an independent and more detailed interrogation of each of the overrides in the 30 patients with compliance <90% was completed. A total of three clinicians independently reviewed each chart and each overridden CHECKS recommendation. Each override was coded to all themes that applied, and these themes were analyzed quantitatively for descriptive purposes.
      The 12 themes further fit into the larger categories common to evaluation of CDS tools and based on a modified version of the user (clinician judgment, environment, or clinician beliefs) function (mechanical issues), representation (work flow), and task analysis (algorithm, interface) (UFuRT) classification
      • Zhang J.
      • Butler K.A.
      UFuRT: A work-centered framework and process for design and evaluation of information systems.
      ,
      • Nahm M.
      • Zhang J.
      Operationalization of the UFuRT methodology for usability analysis in the clinical research data management domain.
      plus a fifth category described as patient factors.
      • Zhang J.
      • Butler K.A.
      UFuRT: A work-centered framework and process for design and evaluation of information systems.
      ,
      • Nahm M.
      • Zhang J.
      Operationalization of the UFuRT methodology for usability analysis in the clinical research data management domain.
      UFuRT framework provides both quantitative and qualitative assessments and was designed to elucidate context and potential usability issues of computer-based systems.
      • Zhang J.
      • Butler K.A.
      UFuRT: A work-centered framework and process for design and evaluation of information systems.
      ,
      • Nahm M.
      • Zhang J.
      Operationalization of the UFuRT methodology for usability analysis in the clinical research data management domain.

       Statistical Analysis

      We used Wilcoxon rank-sum tests or Fisher exact tests, as appropriate, to compare baseline characteristics and glycemia and insulin therapy variables between compliance groups. We used proportional-hazards, linear, and logistic regression, as appropriate, with adjustment for age group and severity of illness to compare hypoglycemia and clinical outcomes between compliance groups. All probability values are two-tailed and have not been adjusted for multiple comparisons. Analyses were performed with SAS software (version 9.4; SAS Institute Inc).

      Results

       Baseline Characteristics and CHECKS Compliance

      Of 698 patients who received the HALF-PINT protocol, 658 (94%) were treated with CHECKS for ≥24 hours with a total of 100,998 instructions (Fig 1). The median number of CHECKS instructions per patient was 119 (interquartile range, 64 to 194). We observed an overall compliance with CHECKS instruction of ≥90% in 628 patients (95%) with 31 of 32 sites represented, with a median of 19 patients per site (range, 1 to 74). We observed compliance of <90% in 30 patients (5%) that represented 16 of 32 sites, with a median of two patients per site (range, 1 to 5). We observed that patients in the low-compliance group were younger (P = .02), more likely to have insulin infusing (P = .002), and less likely to require vasoactive infusions (P = .04) at the time of randomization. We observed no differences in other baseline characteristics that included the reason for ICU admission and Pediatric Risk of Mortality III-12 scores between the low-compliance and high-compliance groups (Table 1). There were no significant associations between CHECKS compliance group and clinical outcomes (e-Table 1).
      Figure thumbnail gr1
      Figure 1Flow diagram of patients who received the Heart and Lung Failure-Pediatric Insulin Titration protocol. CHECKS = Children’s Hospital Euglycemia for Kids Spreadsheet; HALF-PINT = Heart and Lung Failure-Pediatric Insulin Titration.
      Table 1Baseline Characteristics According to Compliance Group
      CharacteristicHigh Compliance (n = 628)Low Compliance (n = 30)P Value
      P values for the comparison between groups were calculated with the use of Wilcoxon rank-sum tests or Fisher’s exact tests, as appropriate.
      Age at ICU admission, y, median (IQR)6.4 (1.7-12.8)2.6 (0.8-9.3).02
       Age group, No. (%).08
      <2 y,177 (28)14 (47)
      2 to <7 y156 (25)7 (23)
      7 to <18 y295 (47)9 (30)
      Female, No. (%)291 (46)18 (60).19
      Black race, No./total (%)154/606 (25)6/27 (22).82
      Hispanic ethnic group, No./total (%)147/627 (23)6/29 (21).83
      Baseline cognitive impairment (Pediatric Cerebral Performance Category >1), No. (%)203 (32)7 (23).42
      Baseline functional impairment (Pediatric Overall Performance Category >1), No. (%)231 (37)13 (43).56
      Any known genetic syndrome, No. (%)116 (18)5 (17)1.0
      Primary reason for ICU admission, No. (%).56
       Respiratory, including infection327 (52)20 (67)
       Cardiovascular, including shock94 (15)2 (7)
       Neurologic58 (9)3 (10)
       Trauma57 (9)1 (3)
       Postoperative care44 (7)2 (7)
       GI or hepatic28 (4)2 (7)
       Other20 (3)0
      Insulin at randomization, No. (%)83 (13)11 (37).002
      Glucocorticoid therapy at randomization, No. (%)319 (51)19 (63).20
      Inotropic support for hypotension at randomization, No. (%)319 (51)9 (30).04
      Invasive mechanical ventilation (endotracheal tube or tracheostomy) at randomization, No. (%)621 (99)29 (97).31
      Extracorporeal membrane oxygenation at randomization, No. (%)31 (5)0.39
      PRISM III-12 score, median (IQR)12 (7-19)10 (5-16).28
      Risk of death based on PRISM III-12 score, %, median (IQR)10.0 (2.9-34.7)7.1 (2.7-23.5).72
      IQR = interquartile range; PRISM III-12 = Pediatric Risk of Mortality III score from first 12 h in the PICU.
      a P values for the comparison between groups were calculated with the use of Wilcoxon rank-sum tests or Fisher’s exact tests, as appropriate.

       Glucose Metrics and Compliance

      Patients in the low-compliance group were more likely to have been assigned randomly to the lower blood glucose target treatment group in HALF-PINT and to have received insulin therapy compared with patients in the high-compliance group (Table 2). The average number of daily blood glucose measurements and the time to reach blood glucose target range were higher in the low-compliance group compared with the high-compliance group, although the percentage of time spent in the blood glucose target range was lower in the low-compliance group. There were no differences in the time-weighted glucose averages or the occurrence of severe hypoglycemia (<40 mg/dL) or any hypoglycemia (<60 mg/dL) between the low-compliance and high-compliance groups (Table 2). Adjustment for study site did not change the results appreciably.
      Table 2Glycemia and Insulin Therapy According to Compliance Group
      VariableHigh Compliance (n = 628)Low Compliance (n = 30)P Value
      For hypoglycemia, P values for the comparison between groups were calculated with the use of logistic regression with adjustment for age group and Pediatric Risk of Mortality III score from first 12 h in the PICU score. For other variables, P values were calculated with the use of Wilcoxon rank-sum tests or Fisher exact tests, as appropriate.
      Percent compliance, %, median (IQR)99.6 (98.5-100)80.8 (73.3-87.0)<.001
      No. of recommendations, median (IQR)121 (65-195)72 (53-165).03
      Randomized to lower target treatment group, No. (%)302 (48)24 (80)<.001
      Treated with insulin therapy, No. (%)500 (80)29 (97).02
      No. of days of insulin therapy, median (IQR)3 (1-7)4 (3-7).21
      Average daily insulin dose, units/kg/d, median (IQR)0.23 (0.01-0.87)0.51 (0.24-0.74).02
      No. of average daily glucose measurements, median (IQR)11.9 (6.6-17.6)18.1 (14.4-20.3)<.001
      Time to the target range, h, median (IQR)2.5 (1.0-6.5)5.5 (3.0-17.0)<.001
      Time in the target range, %, median (IQR)75 (58-92)53 (39-58)<.001
      Time-weighted glucose average, mg/dL, median (IQR)113 (103-128)116 (109-132).16
      Severe hypoglycemia (any relatedness), No. (%)21 (3)2 (7).35
      Any hypoglycemia (any relatedness), No. (%)100 (16)9 (30).07
      IQR = interquartile range.
      a For hypoglycemia, P values for the comparison between groups were calculated with the use of logistic regression with adjustment for age group and Pediatric Risk of Mortality III score from first 12 h in the PICU score. For other variables, P values were calculated with the use of Wilcoxon rank-sum tests or Fisher exact tests, as appropriate.
      Of 30 sites contributing site-level data, eight sites (27%) used insulin protocols before HALF-PINT, and 17 sites (57%) gave nurses the autonomy without requiring physician confirmation to make changes to insulin or dextrose infusions based on CHECKS recommendations. Fifteen of the sites engaged resident physicians to participate in HALF-PINT; 67% of them gave resident physicians the autonomy to make changes to infusions based on CHECKS recommendations. These site-level factors were associated with statistically significant differences in compliance, but these differences were not clinically meaningful (e-Table 2).

       Granular Compliance Quantitative: Compliance by Measured Blood Glucose Level

      The low-compliance group had significantly lower compliance with CHECKS recommendations across all blood glucose ranges (<60, 60 to 79, 80 to 109, 110 to 129, 130 to 149, 150 to 179, 180 to 199, and ≥200 mg/dL) (Fig 2). In the low-compliance group, compliance with CHECKS recommendations was ≤69% in the two extreme blood glucose-measured ranges <60 mg/dL and ≥200 mg/dL and in the 110 to 129 mg/dL and 130 to 149 mg/dL ranges. Across both compliance groups, there were 337 recommendations when the blood glucose was <60 mg/dL, with 89.0% compliance overall. In this range, the low-compliance group received a higher than recommended dextrose dose 28% of the time, and the high-compliance group received a lower than recommended dextrose dose 6% of the time. In the low-compliance group, for the 192 recommendations when the blood glucose was ≥200 mg/dL, insulin was given at the recommended dose only 69% of the time, at a lower than recommended dose 24% of the time, and at a higher than recommended dose 7% of the time.
      Figure thumbnail gr2
      Figure 2Compliance within blood glucose ranges according to compliance group. CHECKS = Children’s Hospital Euglycemia for Kids Spreadsheet; HC = high compliance; LC = low compliance.

       Analysis of CHECKS Overrides: Reasons Provided by Care Team

      The CHECKS interface prompted the user, with a drop-down menu, for a reason to override an instruction and had a comment section (e-Fig 2). The most common reason for CHECKS recommendation override was “Other” (63%), followed by concern on the part of the care team for hypoglycemia or recent hypoglycemia (23%).

       Override Theme and UFuRT Classification by Research Team

      The 30 patients with <90% compliance had 731 overrides, each of which underwent individual review and classification into 12 override themes by three independent clinicians (Table 3). The most common reason (89%) for noncompliance with the protocol under the UFuRT system was bedside clinician beliefs without clinical evidence or the “user” classification: for example, “fear of hypoglycemia” without any documented hypoglycemia or clinical evidence to support the concern. Workflow and clinical circumstances were less common, with 31% falling under the “representation” classification. Noncompliance was related to patient factors only 29% of the time.
      Table 3Override Themes and User, Function, Representation, and Task Analysis Classifications
      User, Function, Representation, and Task Analysis ClassificationOverride ThemeOverride Theme,
      Three independent clinicians assigned themes to each override; more than one theme could be assigned per override (median, 2 [interquartile range, 2-3] themes per override).
      No. (%)
      User, Function, Representation, and Task Analysis Classification, No. (%)
      UserClinician factors (reasonable)513 (70)649 (89)
      Fear of hypoglycemia307 (42)
      Clinician factors (unreasonable)299 (41)
      Key stroke error13 (2)
      FunctionMechanistic/equipment factors63 (9)166 (23)
      “Distrust”57 (8)
      Actual hypoglycemia50 (7)
      RepresentationClinical scenarios194 (27)226 (31)
      Work flow issue36 (5)
      Task analysisCHECKS algorithm problem94 (13)118 (16)
      CHECKS instruction misunderstood/interface issue37 (5)
      Patient factors
      This category was added to the classic User, Function, Representation, and Task Analysis classification as the modification.
      Patient factors213 (29)213 (29)
      CHECKS = Children’s Hospital Euglycemia for Kids Spreadsheet.
      a Three independent clinicians assigned themes to each override; more than one theme could be assigned per override (median, 2 [interquartile range, 2-3] themes per override).
      b This category was added to the classic User, Function, Representation, and Task Analysis classification as the modification.

      Discussion

      We present the first granular analysis of multicenter use of an electronic bedside clinical decision support tool (CHECKS) as a therapeutic intervention protocol in 32 PICUs during a randomized clinical trial (HALF-PINT). The overall high compliance with CHECKS for both HALF-PINT treatment groups supports the continued development and use of electronic CDS systems for research. The CHECKS interface facilitated rapid just-in-time training for a successful, high compliance (median, 99.5% per patient) intervention protocol deployment. Lower compliance with CHECKS occurred infrequently (5% patients) and was associated with less favorable glucose metrics that included more frequent blood glucose measurements, less time in blood glucose target, and more time to reach blood glucose target. Although unfavorable glucose metrics could result from specific patient characteristics, a granular analysis of these patients uncovered that most override decisions (89%) fell under the user classification and only 29% were categorized as patient-specific factors.
      The CHECKS tool was integrated easily into practice as a therapeutic intervention and enabled a better understanding of the interaction between an electronic CDS tool and the bedside clinician.
      • Morris A.H.
      Developing and implementing computerized protocols for standardization of clinical decisions.
      ,
      • Clemmer T.P.
      Computers in the ICU: where we started and where we are now.
      Our results argue against the claim that electronic CDS systems alienate the clinician from the patient interaction, because clinician independence was indicated with the frequent check of “Other” as the reason for noncompliance.
      • Agus M.S.
      • Wypij D.
      • Hirshberg E.L.
      • et al.
      Tight glycemic control in critically ill children.
      ,
      • Timbie J.W.
      • Damberg C.L.
      • Schneider E.C.
      • Bell D.S.
      A conceptual framework and protocol for defining clinical decision support objectives applicable to medical specialties.
      Our data support previous speculation that electronic CDS tools can work well across several environments and multiple patient factors as long as appropriate training, education, and support are provided.
      • Mack E.H.
      • Wheeler D.S.
      • Embi P.J.
      Clinical decision support systems in the pediatric intensive care unit.
      Experience with CHECKS in a given enrollment site with CHECKS correlated with statistically significant differences in compliance, which supports work by others who note that trust or familiarity on the part of a user plays an important role in successful electronic CDS system adoption.
      • Goldberg H.S.
      • Paterno M.D.
      • Grundmeier R.W.
      • et al.
      Use of a remote clinical decision support service for a multicenter trial to implement prediction rules for children with minor blunt head trauma.
      ,
      • Opoku-Boateng G.A.
      User frustration in hit interfaces: exploring past HCI research for a better understanding of clinicians' experiences.
      We chose a just-in-time training model for CHECKS because any individual within the full group of bedside nurses in a given ICU could be assigned to a small volume of patients enrolled in the trial, often with limited or no prior experience with CHECKS. Online training modules consisted of short instructional videos paired with computer-scored competency tests and provided the capability of maintaining detailed records for on-site training of bedside staff in operation of CHECKS and the study devices. The clinical staff’s interaction with the protocol occurred continuously, 24 hours each day, at multiple locations, which made low-latency study monitoring a challenge. A web-based system was used to allow data to be aggregated and reviewed as it was being acquired; lead study investigators and data monitoring staff were alerted automatically to key issues such as protocol deviations as they occurred. This ultimately facilitated rapid iterative refinement of CHECKS.
      Electronic CDS systems have been incorporated internationally by hospitals for improved error reduction, diagnostic accuracy, and better patient outcomes
      • Hunt D.L.
      • Haynes R.B.
      • Hanna S.E.
      • Smith K.
      Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review.
      ; with our data, multicenter PICU trials can be added to this list. Lower compliance with CHECKS was associated with less favorable glucose metrics and with patients who were of younger age or receiving insulin infusions at the time of randomization, which confirms the role of CDS tools to augment, not to replace, clinical judgment.
      • Weber S.
      Clinical decision support systems and how critical care clinicians use them.
      ,
      • Avansino J.
      • Leu M.G.
      Effects of CPOE on provider cognitive workload: a randomized crossover trial.
      The CHECKS user interface did not capture completely reasons for noncompliance with the built-in drop-down menu, because nearly two-thirds of the reasons were marked as “Other.” This, in addition to the high number of user classified overrides, suggests areas for additional improvement in CHECKS and informs the design of future electronic CDS for therapeutic interventions during research.
      CHECKS was able to provide pertinent information to the bedside clinician at the time of care delivery that limited deviations. Importantly, the majority of CHECKS instructions occurred in optimal blood glucose clinical range of 80 to 180 mg/dL (including the target study ranges), which underscores the effectiveness of CHECKS in achieving target blood glucoses. Interestingly, compliance was <70% in the low-compliance group at both blood glucose extremes (<60 and ≥200 mg/dL) and in clinically acceptable ranges just outside of the trial targets (110 to 129 mg/dL and 130 to 149 mg/dL). These data suggest that users invoked independent recognition of an instruction’s potential to move the blood glucose away from a clinically acceptable range and, in those rare cases, overrode the instruction. Our data further demonstrate the bedside clinician’s ability to include factors unanticipated by CHECKS in deciding to override a recommendation. As expected, users cited fear of hypoglycemia as a reason for noncompliance highlighting the decision support (and safety) element of any CDS system.
      A common theme in electronic CDS tool deployment is the notion that individual users may create additional cognitive struggle for themselves.
      • Opoku-Boateng G.A.
      User frustration in hit interfaces: exploring past HCI research for a better understanding of clinicians' experiences.
      Interestingly, excessive clinician burden, and the associated information overload, has justified electronic CDS system use in the past.
      • Weber S.
      Clinical decision support systems and how critical care clinicians use them.
      ,
      • Morris A.H.
      Developing and implementing computerized protocols for standardization of clinical decisions.
      ,
      • Clemmer T.P.
      Computers in the ICU: where we started and where we are now.
      ,
      • Morris A.H.
      Computerized protocols and bedside decision support.
      Our granular analysis of reasons for noncompliance and UFuRT classification validate the observation by others that clinician frustration must be managed appropriately during electronic CDS development.
      • Opoku-Boateng G.A.
      User frustration in hit interfaces: exploring past HCI research for a better understanding of clinicians' experiences.
      ,
      • Lorenz R.
      • Pascual J.
      • Blankertz B.
      • Vidaurre C.
      Towards a holistic assessment of the user experience with hybrid BCIs.
      CHECKS serves as an example of a well-designed electronic CDS system for a therapeutic intervention that was used favorably.
      • Timbie J.W.
      • Damberg C.L.
      • Schneider E.C.
      • Bell D.S.
      A conceptual framework and protocol for defining clinical decision support objectives applicable to medical specialties.
      ,
      • Opoku-Boateng G.A.
      User frustration in hit interfaces: exploring past HCI research for a better understanding of clinicians' experiences.
      ,
      • Lorenz R.
      • Pascual J.
      • Blankertz B.
      • Vidaurre C.
      Towards a holistic assessment of the user experience with hybrid BCIs.
      Further incorporating clinician specialists’ feedback in designing and addressing important performance gaps may enhance widespread adoption of electronic CDS systems in pediatric critical care trials.
      • Timbie J.W.
      • Damberg C.L.
      • Schneider E.C.
      • Bell D.S.
      A conceptual framework and protocol for defining clinical decision support objectives applicable to medical specialties.
      This study has several limitations. Low compliance occurred in only a small number of patients. The preprogrammed reasons for noncompliance were not exhaustive, and the comment section rarely was completed. Most of the patients in the low-compliance group were assigned randomly to the low blood glucose target range during the parent trial, and clinician beliefs about the blood glucose target were not captured completely in CHECKS. CHECKS deployment was part of a National Institutes of Health-supported research investigation, which may not mimic other research environments. Our application of the UFuRT analysis represents a modified version of this analysis and may not be generalizable to other electronic CDS systems.

      Interpretations

      A well-designed electronic CDS system as a therapeutic intervention can be used successfully in pediatric critical trials with minimal just-in-time training. The CHECKS user interface worked well and facilitated easy identification of circumstances outside of the commonly expected clinical scenarios, which provided clear areas for future CHECKS refinement. Granular analysis of instruction compliance provided information about patient factors, clinical factors, workflow, and cultural or user factors that impacted study processes. It is possible that the same information could be extrapolated and used for the successful adoption of CHECKS into clinical practice. This study presents a pivotal step in navigating the possibilities of electronic CDS systems for future pediatric critical care trials. We suggest that future directions focus on electronic CDS systems for other therapeutic interventions and include investigations that export electronic CDS systems used for research into clinical practice, which would support the successful integration of critical care trial evidence into clinical practice.
      Study Question: What information can be learned from a secondary analysis of an electronic clinical decision support (CDS) system performance when used as a therapeutic intervention during a multicenter PICU clinical trial?
      Results: Electronic CDS system per patient compliance was high (99.5%) across a 32-site multicenter PICU trial. Patients with <90% compliance had common features and spent less time in the therapeutic target. Surprisingly, overrides were more often due to bedside clinician factors (89%) rather than patient factors (29%).
      Interpretation: Electronic CDS systems represent an important mechanism for conducting reproducible complex pediatric critical care trials. Analysis of CDS system and bedside clinician interaction provides important areas for refinement of CDS systems and lessons for successful translation of research results into clinical practice.

      Acknowledgments

      Author contributions: The task of reporting study results was accomplished in the following manner: conception (E. H., J. A., G. S., V. N., and M. A.), data acquisition (E. H., J. A., K. C.-W., D. S., and C. S.), data analysis (E. H., L. A.), writing manuscript (E. H.), revising manuscript for important intellectual content (all authors), approval of final copy (all authors).
      Financial/nonfinancial disclosures: None declared.
      ∗HALF-PINT Study collaborators: Michael Agus, David Wypij, Lisa Asaro (Boston Children’s Hospital, Boston, MA); Vinay Nadkarni, Vijay Srinivasan (Children’s Hospital of Philadelphia, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA); Katherine Biagas (Columbia University Medical Center, Columbia University College of Physicians and Surgeons, New York, NY); Peter M. Mourani (Children’s Hospital Colorado, University of Colorado School of Medicine, Aurora, CO); Ranjit Chima (Cincinnati Children’s Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, OH); Neal J. Thomas (Penn State Hershey Children’s Hospital, Penn State College of Medicine, Hershey, PA); Simon Li, Alan Pinto (Maria Fareri Children’s Hospital, Westchester Medical Center, Valhalla, NY); Christopher Newth (Children’s Hospital Los Angeles, Keck School of Medicine of USC, Los Angeles, CA); Amanda Hassinger (John R. Oishei Children’s Hospital [formerly Women and Children’s Hospital of Buffalo], University at Buffalo School of Medicine, Buffalo, NY); Kris Bysani (Medical City Children’s Hospital, Dallas, TX); Kyle J. Rehder (Duke Children’s Hospital and Health Center, Duke University School of Medicine, Durham, NC); Edward Vincent Faustino, Sarah Kandil (Yale-New Haven Children’s Hospital, Yale School of Medicine, New Haven, CT); Eliotte Hirshberg (Intermountain Medical Center, University of Utah School of Medicine, Salt Lake City, UT); Kupper Wintergerst (Norton Children’s Hospital, University of Louisville School of Medicine, Louisville, KY); Adam Schwarz (Children’s Hospital of Orange County, University of California, Irvine, School of Medicine, Orange, CA); Dayanand Bagdure (University of Maryland Medical Center, University of Maryland School of Medicine, Baltimore, MD); Lauren Marsillio (Ann & Robert H. Lurie Children’s Hospital of Chicago, Northwestern University Feinberg School of Medicine, Chicago, IL); Natalie Cvijanovich (UCSF Benioff Children’s Hospital Oakland, UCSF School of Medicine, Oakland, CA); Nga Pham (Children’s Healthcare of Atlanta, Emory University School of Medicine, Atlanta, GA); Michael Quasney, Heidi Flori (C.S. Mott Children’s Hospital, University of Michigan Medical School, Ann Arbor, MI); Myke Federman (Mattel Children’s Hospital, David Geffen School of Medicine at UCLA, Los Angeles, CA); Sholeen Nett (Dartmouth Hitchcock Medical Center, Geisel School of Medicine at Dartmouth, Lebanon, NH); Neethi Pinto (The University of Chicago Comer Children’s Hospital, Pritzker School of Medicine, Chicago, IL); Shirley Viteri (Nemours/Alfred I. duPont Hospital for Children, Sidney Kimmel Medical College at TJU, Wilmington, DE); James Schneider (Cohen Children’s Medical Center of NY, Hofstra Northwell School of Medicine, New Hyde Park, NY); Shivanand Medar (The Children’s Hospital at Montefiore, Albert Einstein College of Medicine, Bronx, NY); Anil Sapru, Patrick McQuillen (UCSF Benioff Children’s Hospital, UCSF School of Medicine, San Francisco, CA); Christopher Babbitt (Miller Women & Children’s Hospital, University of California, Irvine, Medical School, Long Beach, CA); John C. Lin (St. Louis Children’s Hospital, Washington University School of Medicine, St. Louis, MO); Philippe Jouvet (CHU Sainte-Justine, University of Montreal Faculty of Medicine, Montreal, Quebec, Canada); Ofer Yanay (Seattle Children’s Hospital, University of Washington School of Medicine, Seattle, WA); Christine Allen (The Children’s Hospital of Oklahoma, University of Oklahoma College of Medicine, Oklahoma City, OK); Peter Luckett (Children’s Medical Center of Dallas, University of Texas Southwestern Medical Center, Dallas, TX); James Fackler (The Johns Hopkins Hospital, Johns Hopkins University School of Medicine, Baltimore, MD); and Thomas Rozen (The Royal Children’s Hospital, University of Melbourne, Melbourne, Australia).
      Additional information: The e-Figures and Tables can be found in the Supplemental Materials section of the online article.

      Supplementary Data

      References

        • Williams C.N.
        • Bratton S.L.
        • Hirshberg E.L.
        Computerized decision support in adult and pediatric critical care.
        World J Crit Care Med. 2013; 2: 21-28
        • Thompson B.T.
        • Orme J.F.
        • Zheng H.
        • et al.
        Multicenter validation of a computer-based clinical decision support tool for glucose control in adult and pediatric intensive care units.
        J Diabetes Sci Technol. 2008; 2: 357-368
        • Morris A.H.
        • Orme Jr., J.
        • Truwit J.D.
        • et al.
        A replicable method for blood glucose control in critically ill patients.
        Crit Care Med. 2008; 36: 1787-1795
        • Hirshberg E.L.
        • Lanspa M.J.
        • Wilson E.L.
        • et al.
        A pediatric intensive care unit bedside computer clinical decision support protocol for hyperglycemia is feasible, safe and offers advantages.
        Diabetes Technol Ther. 2017; 19: 188-193
        • Agus M.S.
        • Wypij D.
        • Hirshberg E.L.
        • et al.
        Tight glycemic control in critically ill children.
        N Engl J Med. 2017; 376: 729-741
        • Hirshberg E.L.
        • Sward K.A.
        • Faustino E.V.
        • et al.
        Clinical equipoise regarding glycemic control: a survey of pediatric intensivist perceptions.
        Pediatr Crit Care Med. 2013; 14: 123-129
        • Macrae D.
        • Grieve R.
        • Allen E.
        • et al.
        A randomized trial of hyperglycemic control in pediatric intensive care.
        N Engl J Med. 2014; 370: 107-118
        • Macrae D.
        • Tasker R.C.
        • Elbourne D.
        A trial of hyperglycemic control in pediatric intensive care.
        N Engl J Med. 2014; 370: 1355-1356
        • Macrae D.
        • Pappachan J.
        • Grieve R.
        • et al.
        Control of hyperglycaemia in paediatric intensive care (CHiP): study protocol.
        BMC Pediatr. 2010; 10: 5
        • Agus M.S.
        • Hirshberg E.
        • Srinivasan V.
        • et al.
        Design and rationale of Heart and Lung Failure - Pediatric INsulin Titration Trial (HALF-PINT): a randomized clinical trial of tight glycemic control in hyperglycemic critically ill children.
        Contemp Clin Trials. 2017; 53: 178-187
        • Faraon-Pogaceanu C.
        • Banasiak K.J.
        • Hirshberg E.L.
        • Faustino E.V.
        Comparison of the effectiveness and safety of two insulin infusion protocols in the management of hyperglycemia in critically ill children.
        Pediatr Crit Care Med. 2010; 11: 741-749
        • Eslami S.
        • de Keizer N.F.
        • Dongelmans D.A.
        • de Jonge E.
        • Schultz M.J.
        • Abu-Hanna A.
        Effects of two different levels of computerized decision support on blood glucose regulation in critically ill patients.
        Int J Med Inf. 2012; 81: 53-60
        • Mann E.A.
        • Jones J.A.
        • Wolf S.E.
        • Wade C.E.
        Computer decision support software safely improves glycemic control in the burn intensive care unit: a randomized controlled clinical study.
        J Burn Care Res. 2011; 32: 246-255
        • Preissig C.M.
        • Hansen I.
        • Roerig P.L.
        • Rigby M.R.
        A protocolized approach to identify and manage hyperglycemia in a pediatric critical care unit.
        Pediatr Crit Care Med. 2008; 9: 581-588
        • Mack E.H.
        • Wheeler D.S.
        • Embi P.J.
        Clinical decision support systems in the pediatric intensive care unit.
        Pediatr Crit Care Med. 2009; 10: 23-28
        • Randolph A.G.
        • Clemmer T.P.
        • East T.D.
        • et al.
        Evaluation of compliance with a computerized protocol: weaning from mechanical ventilator support using pressure support.
        Comput Methods Programs Biomed. 1998; 57: 201-215
        • Randolph A.G.
        • Pronovost P.
        Reorganizing the delivery of intensive care could improve efficiency and save lives.
        J Eval Clin Pract. 2002; 8: 1-8
        • Randolph A.G.
        • Wypij D.
        • Venkataraman S.T.
        • et al.
        Effect of mechanical ventilator weaning protocols on respiratory outcomes in infants and children: a randomized controlled trial.
        JAMA. 2002; 288: 2561-2568
        • Hunt D.L.
        • Haynes R.B.
        • Hanna S.E.
        • Smith K.
        Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review.
        JAMA. 1998; 280: 1339-1346
        • Tobin M.J.
        • Jubran A.
        Meta-analysis under the spotlight: focused on a meta-analysis of ventilator weaning.
        Crit Care Med. 2008; 36: 1-7
        • Weber S.
        Clinical decision support systems and how critical care clinicians use them.
        J Healthc Inf Manag. 2007; 21: 41-52
        • Pollack M.M.
        • Patel K.M.
        • Ruttimann U.E.
        The pediatric risk of mortality III: Acute Physiology Score (PRISM III-APS): a method of assessing physiologic instability for pediatric intensive care unit patients.
        J Pediatr. 1997; 131: 575-581
        • Steil G.M.
        • Reifman J.
        Mathematical modeling research to support the development of automated insulin-delivery systems.
        J Diabetes Sci Technol. 2009; 3: 388-395
        • Zhang J.
        • Butler K.A.
        UFuRT: A work-centered framework and process for design and evaluation of information systems.
        HCI Int Proc. 2007; : 1-5
        • Nahm M.
        • Zhang J.
        Operationalization of the UFuRT methodology for usability analysis in the clinical research data management domain.
        J Biomed Inform. 2009; 42: 327-333
        • Morris A.H.
        Developing and implementing computerized protocols for standardization of clinical decisions.
        Ann Intern Med. 2000; 132: 373-383
        • Clemmer T.P.
        Computers in the ICU: where we started and where we are now.
        J Crit Care. 2004; 19: 201-207
        • Timbie J.W.
        • Damberg C.L.
        • Schneider E.C.
        • Bell D.S.
        A conceptual framework and protocol for defining clinical decision support objectives applicable to medical specialties.
        BMC Med Informa Decis Mak. 2012; 12: 93
        • Goldberg H.S.
        • Paterno M.D.
        • Grundmeier R.W.
        • et al.
        Use of a remote clinical decision support service for a multicenter trial to implement prediction rules for children with minor blunt head trauma.
        Int J Med Inform. 2016; 87: 101-110
        • Opoku-Boateng G.A.
        User frustration in hit interfaces: exploring past HCI research for a better understanding of clinicians' experiences.
        AMIA Annu Symp Proc. 2015; 2015: 1008-1017
        • Avansino J.
        • Leu M.G.
        Effects of CPOE on provider cognitive workload: a randomized crossover trial.
        Pediatrics. 2012; 130: e547-e552
        • Morris A.H.
        Computerized protocols and bedside decision support.
        Crit Care Clin. 1999; 15: 523-545
        • Lorenz R.
        • Pascual J.
        • Blankertz B.
        • Vidaurre C.
        Towards a holistic assessment of the user experience with hybrid BCIs.
        J Neural Eng. 2014; 11035007