WK Lee, BWY Young

Abstract

This prospective observational study aimed to measure the internal consistency and inter-rater agreement of the modified COMFORT scale for mechanically ventilated patients in a neonatal intensive care unit. A sample of 15 infants requiring mechanical ventilation, generating a total of 27 observations, was recruited. Three different observers including a doctor, a nurse and a senior nurse independently assessed the sedation level of each patient. The measurement tool was modified from the original COMFORT scale. The Spearman's correlation coefficients for the inter-rater agreement between doctors and senior nurses, doctors and nurses as well as nurses and senior nurses on the total scores were 0.760, 0.626 and 0.776 respectively, all p<0.05. The reliability coefficient alpha for the internal consistency was 0.804. We suggest that the modified COMFORT scale may be an appropriate tool to observe sedation level for mechanically ventilated infants. More studies may be required for validation and outcome assessment.

Introduction

The importance of adequate sedation and analgesia in critically ill patients was well recognised. The administration of sedation for anxiety and agitation was recommended in intensive care unit (ICU).¹ Mechanical ventilation was one of the common causes of discomfort encountered in the ICU that might induce anxiety and ventilator dys-synchrony. Despite the advances in ventilator technology, the demand for adequate sedation still existed.² Potential benefits of sedatives for mechanically ventilated patients included reduction of oxygen consumption and stress response, improved ventilation and reduced risk of barotrauma, improved patient care and comfort.³ Unplanned extubation was also found to be associated with under-sedation.^4-9 On the other hand, potential risks of sedative medications were present. They included cardiovascular effects and drug interaction which might be more prominent in patients with multiorgan failure, especially those with hepatic and renal failure.³ Moreover, the use of continuous intravenous sedation might be associated with the prolongation of mechanical ventilation and ICU stay^10,11 as well as other consequences such as physical dependence, tolerance and withdrawal.¹² The use of sedation was more controversial in neonates because of the debate of their potential benefits to reduce complications such as intra-ventricular haemorrhage, pneumothorax and the stress responses against the potential complications.¹³ Recently, health care professionals have now realised that agitation and stress have adverse effects on neonatal well being.¹³ The practice of administering sedation to infants requiring mechanically ventilation is now generally accepted. However, determination of an optimal sedation level and quantitative measurement of this qualitative endpoints become the new challenge.^13,14 Various scoring systems have been used in the development of practice guidelines.² A multidimensional scale, the COMFORT scale, was studied for critically ill paediatric patients.¹⁵ This study was performed to analyse the application of this scale to neonatal population including premature infants.

Methods

The study was performed in our neonatal ICU (NICU) consisting of five beds in 2002. Before the commencement of the study, sedation level of the infants on mechanical ventilator in our NICU was observed by subjective means. To prepare for the study, the aim and observation method was introduced to the medical and nursing staff. A pilot study was also conducted beforehand.

Convenient sample of patients of any post-conceptional age and body weight was recruited for assessment if they were intubated and mechanically ventilated. Exclusion criteria included: 1) unstable cardiovascular status; 2) significant neurological impairment; 3) paralysed with neuromuscular blockers or underlying neuromuscular disorders. Demographic data, indication of ventilation and the dose of the sedative medication were recorded. The patients were each assessed by three independent observers including a doctor, a nurse and a senior nurse working in the NICU. All of them were not the direct caretakers of study subjects. The assessment might be performed for more than once for each patient on different days.

The measurement tool was modified from the original COMFORT Scale.¹⁵ After the modification, it consisted of seven variables: alertness, calmness, respiratory response, movement, mean arterial pressure, heart rate and facial expression (Table 1). A score from 1 to 5 would be assigned to each variable according to different manifestations. The minimum and maximum of total scores were 7 and 35 respectively. A smaller score indicated a deeper sedation level.

Before each assessment, the observers would make an agreement on the baseline heart rate (HR) and blood pressure (BP) by reviewing the observation chart in the prior six hours. The observation was then performed independently by each observer at the same time. Eye-shields if present for phototherapy were removed for observation of the facial expression. BP was measured with oscillometric method if there was not a direct measurement via arterial line.

Inter-rater agreement between any combinations of paired observers was analysed with Spearman's correlation coefficient. Internal consistency of the variables was analysed with reliability coefficient alpha, item-total correlation and alpha if item deleted. P<0.05 was regarded as statistically significant. SPSS 11.0 was used for statistical analysis.

Results

Fifteen patients were recruited. The primary indications for mechanical ventilation were: respiratory distress syndrome (6), pneumonia (3), bronchopulmonary dysplasia (2), meconium aspiration syndrome (1), apnoea (1), sepsis (1) and birth asphyxia (1). Each infant was assessed for one to six times. A total of 27 episodes of assessment were performed. At the time of assessment, the post-conceptional age ranged from 23 to 54 weeks and the body weight ranged from 560 to 3740 grams. Among these 27 episodes, no sedative medication was given in the prior four hours in 15 instances. In the remaining 12 episodes, fentanyl infusion was given at a rate from 1.2 to 4 μg/kg/hour. Midazolam infusion at a rate from 0.02 to 0.2 mg/kg/hour in addition to fentanyl was given in four episodes.

For individual item, the mean score ranged from 1.33 to 2.10 whereas the minimal and maximal score were 1 and 5 respectively. For the total score, the mean, minimal and maximal values were 13.12, 7 and 25 respectively (Table 2).

The Spearman's correlation coefficients for the inter-rater agreement between doctors and senior nurses, doctors and nurses as well as nurses and senior nurses on the total scores were 0.760, 0.626 and 0.776 respectively, all p<0.05 (Figures 1-3). The agreements on the individual items are shown in the Table 3. The reliability coefficient alpha for all the items was 0.804. The alpha for each individual item- total correlation ranged from 0.38 to 0.71. The individual alpha if item deleted ranged from 0.74 to 0.83 (Table 4).

Figure 1 Correlation of total score between doctors and senior nurses.

Figure 2 Correlation of total score between doctors and nurses.	Figure 3 Correlation of total score between nurses and senior nurses.

Discussion

At least 20 different tools¹⁶ for assessing sedation levels in adults was studied since Ramsay scale was first developed in 1974.¹⁷ The Ramsay scale, which has been validated in at least three studies involving 177 patients and more than 1100 assessments, was applied in at least 20 studies. The scale is simple to use but the single item used in the scoring system may lead to loss of clinically significant information and measurement error.¹⁶ In addition, it may not be applicable to paediatric patients. For paediatric patients, at least five different sedation scoring systems were developed.^15,16,18-20 However, all the studies were performed in a wide age range from neonates to older children. The gestational age of the neonates was not specified in the studies. On the other hand, scoring systems for measuring pain in neonates were extensively studied but they were not primarily used to measure sedation level in neonates who were mechanically ventilated.^21-28 Mechanical ventilation is a common intervention for critically ill neonates including premature babies. It may be necessary to develop a sedation score for this group of patients.

Among the existing scoring systems, there was no data to demonstrate which one was superior.¹⁶ We selected the COMFORT scale because of several reasons. Firstly, the number of items was not as large as in some studies such as the Vancouver Sedative Recovery Scale (VSRS) that consisted of 12 items. In addition, only patients after trauma or surgery were recruited in that study.¹⁸ Secondly, as a distinctive feature from other systems, more physiological variables such as heart rate and blood pressure were introduced in this scale. We thought, before the result analysis, that the observation might then be more objective. Thirdly, the results of inter-rater agreement for the total score and the internal consistency were also satisfactory.¹⁵ Lastly, the scale has also been validated.²⁹

However, we made a modification on the original scale by deleting the item of 'muscle tone' because of three reasons. Firstly, the result of internal consistency of this item was found to be the worst in the original study. Secondly, the alpha coefficient of the total COMFORT scale was the largest when this item was deleted. This implied that the overall performance of the scale may improve after the deletion of this item. Thirdly, the assessment of the muscle tone should involve handling the patients, which may then increase the risk of nosocomial infection. Hand washing was otherwise required and more time would then be necessary for each assessment.

There was another difference in the study design of the current study from the original COMFORT study. We included infants in the NICU only whereas infants from newborn to aged 204 months were recruited in the original study.

There were some differences when comparing our results to the original study. Firstly, the alpha coefficient for the internal consistency in our study was smaller (0.80 vs. 0.90). Alpha coefficient measures the average of reliability of all individual items in a scale. A greater value would indicate a greater overall reliability of the scale. Secondly, the Pearson correlation coefficient for inter-rater agreement of the original study was 0.84. For our study, we applied a non-parametric test namely Spearman's correlation coefficient because of our small sample size. Nevertheless, the results were still quite satisfactory.

Surprisingly, the inter-rater agreement for two physiological items, i.e., mean arterial pressure and heart rate, were not very great (Table 3). It was expected to be good since these parameters were apparently more objective. However, subjective impression to score these two items still existed for the terms like 'infrequent', 'frequent' and 'sustained', etc. The assessor might also unreliably estimate 'elevation of 15% or more'. In addition, the internal consistency of these two items was not as satisfactory as expected. The item-total correlation, which measures the correlation of each individual item to others, was 0.38 and 0.51 only for mean arterial pressure and heart rate respectively (A smaller value would indicate a smaller correlation). Alpha if item deleted, which measures the overall internal consistency of the scale if the individual item is deleted, was greatest (0.83) for mean arterial pressure (A greater value would indicate that the reliability of that item is smaller) (Table 4). These results imply that the physiological variables might not be as reliable as originally thought. However, this result was similar to a study performed in 18 mechanically ventilated infants and children³⁰ as well as to another study performed in 78 infants and children with or without ventilation,³¹ in which the correlation or reliability of these two items was the worst. It was postulated that these physiological responses were affected by haemodynamic disturbances and might be independent of comfort levels.³⁰

There were several limitations in our study. Firstly, the patients were recruited by convenient method only. This method was chosen because of the limitation of our existing setting so as to minimise the effect to the patients' care. Bias was possible in the selection of patients. Indeed the scores of individual items as well as the total scores tended to be small. The current result might not reflect the real situation when the patients were actually poorly sedated. Secondly, we did not perform a 'validation' for the current study. It might be difficult to identify a gold standard for validation. As mentioned before, other scoring systems such as the Ramsay score had their own limitations. Observation by an 'expert' as the gold standard as applied in some studies might be too subjective even he or she was experienced with this aspect. Before the use of COMFORT scale, there was no standard practice among our staff and the assessment of sedation relied on individual knowledge, skill and experience. In the future, the validation study of any scoring system may require an objective tool such as the bispectral index.³²

Although the sample size of the current study was small, it served the purpose to guide the future application in mechanically ventilated neonates including those were premature. After proposing a standard tool and skill, application in patients may be tried and further protocol may then be developed for different interventions like adjusting sedation medication according to the different scores. Future studies may be considered on the effect of patients' outcomes such as duration of mechanical ventilation and episodes of unplanned extubation, etc. that have been performed in adult patients.³¹

References

1. Shapiro BA, Warren J, Egol AB, et al. Practice parameters for intravenous analgesia and sedation for adult patients in the intensive care unit: an executive summary. Society of Critical Care Medicine. Crit Care Med 1995;23:1596-600.

2. Tung A, Rosenthal M. Patients requiring sedation. Crit Care Clin 1995;11:791-802.

3. Mazzeo AJ. Sedation for the mechanically ventilated patient. Crit Care Clin 1995;11:937-55.

4. Brown MS. Prevention of accidental extubation in newborns. Am J Dis Child 1988;142:1240-3.

5. Little LA, Koenig JC Jr, Newth CJ. Factors affecting accidental extubations in neonatal and pediatric intensive care patients. Crit Care Med 1990;18:163-5.

6. Maguire GP, DeLorenzo LJ, Moggio RA. Unplanned extubation in the intensive care unit: a quality-of-care concern. Crit Care Nurs Q 1994;17:40-7.

7. Chevron V, Menard JF, Richard JC, Girault C, Leroy J, Bonmarchand G. Unplanned extubation: risk factors of development and predictive criteria for reintubation. Crit Care Med 1998;26:1049-53.

8. Betbese AJ, Perez M, Bak E, Rialp G, Mancebo J. A prospective study of unplanned endotracheal extubation in intensive care unit patients. Crit Care Med 1998;26:1180-6.

9. Boulain T. Unplanned extubations in the adult intensive care unit: a prospective multicenter study. Association des Reanimateurs du Centre-Ouest. Am J Respir Crit Care Med 1998;157(4 Pt 1):1131-7.

10. Kollef MH, Levy NT, Ahrens TS, Schaiff R, Prentice D, Sherman G. The use of continuous i.v. sedation is associated with prolongation of mechanical ventilation. Chest 1998;114:541-8.

11. Kress JP, Pohlman AS, O'Connor MF, Hall JB. Daily interruption of sedative infusions in critically ill patients undergoing mechanical ventilation. N Engl J Med 2000;342:1471-7.

12. Tobias JD. Tolerance, withdrawal, and physical dependency after long-term sedation and analgesia of children in the pediatric intensive care unit. Crit Care Med 2000;28:2122-32.

13. Alexander SM, Todres ID. The use of sedation and muscle relaxation in the ventilated infant. Clin Perinatol 1998;25:63-78.

14. Park G, Coursin D, Ely EW, et al. Commentary. Balancing sedation and analgesia in the critically ill. Crit Care Clin 2001;17:1015-27.

15. Ambuel B, Hamlett KW, Marx CM, Blumer JL. Assessing distress in pediatric intensive care environments: the COMFORT scale. J Pediatr Psychol 1992;17:95-109.

16. De Jonghe B, Cook D, Appere-De-Vecchi C, Guyatt G, Meade M, Outin H. Using and understanding sedation scoring systems: a systematic review. Intensive Care Med 2000;26:275-85.

17. Ramsay MA, Savege TM, Simpson BR, Goodwin R. Controlled sedation with alphaxalone-alphadolone. Br Med J 1974;2:656-9.

18. Macnab AJ, Levine M, Glick N, Susak L, Baker-Brown G. A research tool for measurement of recovery from sedation: the Vancouver Sedative Recovery Scale. J Pediatr Surg 1991;26:1263-7.

19. Hartwig S, Roth B, Theisohn M. Clinical experience with continuous intravenous sedation using midazolam and fentanyl in the paediatric intensive care unit. Eur J Pediatr 1991;150:784-8.

20. Hughes J, Gill A, Leach HJ, et al. A prospective study of the adverse effects of midazolam on withdrawal in critically ill children. Acta Paediatr 1994;83:1194-9.

21. Grunau RV, Craig KD. Pain expression in neonates: facial action and cry. Pain 1987;28:395-410.

22. McIntosh N, Van Veen L, Brameyer H. The pain of heel prick and its measurement in preterm infants. Pain 1993;52:71-4.

23. Hodgkinson K, Bear M, Thorn J, Van Blaricum S. Measuring pain in neonates: evaluating an instrument and developing a common language. Aust J Adv Nurs 1994;12:17-22.

24. Rushforth JA, Levene MI. Behavioural response to pain in healthy neonates. Arch Dis Child Fetal Neonatal Ed 1994;70:F174-6.

25. Guinsburg R, Kopelman BI, Anand KJ, de Almeida MF, Peres Cde A, Miyoshi MH. Physiological, hormonal, and behavioral responses to a single fentanyl dose in intubated and ventilated preterm neonates. J Pediatr 1998;132:954-9.

26. Quinn MW, de Boer RC, Ansari N, Baumer JH. Stress response and mode of ventilation in preterm infants. Arch Dis Child Fetal Neonatal Ed 1998;78:F195-8.

27. Jain A, Rutter N. Does topical amethocaine gel reduce the pain of venepuncture in newborn infants? A randomised double blind controlled trial. Arch Dis Child Fetal Neonatal Ed 2000;83:F207-10.

28. Debillon T, Zupan V, Ravault N, Magny JF, Dehan M. Development and initial validation of the EDIN scale, a new tool for assessing prolonged pain in preterm infants. Arch Dis Child Fetal Neonatal Ed 2001;85:F36-41.

29. Brunow de Carvalho W, Lucas da Silva PS, Paulo CS, Fonseca MM, Belli LA. Comparison between the Comfort and Hartwig sedation scales in pediatric patients undergoing mechanical lung ventilation. Sao Paulo Med J 1999;117:192-6.

30. Carnevale FA, Razack S. An item analysis of the COMFORT scale in a pediatric intensive care unit. Pediatr Crit Care Med 2002;3:177-80.

31. Ista E, van Dijk M, Tibboel D, de Hoog M. Assessment of sedation levels in pediatric intensive care patients can be improved by using the COMFORT "behavior" scale. Pediatr Crit Care Med 2005;6:58-63.

32. Fraser GL, Riker RR. Monitoring sedation, agitation, analgesia, and delirium in critically ill adult patients. Crit Care Clin 2001;17:967-87.