Man vs. Machine -- A new approach to hand hygiene auditing
Description of the Call:
Join us as Dr. Jocelyn Srigley talks about her research on hand hygiene auditing. We will discuss the challenges of measuring hand hygiene compliance by direct observation and whether electronic monitoring systems may offer a potential solution WATCH: http://bit.ly/1dPQiM2
Published on: Mar 3, 2016
Transcripts - Man vs. Machine -- A new approach to hand hygiene auditing
MAN VS. MACHINE:
A NEW APPROACH TO HAND
Jocelyn Srigley, MD, MSc, FRCPC
Corporate Director, Infection Control, Provincial Health Services Authority
Medical Microbiologist, BC Children’s & Women’s Hospital
May 5, 2015
• No conflicts of interest
• Received salary support from the AMMI
• Project supported by grants from
Canada Health Infoway and the Health
Technology Exchange, and by Infonaut
Inc. and GOJO Industries Inc.
• To review methods of hand hygiene
• To present new evidence supporting the
existence of the Hawthorne effect in hand
hygiene compliance monitoring
• To review the evidence for the efficacy of hand
hygiene monitoring technology (HHMT)
• To discuss the implications of these findings,
including public reporting of hand hygiene
rates and the use of HHMT
Hand Hygiene Compliance
• Improving healthcare worker (HCW) hand hygiene
compliance is an important way to reduce healthcare-
associated infections (HAIs)
• A multifaceted approach, including measurement of
compliance, is recommended1
• Measurement options include:2
– Direct observation
– Product consumption
– Hand hygiene monitoring technology
Measuring Hand Hygiene
1WHO, 2009. 2Boyce, 2011.
• “Gold standard”
– Only method that can assess all indications for hand
– One of the few methods that can assess technique
– Provides opportunity for education
– Labour intensive
– Can only assess small samples of hand hygiene
– Questionable inter-rater reliability
– Potential for errors in measurement
2Boyce, 2011. 3Haas et al., 2007.
– Observer bias
• Unit-based observers report higher compliance
rates than non-unit-based observers4
– Selection bias
– Hawthorne effect
Sources of Measurement Error
4Dhar et al., 2010.
• Hand hygiene monitoring technology
– Electronic monitoring systems – e.g. real-
time locating systems (RTLS),
radiofrequency identification (RFID)
– Video monitoring
A New Solution?
The Hawthorne Effect
• Tendency of people to change their behavior when
they are aware of an observer3
• Original Hawthorne studies 1924-19325,6
– Concept was not mentioned until 19507
• Many authors have since questioned whether the
original Hawthorne studies actually showed a
• However, the Hawthorne effect is widely assumed to
play a role in hand hygiene behavior
The Hawthorne Effect
3Haas et al., 2007. 5Mayo, 1933. 6Roethlisberger et al., 1939. 7Adair, 1984. 8Jones, 1992.
• 17 studies of the Hawthorne effect and hand hygiene
• 6 in public washrooms:
– 90% vs. 16%9
– 77% vs. 39%10
– 90% vs. 70%11
– 91% vs. 55%12
– 90% vs. 44%13
– 79% vs. 73%14
• 2 in other non-health care settings
– Petting zoos,15 homes16
• 9 related to hospital hand hygiene compliance monitoring17-25
Does the Hawthorne Effect Exist?
9Pedersen et al., 1986. 10Munger et al., 1989. 11Edwards et al., 2002.
12Drankiewicz et al., 2003. 13Nalbone et al., 2005. 14Monk-Turner, et al., 2005.
16Erdozain et al., 2013. 16Ram et al., 2010.
• Hand hygiene compliance increased when audits were announced to units in
advance compared to when they were unannounced:
– 9.1% to 29.5%20
– 29% to 45%21
– 47.4% to 55.7%22
• Increased hand hygiene compliance on high-performing units when an overt
auditor (known to the units) was compared to a covert auditor, but not on low-
• Compliance rate as measured by medical students (44.1%) was significantly
lower than those measured by infection control nurses (74.4%) and unit HH
Studies of Hospital Audits
20Tibballs, 1996. 21Eckmanns et al., 2006. 22Maury et al., 2006. 23Kohli et al.,
2009. 24Pan et al., 2013.
• Pilot study of an RTLS
• Hand hygiene compliance was 88.9%
during audits, compared to an overall
compliance of 31.5% for the days when
the audits took place
• Limited by small sample size and lack of
controlling for potential confounders
Use of Electronic Monitoring to
Measure the Hawthorne Effect
25Cheng et al., 2011.
• Some evidence to support the existence
of a Hawthorne effect in hand hygiene
compliance, but existing studies have
• Little is known about temporal and
spatial boundaries of the Hawthorne
effect or modifying factors that may play
Summary of Literature
• To determine the magnitude of the
Hawthorne effect in hand hygiene
compliance monitoring using an
electronic monitoring system
• RTLS was installed on two multi-organ
transplant units from July 2012 to March
• Generated continuous real-time location
data via ultrasound tags worn by staff
• Measured every use of alcohol-based
hand rub (ABHR) and soap dispensers
Electronic Monitoring System
Patients, staff and
equipment wear active tags.
Active tags send location
information every few seconds
over a wireless network.
Unit Floor Plan
• Retrospective cohort study
• Cohort = dispensers on the two units
• Exposure = presence of auditor
• Outcome = hand hygiene event rate (#
dispenses per dispenser per hour)
• Auditors wore system tags to track the exact time of auditing and
their location on the units
Auditors were blinded to study hypothesis and conducted
audits as per usual practice in accordance with the Just Clean
Your Hands program
• Audits were performed 1-2 times monthly on each unit from
November 2012 to March 2013
• Number of dispenses was determined for areas within eyesight of
the auditor when he/she was in a defined location for at least 5
Separate counts for dispensers in rooms and hallways
• Count was converted into an event rate per dispenser per hour
• Area of the unit not visible to the auditor at the
same time period during the audit
– Control for confounding related to time
• Same area where the auditor was located at
1, 2, and 3 weeks prior to the audit
– Control for confounding related to location
• Same area where the auditor was located 5
minutes prior to auditor’s arrival
– Reverse causality bias
Out* In Total**
Results: Location Comparison
Out* In Total*
1 Week Prior
Results: Time Comparison
Out* In Total**
Prior to Arrival
Results: Prior to Auditors’ Arrival
• Hand hygiene event rate is ~3 times higher within
eyesight of the auditor compared to other
locations at the same time and the same location
in previous weeks
– The effect is seen only in hallway dispensers,
where the auditor can be seen, and not inside
– The increase in event rate happens after the
auditor’s arrival, not before
Summary of Results
• Observational study, therefore cannot attribute
• Measuring hand hygiene events, not HCW compliance
– To get a denominator, all HCWs would have to be
wearing system tags
• Some hand hygiene events may have been performed
by untagged HCWs or visitors
• System itself may have caused a Hawthorne effect
• Study conducted with a relatively small number of
observations on multi-organ transplant units
Efficacy of Hand Hygiene
– How accurately do electronic/video monitoring systems
(EMS/VMS) measure hand hygiene compliance?
– Limited data
– Do EMS/VMS lead to improvements in hand hygiene
– Potential mechanisms
• Real-time reminders
• Enhanced Hawthorne effect
Is Technology the Answer?
• To determine whether HHMT increases directly
observed hand hygiene compliance among HCWs
compared to usual care
• To determine whether HHMT reduces HAI incidence
or improves other measures of hand hygiene
– Hand hygiene frequency
– Volume of soap and ABHR use
– Compliance as defined by the individual HHMT [i.e.
system-defined compliance (SDC)]
• Systematic review following PRISMA guidelines26
• Searched multiple databases from inception until Dec 31,
• Eligibility criteria
– Experimental and quasi-experimental studies of HHMT
conducted in acute or long-term care that measured
hand hygiene and/or HAI incidence
– Excluded if HHMT was installed solely to evaluate
another intervention or if study focused on hand
hygiene at ward/hospital entrances or in OR
– Peer-reviewed, English language publications
• All steps in selection, data extraction and risk of bias
assessment27,28,29 performed independently by 2 authors
26Moher et al., 2009. 27Higgins et al., 2011. 28Harris et al., 2004. 29Schweizer et
Study Setting Popula-
EMS ABHR +
Room exit No Voice
with a hand
prior to or
within 10s of
P2 vs. P1:
P3 vs. P1: +5%
EMS ABHR Room
No Voice and
with a hand
P2 vs. P1:
Reminders without Feedback
Study Design Study
Medical ICU All HCW VMS ABHR +
No SDC Proportion of
with a hand
prior to or
within 10s of
time in room >
P2 vs. P1: +75.1%*
P3 vs. P1: +81.4%*
Surgical ICU All HCW VMS ABHR +
No SDC As above P1(monitoring): 30.4%
P2 vs. P1: +51.9%*
EMS ABHR Not tracked Aggregate,
No Hand hygiene
N/A Control (monitoring):
Feedback and Reminders
14 nurses EMS ABHR +
Vibration SDC, hand
with a hand
within 60s prior
to entry or 20s
prior to exit
within 20s of
reminders) : 6.61
P2 vs. P1: -0.13
P3 vs. P1: +3.64
RCT 2 wards +
231 nurses EMS ABHR Zone
Vibration SDC Proportion of
with a hand
within 6s of entry
or 60s of exit
• No studies met primary objective (directly observed
• Study at lowest risk of bias showed no clinically
significant effect of an RTLS
• VMS appear promising but studies at moderate risk of
• Insufficient evidence to recommend adoption of HHMT
as an improvement strategy
• Future trials must include stronger designs, control
groups, and system independent measures of hand
Summary of Results
• BC, Ontario, and other provinces have
made hospital hand hygiene compliance
rates publicly reportable
• Public reporting may increase the
potential for gaming and bias37
• Indicator-based improvement vs.
37Muller et al., 2011.
• BC provincial average (FY2013-14)
– 72% for moment 1 and 81% for moment 438
• Ontario provincial average (FY2013-14)
– 86% for moment 1 and 91% for moment 439
• A systematic review found a median HH
compliance rate of 40%40
• Despite the significant increase in compliance
since public reporting began in Ontario, there has
been no change in HAI rates41
38PICNet, 2015. 39Health Quality Ontario, 2015. 40Erasmus et al., 2010.
• Actual HCW hand hygiene compliance
rates are not as high as reported
– May be up to 3x lower
• Potential solutions
– Stop public reporting
– Change from direct observation to another
method of hand hygiene compliance
Direct Observation Hand Hygiene Monitoring Technology
Subject to observer and selection biases Lower likelihood of bias
Hawthorne effect May be less subject to Hawthorne effect
Questionable inter-rater reliability Consistent, algorithm-based data collection
Few observations, usually during peak
hours of patient care activity
Constant, real-time assessment of all hand
Measures all 4 moments of hand hygiene Often uses room entry/exit as denominator
Ability to assess technique Most systems unable to assess technique
Can provide feedback/education to HCWs May or may not provide feedback
Generally accepted by HCWs May be less acceptable to HCWs
Labour intensive Can be expensive to install and maintain
Can compare across facilities Unable to compare compliance rates
Comparison of Methods
• HHMT has advantages but it is not a panacea
– Institutions need to weigh costs and benefits in
their particular setting
– Ideally HHMT should be installed with the goal of
• Ongoing efforts are necessary to truly improve hand
hygiene compliance and reduce HAIs
– Focus on changing individual behaviour and
– Frontline ownership (FLO)
Dr. Michael Gardam
Dr. G. Ross Baker
Dr. Colin Furness
Systematic review co-authors:
Dr. Matthew Muller
Dr. Michael Gardam
Dr. Geoff Fernie
Dr. Gerald Lebovic
Mary Jane Salpeter
Dr. Geoff Anderson
Dr. Whitney Berta
Dr. Monique Herbert
Dr. Laura Rosella
Dr. Gerald Evans
1. World Alliance for Patient Safety. WHO guidelines on hand hygiene in health care (May 2009). Geneva, Switzerland:
World Health Organization, 2009; [cited May 27, 2013]. Available from:
2. Boyce JM. Measuring healthcare worker hand hygiene activity: current practices and emerging technologies. Infect
Control Hosp Epidemiol 2011;32(10):1016-28.
3. Haas JP, Larson EL. Measurement of compliance with hand hygiene. J Hosp Infect 2007;66(1):6-14.
4. Dhar S, Tansek R, Toftey EA, et al. Research brief: observer bias in hand hygiene compliance reporting. Infect
Control Hosp Epidemiol 2010;31(8):869-70.
5. Mayo E. The human problems of an industrial civilization. New York: MacMillan, 1933.
6. Roethlisberger FJ, Dickson WJ. Management and the worker. Cambridge, Mass.: Harvard University Press, 1939.
7. Adair JG. The Hawthorne effect: a reconsideration of the methodological artifact. J Appl Psychol 1984;69(2):334-45.
8. Jones SRG. Was there a Hawthorne effect? Am J Sociol 1992;98(3):451-68.
9. Pedersen DM, Keithly S, Brady K. Effects of an observer on conformity to handwashing norm. Percept Motor
10. Munger K, Harris SJ. Effect of an observer on handwashing in a public restroom. Percept Motor Skill 1989;69:733-4.
11. Edwards D, Monk-Turner E, Poorman S, Rushing M, Warren S, Willie J. Predictors of hand-washing behavior. Soc
Behav Personal 2002;30(8):751-6.
12. Drankiewicz D, Dundes L. Handwashing among female college students. Am J Infect Control 2003;31:67-71.
13. Nalbone DP, Lee KP, Suroviak AR, Lannon JM. The effects of social norms on male hygiene. Individual Differences
14. Monk-Turner E, Edwards D, Broadstone J, Hummel R, Lewis S, Wilson D. Another look at hand-washing behavior.
Soc Behav Personal 2005;33(7):629-34.
15. Erdozain G, KuKanich K, Chapman B, Powell D. Observation of public health risk behaviours, risk communication
and hand hygiene at Kansas and Missouri petting zoos – 2010-2011. Zoonoses Public Hlth 2013; 60:304-10.
16. Ram PK, Halder AK, Granger SP, Jones T, Hall P, Hitchcock D, et al. Is structured observation a valid technique to
measure handwashing behavior? Use of acceleration sensors embedded in soap to assess reactivity to structured
observation. Am J Trop Med Hyg 2010;83(5):1070-6.
17. Pittet D, Simon A, Hugonnet S, Pessoa-Silva CL, Sauvan V, Perneger TV. Hand hygiene among physicians:
performance, beliefs, and perceptions. Ann Intern Med 2004;141:1-8.
18. Chen LF, Carriker C, Staheli R, Isaacs P, Elliott B, Miller BA, et al. Observing and improving hand hygiene
compliance: implementation and refinement of an electronic-assisted direct-observer hand hygiene audit program.
Infect Control Hosp Epidemiol 2013;34(2):207-10.
19. Bittner MJ, Rich EC, Turner PD, Arnold WH. Limited impact of sustained simple feedback based on soap and paper
towel consumption on the frequency of hand washing in an adult intensive care unit. Infect Control Hosp Epidemiol
20. Tibballs J. Teaching hospital medical staff to handwash. Med J Aust 1996;164:395-8.
21. Eckmanns T, Bessert J, Behnke M, Gastmeier G, Rüden H. Compliance with antiseptic hand rub use in intensive care
units: the Hawthorne effect. Infect Control Hosp Epidemiol 2006;27:931-4.
22. Maury E, Moussa N, Lakermi C, Barbut F, Offenstadt G. Compliance of health care workers to hand hygiene:
awareness of being observed is important. Intens Care Med 2006;32:2088-9.
23. Kohli E, Ptak J, Smith R, Taylor E, Talbot EA, Kirkland KB. Variability in the Hawthorne Effect with regard to hand
hygiene performance in high- and low-performing inpatient care units. Infect Control Hosp Epidemiol 2009;30:222-5.
24. Pan SC, Tien KL, Hung IC, Lin YJ, Sheng WH, Wang MJ, et al. Compliance of health care workers with hand
hygiene practices: independent advantages of overt and covert observers. PLoS ONE 2013;8(1):e53746.
25. Cheng VC, Tai JW, Ho SK, et al. Introduction of an electronic monitoring system for monitoring compliance with
Moments 1 and 4 of the WHO "My 5 Moments for Hand Hygiene" methodology. BMC Infect Dis 2011;11:151.
26. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and
meta-analyses: the PRISMA statement. PLoS Med 2009;6:e1000097.
27. Higgins JP, Altman DG, Gotzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in
randomised trials. BMJ 2011;343:d5928.
28. Harris AD, Bradham DD, Baumgarten M, Zuckerman IH, Fink JC, Perencevich EN. The use and interpretation of
quasi-experimental studies in infectious diseases. Clin Infect Dis 2004;38:1586-91.
29. Schweizer ML, Reisinger HS, Ohl M, et al. Searching for an optimal hand hygiene bundle: a meta-analysis. Clin
Infect Dis 2014;58:248-59.
30. Swoboda SM, Earsing K, Strauss K, Lane S, Lipsett PA. Electronic monitoring and voice prompts improve hand
hygiene and decrease nosocomial infections in an intermediate care unit. Crit Care Med 2004;32:358-63.
31. Venkatesh AK, Lankford MG, Rooney DM, Blachford T, Watts CM, Noskin GA. Use of electronic alerts to enhance
hand hygiene compliance and decrease transmission of vancomycin-resistant Enterococcus in a hematology unit. Am
J Infect Control 2008;36:199-205.
32. Armellino D, Hussain E, Schilling ME, et al. Using high-technology to enforce low-technology safety measures: the
use of third-party remote video auditing and real-time feedback in healthcare. Clin Infect Dis 2012;54:1-7.
33. Armellino D, Trivedi M, Law I, et al. Replicating changes in hand hygiene in a surgical intensive care unit with
remote video auditing and feedback. Am J Infect Control 2013;41:925-7.
34. Marra AR, D'Arco C, Bravim BA, et al. Controlled trial measuring the effect of a feedback intervention on hand
hygiene compliance in a step-down unit. Infect Cont Hosp Epidemiol 2008;29:730-5.
35. Levchenko AI, Boscart VM, Fernie GR. The effect of automated monitoring and real-time prompting on nurses' hand
hygiene performance. Comput Inform Nurs 2013;31:498-504.
36. Fisher DA, Seetoh T, Oh May-Lin H, et al. Automated measures of hand hygiene compliance among healthcare
workers using ultrasound: validation and a randomized controlled trial. Infect Control Hosp Epidemiol 2013;34:919-
37. Muller M, Detsky A. Public reporting of hospital hand hygiene compliance—helpful or harmful? J American Med
38. Provincial Infection Control Network of British Columbia (PICNet). Hand cleaning compliance in BC acute care
facilities: fiscal year 2013/2014. Vancouver, BC: Provincial Infection Control Network of British Columbia (PICNet),
June 2014; [cited 2015-04-07]. Available from: https://www.picnet.ca/wp-content/uploads/PICNet-HCC-Annual-
39. Health Quality Ontario [homepage on the Internet]. Toronto, ON: Queen's Printer for Ontario, 2015; [cited 2015-04-
07]. Public Reporting; [background screen]. Available from: http://www.hqontario.ca/public-reporting/patient-safety.
40. Erasmus V, Daha TJ, Brug H, et al. Systematic review of studies on compliance with hand hygiene guidelines in
hospital care. Infect Control Hosp Epidemiol 2010;31(3):283-94.
41. DiDiodato G. Has improved hand hygiene compliance reduced the risk of hospital-acquired infections among
hospitalized patients in Ontario? Analysis of publicly reported patient safety data from 2008 to 2011. Infect Control
Hosp Epidemiol 2013;34:605-10.
Subtitle Goes Here
336 Clean Shots
More than 11,700 gallery visits
More than 5,700 votes
Thank you to everyone who
took part and made this
campaign a success!