Background: Effective infection prevention and control is integral to the delivery of safe patient care, and with ever increasing demands on their attention, systems to help navigate policy-based guidance can support infection preventionists (IPs) and their healthcare colleagues. Here we present the validation process and results for an internal infection control chatbot. Methods: Eleven frequently asked questions were selected to evaluate the performance of ChatUCM, an AI-powered chatbot. Each question was posed five times and response quality was scored by intent recognition, relevance, accuracy, completeness, and consistency. A category pass threshold was set at 75%. IP feedback was incorporated in a ChatUCM update and response quality was scored again. Reference document edits and real-time prompt modifications were made based on post-update scores. A final round of scoring was performed using similar validation metrics. Two IPs scored a subset of five out of the eleven frequently asked questions to assess interrater reliability. Six nursing managers were invited to ask ChatUCM any questions and score its responses by intent recognition, their ability to trust response accuracy, whether their question was fully answered, the achievement of intended goals, and ease of use. 29 questions were submitted. An IP reviewer also scored these ChatUCM responses as optimal, suboptimal/ no harm, and suboptimal/ possible harm. Results: The impact of the update on scores was variable. ChatUCM performed well on the same 7 out of 11 questions before and after validation round 1 (Fig. 1). After the final round of retraining, ChatUCM passed all categories with high interrater reliability for our frequently asked questions (Fig. 2). Response quality was less predictable outside of these eleven questions (Fig 3, Fig 4). To ensure its safe use, multiple safety checks were created, including disclaimer language, citation links, education, and a query report to facilitate audits. We encountered multiple surprises while validating ChatUCM. The first update resulted in significant improvements, such as the ability to digest data in table and flow diagram formats. It also saw category scores decrease for six questions, three of which had overall scores drop. ChatUCM struggled with accuracy for questions that required accounting for multiple clinical factors (ex: shingles guidance). Conclusions: These findings suggest that ChatUCM could substantially reduce IP workload by accurately answering questions contained in internal SOPs and policies. Additional development will be needed to improve reliability before the tool can function independently without human oversight, particularly when multiple clinical factors must be taken into account.
Background: The terminal cleaning procedure for disinfection of a hospital room is an essential, yet difficult to monitor, step in infection prevention. Current methods for auditing require intensive human resources investment, depend on self-report, or employ spot checks using ATPase or glo gel audits. Inconsistent feedback and self-report are notoriously poor ways to drive behavior change or evaluate quality. Methods: We assessed a new AI platform developed by Myna Technologies that monitors cleaning and disinfection. The system incorporates thermal and optical cameras to detect percentage of total surface area cleaned including adequate contact time for two high touch surfaces: bedside table and mattress, in our Environmental Services (EVS) training suite. A researcher, trained by our hospital’s EVS team, performed 20 cleaning passes each for the mattress and bedside table. Each pass covered 25%, 50% 75% or 100% of total surface area; we compared the planned cleaning percentage to the device-observed results for both cameras. A Fisher’s exact test analysis was performed for fully clean (100% surface) and not clean Results: See Tables 1 and 2 for cleaning plan and device results. The mattress was 100% cleaned 6 times of 20 passes. The thermal camera correctly identified complete clean 100% of the time Discussion: Initial assessment of this novel AI technology to monitor disinfection in real time shows promise for detecting adequate cleaning. A formal validation trial comparing results of the automated system with direct observation, glo-gel marking and ATPase for all high touch surfaces, utilizing multiple cleaners is underway.
