documentsn.com

The testing administrator was John Fellian, Product Owner, Team Leader, and developer of CDEat Iatric Systems. As the Product Owner and Lead developer on the Clinical Document Exchange(CDE) product, John has the most intimate knowledge of the functionality of the UI and how itwas designed to be used. John has been the product owner for CDE at Iatric Systems sinceMarch of 2017. He has 12 years of experience in his position with Iatric Systems and has aBachelor of Science Degree from Northeastern University.The data logger and Meaningful Use expert was Jeremy Blanchard. Jeremy is the Director ofAnalytics at Iatric Systems and holds a Master of Science degree in Health Informatics fromNortheastern University. He has held positions of developer and product owner for variousproducts during his 11 years at Iatric Systems.Each participant reviewed and signed an informed consent and release form (See Appendix 3).The administrator moderated the session including administering instructions and tasks. Theadministrator also obtained post-task rating data and took notes on participant comments. Asecond person served as the data logger and monitored task times, took notes on task success,path deviations, number and type of errors, and comments.Participants were instructed to perform the tasks (see specific instructions below): As quickly as possible making as few errors and deviations as possible. Without assistance; administrators could give immaterial guidance and clarification on tasks, butnot instructions on use. Without using a think aloud technique.For each task, the participants were given a written copy of the task. Task timing began once theadministrator finished reading the question and verbalized, “Start.” The task time was stoppedonce the participant indicated they had successfully completed the task and verbalized, “Done”,or the data filing process began. Scoring is discussed below in Section 3.9.Following the session, the administrator gave the participant 2 post-test questionnaires (the FinalQuestions survey and the System Usability Scale, see Appendix 5 and 6 respectively), andthanked everyone individually for their participation. Participants' demographic information, tasksuccess rate, time on task, errors, deviations, verbal responses, and post-test questionnaire wererecorded into a spreadsheet.

3.5 Test LocationTesting was conducted remotely over an internet-based video conference. Participants used theirpersonal workspaces and assumed keyboard and mouse control of an Iatric PC for the purposesof performing the testing steps. The administrator and data logger were able to see the desktopof the PC, the participants face via webcam, and audio via conference line. Each environmentwas of a minimal noise volume, comfortable to the user, and free of distractions.3.6 Test EnvironmentThe EHRUT would be typically used in a healthcare office or facility. In this instance, the testingwas conducted over a GoToMeeting video conference. For testing, the participants assumedkeyboard and mouse control of an Iatric owned Dell OptiPlex 7740 desktop computer runningWindows 10. The participants used a wireless mouse and a keyboard when interacting with theEHRUT.The EHRUT was used on a 24-inch screen, with 1920/1080 resolution and in conjunction with abuilt-in webcam. The application was set up according to the product owner’s documentationdescribing the system set-up and preparation. The application itself was running on a platformusing a training / test database on a LAN / WAN connection. Technically, the system performance(i.e., response time) was representative to what actual users would experience in a fieldimplementation. Additionally, participants were instructed not to change any of the default systemsettings (such as control of font size).3.7 Test Forms and ToolsDuring the usability test, various documents and instruments were used, including:1. Informed Consent & Nondisclosure Form2. Moderator’s Guide3. Post-test Questionnaire4. System Usability Scale (SUS) Survey5. Demographics SurveyExamples of these documents are found in Appendices 3-7 respectively. The Moderator’s Guidewas devised to capture required data. The participant’s interaction with the EHRUT was capturedand recorded digitally with screen capture software running on the test machine. A web camerarecorded each participant’s facial expressions synced with the screen capture, and verbal

comments were recorded with a microphone. The test moderator later analyzed the videorecordings.3.8 Participant InstructionsThe administrator reads the following instructions aloud to each participant (also see the fullmoderator’s guide in Appendix [B4]):I want to thank you for participating in this study. Your input is very important. Oursession today will last no more than 60 minutes.In this study, we are evaluating the usability of our clinical reconciliation system. I willprovide a brief tutorial on how to use the system, then ask you to perform 4 tasks andcomplete some surveys afterwards. We would like to record this session using awebcam and screen recording device. All information you provide will be keptconfidential and your name will not be associated with your comments at any time.Following the procedural instructions, participants were shown the EHR. The administrator gavethem task packets and the following instructions:Now we'll begin the study. This part will be recorded. You'll be asked to complete fourtasks. I will read the scenario, then say "please start". Please begin working the task,feel free to perform whatever steps you feel necessary to complete the task as quicklyand easily as possible. Due to the nature of the test, we are not allowed to guide you orgive feedback to you during the test. Remember, our goal is to observe how you wouldcomplete the tasks on your own. You may ask specific questions regarding informationthat may be missing, but I cannot guide or direct you on how to complete a task.When you have completed a task, say "done" out loud. I will ask you to rate the ease ofcompletion for the task on a scale of 1 to 5. We will then move on to the next task andrepeat until we've completed all 4 tasks.Do you have any questions before we start?

Participants were then shown the 4 tasks to complete. Tasks were displayed on the screen oneat a time next to the EHR window, so the user could easily review the task instructions. After atask was completed the next task’s instructions were displayed. Tasks are listed in themoderator’s guide in Appendix 4.3.9 Usability MetricsAccording to the NIST Guide to the Processes Approach for Improving the Usability of ElectronicHealth Records, EHRs should support a process that provides a high level of usability for allusers. The goal is for users to interact with the system effectively, efficiently, and with anacceptable level of satisfaction. To this end, metrics for effectiveness, efficiency and usersatisfaction were captured during the usability testing. The goals of the test were to assess:1. Effectiveness of CDE by measuring participant success rates and errors2. Efficiency of CDE by measuring the average task time and path deviations3. Satisfaction with CDE by measuring ease of use ratingsDATA SCORINGThe following table (Figure 3) details how tasks were scored, errors evaluated, and the time dataanalyzed.Figure 3 – Observed Data Scoring DetailsMeasuresEffectiveness:Rationale and ScoringA task was counted as a “Success” if the participant was able to achieve the correctoutcome, without assistance, within the time allotted on a per task basis.Task SuccessThe total number of successes were calculated for each task and then divided by thetotal number of times that task was attempted. The results are provided as apercentage.Task times were recorded for successes. Observed task times divided by the optimaltime for each task is a measure of optimal efficiency.Optimal task performance time, as benchmarked by expert performance underrealistic conditions, is recorded when constructing tasks. Target task times used fortask times in the Moderator’s Guide must be operationally defined by taking multiplemeasures of optimal performance and multiplying by some factor [e.g., 3] that allowssome time buffer because the participants are presumably not trained to expertperformance. Thus, if expert, optimal performance on a task was [30] seconds thenallotted task time performance was [x * 3] seconds. This ratio should be aggregatedacross tasks and reported with mean and variance scores.

Effectiveness:Task FailuresIf the participant abandoned the task, did not reach the correct answer or performed itincorrectly, or reached the end of the allotted time before successful completion, thetask was counted as a “Failures.” No task times were taken for errors.The total number of errors was calculated for each task and then divided by the totalnumber of times that task was attempted. Not all deviations would be counted aserrors. This should also be expressed as the mean number of failed tasks perparticipant.On a qualitative level, an enumeration of errors and error types should be collected.Efficiency:TaskDeviationsThe participant’s path (i.e., steps) through the application was recorded. Deviationsoccur if the participant, for example, went to a wrong screen, clicked on an incorrectmenu item, followed an incorrect link, or interacted incorrectly with an on-screencontrol. This path was compared to the optimal path. The number of steps in theobserved path is divided by the number of optimal steps to provide a ratio of pathdeviation.It is strongly recommended that task deviations be reported. Optimal paths (i.e.,procedural steps) should be recorded when constructing tasks.Efficiency:Task TimeSatisfaction:Task RatingEach task was timed from when the administrator said “Begin” until the participantsaid, “Done.” If he or she failed to say “Done,” the time was stopped when theparticipant stopped performing the task. Average time per task was calculated foreach task. Variance measures (standard deviation) were also calculated.Participant’s subjective impression of the ease of use of the application wasmeasured by administering both a simple post-task question as well as a postsession questionnaire. After each task, the participant was asked to rate “Overall, thistask was:” on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averagedacross participants.Common convention is that average ratings for systems judged easy to use shouldbe 3.3 or above.To measure participants’ confidence in and likeability of the CDE product overall, thetesting team administered the System Usability Scale (SUS) post-test questionnaire.Questions included, “I think I would like to use this system frequently,” “I thought thesystem was easy to use,” and “I would imagine that most people would learn to usethis system very quickly.” See full System Usability Score questionnaire in Appendix6.

4. Results4.1 Data Analysis and ReportingThe results of the usability test were calculated according to the methods specified in theUsability Metrics section above. If able to obtain additional resources (such as more testing timeor more usability team resources), it is preferable to test with participants on an individual basis.This will decrease the stress of perceived competition and eliminate interaction betweenparticipants. The usability testing results for the EHRUT are detailed in the Execute Summary onpg. 4 (see Table 1).The results should be seen considering the objectives and goals outlined in Section 3.2 StudyDesign. The data should yield actionable results that, if corrected, yield material, positive impacton user performance. The results from the SUS (System Usability Scale) scored the subjectivesatisfaction with the system based on performance with these tasks to be: 68.75. Broadlyinterpreted, scores under 60 represent systems with poor usability; scores over 80 would beconsidered above average.4.2 Discussion of the findingsTo determine the success of CDE, the usability team analyzed the data while keeping severalareas in mind: effectiveness, efficiency, satisfaction and areas for improvement. To gatherconclusions about the effectiveness of CDE, the team analyzed success and failure rates oftasks. To look at efficiency, the task time and deviation data was noted and interpreted. Formeasures of satisfaction, the SUS scores and reported task ratings were analyzed. The teamdetermined areas for improvement by taking note of verbalizations from participants, analyzingthe Final Questions survey data for written comments, and relying on observations from theadministrator and data loggers.EffectivenessBased on the success, failure and path deviation data, CDE can be described as effective. CDEtested well in terms of effectiveness; one out of four tasks had a 100% success rate.Task 1 had a user request a restart due to not being able to see the full GoToMeeting screen andbeing at a disadvantage. We counted the first attempt at Task 1 for this user as a failure because

the screen issue was not brought to the administrator’s attention until after the timer had alreadystarted.Task 2 had a high error rate due to the user’s interpretation of the scenario and was agreedamong the testing team to be slightly ambiguous and could cause an error depending on how theuser determined the data to be “medically relevant”. The testing team did not update the script,so that all users were given the exact same opportunity for success.Task 4 had 2 users that needed to restart the task due to exiting from the screen accidently andneeding to reset the scenario. The first attempts were counted as failures.Due to the reconciliation functionality being new to all users who participated in the study someusers took slightly longer then the allotted task time. We retained these times because as theusers progressed through the task and their comfort level rose we saw progressively faster times.All users stated that the initial attempt is a little intimidating, but as you progress through the tasksthe system becomes familiar quickly and easier to use.EfficiencyBased on the observations of task time and deviation data, this usability test helped the teamdetermine that CDE can be considered efficient, but there is room for improvement. For instance,most of our optimal task times were 45-90 seconds on task. Most tasks were completed in lesstime than anticipated. Tasks 2, 3, and 4 are repetitive tasks requiring the user to perform similaractions to the preceding task. By the time the users got to Task 4, 6 of 10 users were able tocomplete the Task in 45 seconds or less, which greatly exceeded expectations.In terms of average task success, even with our small data set of 10 participants, most performedwell. Our participants had a 100% success rate with Task 3. Task 1 and Task 4 would have seena 100% success rate as well, but 1 users had a non-product related issue and 2 users of Task 4exited the screen which required a restart of the task.We plan to account for some of the issues experienced by users that led to confusion orprevented a user from being able to successfully complete a task on their first attempt.No users of the software had previous experience with the reconciliation functionality prior to theusability test and this led to some confusion and higher task times for some users.

SatisfactionBased on the task ratings and SUS results data, the participants were generally satisfied with theusability of CDE. The SUS score of 68.75 indicates that the participants perceived CDE asusable, but with room for improvement to become intuitive and more user-friendly. Most usersscored the individual task as either Easy or Very Easy.While task success rates may differ from this perceptive data, the usability team finds that overall,the participants were satisfied with most tasks and did not per

EHR Usability Test Report of Clinical Document Exchange version 1.5 Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports Product Under Test: Clinical Document Exchange version 1.5 Date of Usability Test: August 28, 2018 Date of Report: September 7, 2018