Library One Search (Summon) Usability Test Brief Report date: Test date(s): Location: November 06, 2015 October 27-30, 2015 Tempe, Arizona Prepared by: Email: Tammy Wolf, Jordyn Kush tammy.wolf@asu.edu, jordyn.kush@asu.edu Executive Summary During the process of investigating ways to improve discovery and retrieval of library resources, the Discovery Services Team for the Libraries at Arizona State University conducted a research study to assess how members of the ASU community are interacting with our current discovery tool (Summon 2.0) and to what degree it is effective in supporting user needs. The "Summon" application, labeled the “Library One Search” by ASU Libraries, attempts to address the difficulties associated with the complex retrieval and navigation of the library’s disparate collections by providing a unified search interface. The study was also designed to establish recommendations for improvements to Summon’s overall design, functionality, and integration with other library applications. Methodology Who we tested A total of sixteen (16) participants, having the following characteristics, evaluated the processes within the Library One Search/Summons search tool for Arizona State University. Participants were both members and non-members of the University. Findings were not necessarily affected by this aspect as some affiliates had not previously interacted with the tool prior to testing. Audience Type Age Student (ASU affiliate) 5 18-24 11 Faculty (ASU affiliate) 2 25-31 1 Online users (non-ASU) 9 32-38 2 TOTAL (participants) 16 39-45 1 45 or older 1 TOTAL (participants) 16 What participants did Participants spent about 15-20 minutes interacting with the tool to complete or answer a range of 8-10 tasks or questions. Live participants were asked to fill out two questionnaires; (1) Demographics and (1) System Usability Score. Online participants had their experiences observations documented via screen and voice recordings, and answered three questions about their experience at the end of the test. What data we collected Task completion rates and verbal feedback were the main data collected. A System Usability Score was determined based on the information gathered from in-person participants. Findings and recommendations Summary: A combination of questions and clearly defined tasks were given to the participants. Below are sample responses to the relevant questions from live testing sessions and responses from online participants. Also included is a summarization of the tasks completed by all participants in the test. Participant questions Question Sample Responses Are you familiar with the Library One Search tool? (live participants) 3 of 7 users were completely unfamiliar. Others who were ranged from never using the tool to using it somewhat regularly. Some users noted they are overwhelmed by amount of info and desire more hierarchy/organization of results. What information do you think this would provide? (live participants) All participants correctly defined the purpose of the site. Participants used phrases from major headers and labels to describe the purpose of the tool. What frustrated you most about this tool? (online participants) • “…the whole words were not properly displayed on the "Refine your search" column and also it refreshed every time I clicked on something, not allowing me to makes changes very quickly. I had to wait a few seconds before I could change anything else.” • “Library homepage was a bit difficult to find via the ASU homepage.” How would you improve this site? (online participants) • “I would fix the Filters column so that everything is displayed properly and I would also allow changes to the filters while the results loaded.” • “I must say I am very impressed with the ASU library’s resources. For every publication there was… access to basic information and a small summary for each publication and included multiple citation methods which to a student is amazing and very helpful.” • “Add quicker access to the library’s homepage via ASU’s homepage.” • “I like that the website is so well presented and sorted. It is very detailed and professional. It is quite easy to use and understand and has a lot of options available.” • “It was getting easier to use the longer I was navigating through it.” • “Everything was super streamlined. I was able to find everything really easily in order to complete my tasks. The sidebar’s filters were very easy to use, and I loved the ability to roll over the article to get more information about it.” What did you like about the site? (online participants) Participant tasks Task 1 - Using the Library One Search tool, please look up information on “Biodiversity” Number of participants Percent successful 16 100% Findings Recommendations • 16 participants completed the task with ease. • 15 participants found the tool via the Library Homepage • 1 participant used asu.edu search “Library One” and was brought to tool’s specific page (lib.asu.edu/one) • No recommendations for this portion. All participants interacted well with this step despite source of search. Task 2 - Please point out any database recommendations relevant to this subject Number of participants Percent successful 16 43.75% Findings Recommendations • 7 of 16 participants completed the task with ease by finding the database recommendations listed above the results • 9 of 16 participants needed prompting, had significant difficulty, or failed in completing the task. Most users said they would find “Database Recommendations” to be helpful if research was being done with a specific field/intent whether they located the recommendations or not. • Finding a more clear way to differentiate the “Database Recommendations” section from the results would help users identify more clearly. Task 3 - Identify the number of Scholarly & Peer Reviewed articles are available in Full-text from the years 2000-2005. Number of participants Percent successful 16 43.75% Findings Recommendations • 7 of 16 live participants completed the task with ease by identifying/using all three facets to answer. • 7 of 16 live participants needed prompting or had significant difficulty completing the task by identifying/using at least 1 of the 3 facets • 2 of 7 were completely unsuccessful in this task and did not identify/use the facets Most live participants struggled to use the filtering facets. • Recommending in-depth test be run on use of facets to measure/track data such as gaze plots and time on task • Facet section as a whole should be more evident and salient • Facet options should not be cut off or shortened so user is clear on the option • Some users experienced difficulty/frustration with facets loading while attempting to filter (e.g. Date; inputting first date would resubmit query and users could not input end date facet until query was done). Fine tuning these interactions will help the users have a more fluid, stress-free interaction Task 4 - Identify the number of articles available in Spanish and Italian Number of participants Percent successful 16 68.75% Findings Recommendations • 11 of 16 live participants completed the task with ease by identifying/using the language facet. • 5 of 7 live participants were completely unsuccessful at identifying/using language facet. Despite most live users taking note of the facets during the test, 5 of 7 failed to locate facet at all. Online users had a 100% success rate at identifying/using the facet. • Recommendations from above apply Note: Language facet is significantly lower in the facet sidebar. However, it is proximate to the very visual “Publication date” facet. Task 5 - Locate the top Journal Article on Biodiversity, save this to your personal folder Number of participants Percent successful 16 81.25% Findings Recommendations • 13 of 16 participants completed the task by identifying/using the folder icon to the right of the search result listing. • 3 of 16 participants missed the folder icon and attempted other paths (e.g. “Save” “Export”). Most users were successful despite often being unsure that they were selecting the correct icon. • Add an identifier, letting the user know that the folder icon will save the item to their temporary folder. • Make information about folder behavior in grey box more evident • Further testing could be useful to study expectations of folder. Participants did not seem aware that the items sent to this folder would not be saved for any length of time beyond their current session Task 6 - Identify whether or not you are able to make a citation for this Journal Article Number of participants Percent successful 16 93.75% Findings Recommendations • 15 of 16 participants completed the task with ease by identifying at least one way to make a citation. • 1 participant was unable to complete the task by finding a method of citation. • No recommendations needed as most users found a way to make a citation out of the multiple locations available. • 7 participants found the citation tool via the Summon sidebar. • Other areas located were: - Temp folder - 360link sidebar - Within the detailed page information for some results Note: Accuracy of citations is key or users will not take the time to interact with this feature. Several users pointed out that certain citations were incomplete/inaccurate Task 7 – Select the Journal Article to pull up more detailed information (360link sidebar). Is this information open access? What is the publication frequency of the journal? Number of participants Percent successful 16 43.75% Findings Recommendations • 7 of 16 participants completed the task by identifying both pieces of information in the sidebar. • 4 of 16 participants struggled to identify both pieces of information in the sidebar • 5 of 16 participants were completely unsuccessful at identifying both pieces of information in the sidebar. • Further testing may provide useful, detailed feedback in the hierarchy of information and use of the 360link sidebar. Many users looked at the tool but the information was not salient. Note: The term “open access” was not salient and may have cause confusion among some participants. Locating the publication frequency is a clearer indicator of interaction with the 360link sidebar. • 8 of 9 participant completed the task by locating the publication frequency Summary Overall, tests were very successful and informational. M users were able to find their way around the Summon interface with a fair amount of ease. Our live participants took part in a questionnaire to determine a System Usability Score (SUS). The th interface received a score of 66.78, with 68 being an “average” score (or 50 percentile). The SUS is an industry standard method for gauging the usability of a given product/service. To read more on this, please visit the usability.gov website. In conclusion, the findings show the interface is operable but there is room for improvement. We suggest minor changes be considered to enhance the site with the intent of making it more user-friendly and intuitive. To add, we believe further in-depth analysis could continue to benefit both users and Summon.