Webinar #3: Being a Better Ally to All
October 16, 2019

Evaluation Results
AAHSL/NNLM/MLA Diversity, Equity, and Inclusion Webinar Series


2019 - 2020


Prepared by
June Lee, Sc.D., Evaluation Specialist
Serene Myers, MPH, Evaluation Specialist
National Network of Libraries of Medicine (NNLM) Evaluation Office
University of Washington
Health Sciences Libraries
Seattle, WA  98195-7155



 Funded by the National Library of Medicine under Grant No. UG4LM012343 with the University of Washington.

Executive Summary


1. Evaluation Summary

The third of 9 webinars, Being a Better Ally to All, was hosted on October 16, 2019. The class topics included improving listening skills during difficult conversations or conflict, asking better questions through conscious and unconscious bias, and taking responsibility for who and how a person shows up in relationships with key stakeholders, leaders, colleagues, etc.

The class evaluation survey was modified from the existing NNLM training evaluation form to include class-specific learning outcome questions. An evaluation link was provided to the class attendees on October 16, 2019. As the survey remains open for people seeking Continuing Education (CE) credit, this report included the surveys completed from October 16th, 2019 through October 24th, 2019.  A total of 240 people attended the class and 104 surveys were completed with a response rate of 43 percent. 

Survey data were subsequently downloaded from REDCap[i] and analyzed by the NEO Evaluation Specialists using SPSS version 26.0. for uni-variate analysis. A paired-sample test was conducted to compare the difference in the respondents’ expertise prior to and after taking the class.

 2. Key Findings[ii]

·         Primary learning outcomes: Nearly all of the respondents achieved the primary learning outcomes for the class. A minimum of 97 percent of the respondents reported that they 1) would commit to further developing their listening skills, in order to be a better ally to all (n=103, 100%), 2) are willing to do their part as not to burden others with the responsibility of educating them (n=102, 99%), and 3) understand the connection between actively trying and being responsible for making mistakes (n=101, 97%).

·         Meeting expectations of respondents: The class exceeded or met expectations for all of the respondents. About 46 percent of the respondents (n=48, 46%) reported that the class exceeded their expectations and the remaining 54 percent of the respondents reported that the class met all, most, or some of their expectations (n=56, 54%).

·         Comments about meeting expectations: A total of 53 respondents (51%) provided comments and the most common responses were about the skillful presenter (n=12, 23%) or the informative session (n=10, 19%).

·         Knowledge gain from the class: Fifty-five (55) out of sixty-eight (68) respondents (n=55, 81%) reported an increase in their expertise after the class. The average rating of the expertise was 34 prior to taking the class on a scale of 0 to 100, which increased to 52 after taking the class.

·         Experience with the class: Nearly all of the respondents somewhat or strongly agreed that the class was engaging (n=101, 98%), the technology used in the class was appropriate and supported their learning (n=102, 99%), and the instructor was knowledgeable and well-prepared (n=100, 99%).

·         Comments about the presenter: A total of 53 (51%) respondents provided comments that were grouped by theme and the most common themes were knowledge and preparation of instructor (n=18, 34%) and overall presentation (n=16, 30%). All the comments were positive except some of the comments in the ‘other’ category (n=3, 6%).

·         Most helpful part of the class: Fifty-seven (57) respondents (55%) provided comments. The respondents reported the most helpful part of the class to be examples (n=9, 16%), content on listening (n=6, 11%), and everything (n=6, 11%).

·         Areas for improvement: Thirty-five (35) respondents (34%) provided comments. The most common suggestions were for more participation and interaction (n=4, 11%) and additional resources or examples (n=4, 11%).

·         Likelihood of recommending the class to a colleague: About 88 percent of the respondents (n=89) stated that they would recommend it to a colleague and about 11 percent (n=11) stated that ‘maybe’ they would recommend it to a colleague. Only one (1) respondent (1%) reported that they would not recommend the class to a colleague.

·         General comments: A total of 22 respondents (21%) provided comments. The most common response was about the positive experiences with the presenter and class (n=8, 36%) and appreciation for the presenter (n=7, 32%).

·         Medical Library Association (MLA) Continuing Education (CE) credit: About 88 percent (n=91) of the respondents wanted to receive MLA CE credit. 

Survey Results

Q1. The first set of questions is about your experience with the content of the class. To what extent do you agree or disagree with the following statements?

The first set of questions assessed three main learning outcomes of the class. As seen in Figure 1, over 90 percent of the respondents strongly or somewhat agreed that:

·         Q. 1a. I will commit to further developing my listening skills, in order to be a better ally to all (100%, n=103).

·         Q. 1b. I am willing to do my part as not to burden others with the responsibility of educating me (99%, n= 102).

·         Q. 1c. I understand the connection between actively trying and being responsible for making mistakes (97%, n=101).

Only one percent (1%, n=1) somewhat agreed with Q.1b and about 2 percent (2%, n=2) somewhat disagreed with Q.1c. 


Q2. Did the class meet the respondents’ expectations (n=104)?

Overall, the class exceeded or met the respondents’ expectations (Figure 2). About half (46%, n=48) of the respondents reported that the class exceeded their expectations and another half (46%, n=48) stated that it met most or all of their expectations. About eight (8) percent of the respondents (n=8) noted that the class met some of their expectations. There was no respondent who stated that the class met almost none of the expectations or did not meet any of their expectations. 

Q2a. Please describe how the class did or did not meet the expectations (n=53).

A total of 53 respondents (51%) provided comments on how the class did or did not meet the expectations. Three respondents reported N/A and were removed from analysis.  The comments were related to:

·         The presenter was skillful (n=12, 23%).

·         The session was informative (n=10, 19%).

·         The class increased awareness (n=8, 15%).

·         The session generally met or exceeded expectations (n=4, 8%).

·          The examples and explanations provided in the class were helpful (n=4, 8%).

·         Good visuals were provided (n=2, 4%)

·         Other (n=16, 30%)

An example of ‘other’ comments includes:

          ”It would be great if we could see the speaker as well as the slides.  I think listening is at least a four-dimensional skill - ears, eyes, head, and  heart.  It helps for me to see the speaker.   I am hearing impaired.”

        “I gained confidence in taking responsibility and asking questions, and I learned how to approach situations where I am experiencing discomfort and lack of knowledge on my part.”

          “I agree that communication is key, but believe in male/female due to religious reasons.  I treat everyone with respect.”

          “I really liked the discussion on social justice warriors, and what our roles there are as allies. I also appreciated the statement that we own some of the responsibility of educating ourselves, not relying solely on others to tell us what we need to know.”

          “I just wasn't sure what I was going to hear, so I didn't know what my expectations even were.”

 Q3. Please rate your expertise in this subject PRIOR to taking this class (n=79).

The respondents were asked to rate their expertise prior to taking the class on a rating scale that ranged from novice (0), competent (50), to expert (100) on a continuum. The average score for expertise prior to taking this class was 34 and the most common response was 50 (n=11, 14%).

 Q4. Please rate your expertise in the class subject NOW (n=82).

The respondents were asked to rate their expertise after taking the class on a rating scale that ranged on the same rating scale as Q3. The average score for expertise after the class was 52. The most common score the respondents reported was 50 (n=7, 9%).

To assess the knowledge gain after the class, the individual ratings from the respondents before taking the class were subtracted from the scores after taking the class. Fourteen (14) respondents were excluded as they did not respond to Q3 or Q4, resulting in a final sample size of 68 available for comparison. The average individual score difference before and after taking the class was 15 ((t (67) = 6.8, p<.001). Of 68 respondents, about 81 percent (n=55) reported the knowledge gain and the biggest score increase reported was 57. About 15 percent of the respondents (n=10) reported a decrease in expertise after the class and about 4 percent (n=3) reported no difference before and after the class (Figure 3).

 

Q5. To what extent do you agree or disagree with the following statements?

The respondents were asked to rate their overall experience with the class (Figure 4). Nearly all of the respondents strongly or somewhat agreed that:

·         Q.5a. They found this class to be engaging (98%, n=101).

·         Q.5b. The technology used in the class was appropriate and supported their learning (99%, n=102). 

·         Q. 5c. The instructor was knowledgeable and well-prepared (99%, n=100)

Two (2) respondents somewhat disagreed with Q.5a and only one (1) respondent somewhat disagreed with Q.5b or Q.5c. There was no respondent who strongly disagreed with any of these statements. 

Q5c.i. Please comment on your assessment of the instructor in the previous question (n=53).

A total of 53 (51%) respondents provided comments that were grouped by theme: 1) knowledge and preparation of instructor (n=18, 34%), 2) overall presentation (n=16, 30%), 3) engagement (n=15, 28%), 4) clarity (n=6, 11%), 5) use of examples (n=5, 9%), 6) organization and preparation (n=5, 9%), 7) Adaptability/willingness to learn (n=4, 8%), and 8) other (n=3, 6%). All the comments were positive except some of the comments in the ‘other’ category. Nineteen (19) respondents had comments in more than one area. Two (2) N/A responses were removed from analysis.

Examples of comments by each theme include:

1)      Knowledge and preparation of the presenter (n=18)

         “Very well-versed about the topic and very enthusiastic.”

2)      Overall presentation (n=16)

          “Very thought provoking and informative!”

3)      Engagement (n=15)

          “Jessica is a really engaging and energetic speaker. She is great at providing examples and stories to illustrate her points.”

4)      Clarity (n=6)

         “Clear, thoughtful discussion, very responsive to comments.”

5)      Use of examples (n=5)

         “Instructor explains clearly and used good examples, and also accepted questions and ideas very comfortably.”

6)      Organization and preparation (n=5)

          “Jessica is a good presenter and the session was well organized.”

7)      Adaptability/willingness to learn (n=4)

        “Very conversational, able to adapt to questions on-the-fly while demonstrating that she is listening and learning herself.”

8)      Other (n=3)

       “She does know a lot, it is very evident. Sounds like a lot of theory though, not practical use, or perhaps she doesn't know how to tell laypeople.”

 

Within the “other” category, two participants expressed a desire for more interaction and more actionable points and examples.

 

Q6. What part of this class was MOST helpful (n=57)?

Fifty-seven (57) respondents (n=57, 55%) provided comments. The respondents reported the most helpful part of the class to be:

·         Examples (n=9, 16%)

·         Content on listening (n=6, 11%)

·         Everything (n=6, 11%)

·         Presenter (n=4, 7%)

·         General information provided (n=4, 7%)

·         List content (n=4, 7%)

·         Q & A (n=3, 5%)

·         Concept of taking responsibility/self-education (n=3, 5%)

·         Concepts related to being a better ally (n=3, 5%)

·         Communication with others (n=3, 5%)

·         The platinum rule (n=3, 5%)

·         Stories (n=3, 5%)

·         Visuals (n=2, 4%)

·         other (n=5, 9%)

Example of ‘other’ comments include:

         “Learning how to approach work w self-care and how to build sustainable advocacy practices that don't alienate people in your community. My notes say, "You can't always be at battle with other people because ultimately, it's about listening and building.”

          “Being reminded to interact with people with a spirit of humility and inquiry - we don't know more about others' situations than they know about themselves.”

  

Q7. How could this class be improved (n=35)?

A total of 35 respondents (n=35, 34%) provided comments after excluding four N/A responses. About 14 percent of respondents (n=5) reported not having any suggestions at this time. Eleven (11) percent of the respondents (n=4) expressed satisfaction or appreciation for the session.   Another 11 percent of respondents (n=4) expressed a desire for more participation/interaction. An additional 11 percent of participants (n=4) suggested additional resources or examples, followed by increased time (n=3, 9%), and appropriate time allocation for questions (n=2, 6%). Two respondents reported that they were not sure or didn’t know how the class could be improved (n=2, 6%).

About 20 percent (n=7) provided other comments and examples include:

          Interaction skills take a lot of experience, and a lot of self-evaluation, a single class doesn't make anyone an expert.  Perhaps address self-evaluation "how could I have handled that interaction better?.
          Perhaps a co-presenter of color?
      Please tell us how we can apply this in libraries.
         A few checks for understanding scattered throughout, and more Q&A time

Q8. Are you likely to recommend this class to a colleague (n=101)?

The respondents were asked to rate how likely they are to recommend this class to a colleague. About 88 percent of the respondents (n=89, 88%) stated that they would recommend it to a colleague and about 11 percent (n=11, 11%) stated that ‘maybe’ they would recommend it to a colleague. One (1) respondent (1%) reported that they would not recommend the class to a colleague.

 Q9. Please share any other comments you have about this class (n=22).

A total of 22 respondents (n=22, 21%) provided comments after excluding 1 response of ‘none’ and 1 N/A response. Three participants had comments that contained more than one theme (14%). About 36 percent of respondents reported positive experiences with the presenter/class (n=8), followed by appreciation for the presenter (n=7, 32%). Twenty-three percent (n=5) of the respondents stated that they found the class to be informative. Two participants stated they would recommend this series to others (n=2, 9%).  About 14% percent (n=3) provided other comments and examples include:

 I really appreciate the multiple format options: CC captions, slides shared, recordings shared later, documents linked from the NNLM site, etc.
Do you think that if someone is willing to question what he/she doesn't know already shows a sense of self-awareness?
This class was fine.

Q10. Do you want to receive Medical Library Association Continuing Education credit for this class (n=103)?

About 88 percent (n=91) of the respondents wanted to receive Medical Library Association Continuing Education credit.

 

*For a complete list of comments from the second DEI class, please refer to the supplemental document, Appendix 1: Comments from the Third DEI Webinar. *For a copy of the survey, please refer to Appendix 2: Third DEI Webinar Survey Questionnaire.


[i] Survey data were collected and managed using REDCap electronic data capture tools hosted at the Institute of Translational Health Sciences (ITHS) with grant support (UL1 RR025014 from NCRR/NIH). REDCap (Research Electronic Data Capture) is a secure, web-based application designed to support data capture for research studies, providing: 1) an intuitive interface for validated data entry; 2) audit trails for tracking data manipulation and export procedures; 3) automated export procedures for seamless data downloads to common statistical packages; and 4) procedures for importing data from external sources.

 

[ii] Sample size varies by question. The percentage of the responses for each question is valid percent only excluding missing values and was based on the total number of respondents who answered each question.