The template for this protocol is available at Usability.gov. It was modified for the purpose of this project. Direct quotes have been identified and the source has been cited in the Reference Section.

TEST PROTOCOL FOR RAMPANT STRATEGY

Team: O.M.N.I. KREW

Members: Kevin Nyquist, Eliana Gelman, Will Howell, Christian Nam, Rita Mathew

Milestone 4

Rampant Strategy is a platform being used by students in large undergraduate classrooms at the University of Georgia. It was deployed in classrooms to allow teachers and teaching assistants - to communicate with students effectively regarding assignments; to give instructions to students regarding coursework; to make feedback and grades accessible in a timely manner; and, allow students to access the platform functionalities to easily submit homework online.

The user experience of Rampant Strategy, however, was far from ideal. Students are hindered from completing coursework because of non-optimal design. Cluttered and confusing mapping causes students to have issues navigating pages, saving work, and accessing feedback from the instructor. Grades are affected because of difficulty submitting polls and assignments. There are no signifiers on the buttons to indicate interactability. This lack of signifiers causes students to miss vital information about assignments. Poor design has caused issues of accessibility and effectiveness. Rather than promoting active learning for students, and providing timely feedback to students, Rampant Strategy fell short of meeting the needs of students, teaching assistants and instructors.

Therefore the aim of this protocol is to conduct a usability test during the re-design and development of Rampant Strategy to improve user experience. According to Usability.gov, “the goals of usability testing include establishing a baseline of user performance, establishing and validating user performance measures, and identifying design concerns to be addressed in order to improve user experience as measured by accessibility and effectiveness.” Using a systems approach to design, we define accessibility as the process which ensures that users have information, activities and/or environments available to complete tasks comfortably and without major complications; or the practice of making these aspects of a system sensible, meaningful, and usable for as many people as possible. Effective communication is the interchange between two or more persons for the purpose of delivering, receiving, and understanding a message successfully in a timely manner.


A. Objective: The usability test objectives for this project are as follows: 

  1. To determine design inconsistencies and usability problem areas within the user interface specifically for accessibility and effectiveness.

  2. Establish a CONTROL GROUP for baseline user performance and user-satisfaction levels of the user interface with a control group.

  3. Test the prototype with EXPERIMENTAL GROUP (undergraduate students) at UGA.

B. Research Question: 

  1. How accessible are affordances and signifiers to users?

  2. How effective are the functionalities of Rampant Strategy?

  3. Do the affordances and signifiers allow for effective communication between students and teachers?

  4. What are the user assumptions made by designers?

C. Hypothesis: 

  1. By recognizing and responding to bias, the prototype makes Rampant Strategy effective and accessible which results in improvement of user experience. 

  2. The modifications in design incorporating affordances and signifiers to address user needs for accessibility and effective use of the platform results in improvement of user experience.

Null Hypothesis:

Modifications in design incorporating affordances and signifiers to address user needs for accessibility and effective use of the platform do not result in improvement of user experience.

D. METHODOLOGY

Good designers “start by trying to understand what the real issues are.” (Norman, 2013. p. 218). Therefore, rather than using traditional measures of people, such as age, education, and income, the study will be based on an activity centered approach of first gaining a deep understanding of students’ needs around activities to be performed. In order to obtain IRB approval the following documents will be submitted - Letter of informed consent, Letter to students seeking participation, Instruments for data collection - survey, interview.

  1. Letter of Informed Consent – This is an important step in any research protocol. The Belmont Report was written by the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. Please see attached. 

  1. Letter to Students seeking participation - This letter will be sent to all the students in the Management Information Systems class seeking their participation in the study.

  1. Instruments for data collection – Please see attached for specific data that will be collected. There are advantages and disadvantages with every instrument. Two instruments were chosen in order to confirm the results for validity and reliability. 

A survey is useful to gather data on demographics of users, how widespread are the needs, and other instructional and environmental factors. This study is activity centered and thus no demographic information will be collected. In addition, students will be informed that it is Rampant Strategy that is being tested. 

Interviews are important sources of understanding contextual factors such as perception of task relevance, opportunities to perform; and cultural factors such as whether hierarchy has any influence on performance. They provide an opportunity to ask open-ended questions.

Participants


Testing Table


E. ANALYSIS

Data will be used to assess whether usability goals regarding accessibility and effectiveness have been achieved. Difference in responses to the survey questions will be analyzed with statistical software such as ANOVA, linear regression or t-test. If the change is statistically significant then it will corroborate our hypothesis that the modifications in design to incorporate affordances and signifiers address user needs for accessibility and effective use of the platform results in improvement of user experience.

F. BENEFITS

To improve this educational technology tool, we propose a more intuitive interface that would benefit all parties involved. The proposed changes to Rampant Strategy, such as replacing the button on the side nav-bar with a drop-down on the top nav-bar and splitting the current submit button into Save and Submit buttons, would help to address the confusion and improve the usability of the platform. These changes could significantly enhance the learning experience for students and improve the efficiency of instructors and teaching assistants in conducting classes and grading assignments.

References

Norman, D. (2013). The Design of Everyday Things. Basic Books. p. 218.

Usability.gov (n.d.) https://www.usability.gov/


UNIVERSITY OF GEORGIA

INFORMED CONSENT

Testing the prototype of Rampant Strategy New Improved version

Dear Participant,

My name is Michael Cotterell and I am a faculty member in the Department of Computer Science at the University of Georgia. I am inviting you to take part in a research study.

You are being asked to participate in a survey to provide feedback on the prototype of Rampant Strategy.  Rampant Strategy is used in large classrooms for students and faculty to communicate assignments, deadlines, quizzes and any other issue related to the course. You are being asked to participate in this study because you are currently a student at the University of Georgia. Your participation in this research is voluntary. If you choose to participate in this study, you will be asked to complete a brief online survey that should take less than 10 minutes to complete. Your participation in this research is voluntary. If you choose to participate in this study, you will be asked to participate in a brief 15 minute interview via Zoom.

Researchers 

Principal Investigator:

Michael Cotterell, PhD, Department of Computer Science, University of Georgia

Co-Investigators:

Rita Mathew, Doctoral Student, Workforce Education, Department of Workforce Education and Instructional Technology, College of Education. Kevin Nyquist, Eliana Gelman, William Howell, Christian Nam.

Risks and Benefits

Risks: There are no known risks at this time to participate in this study. This research involves the transmission of data over the Internet. Every reasonable effort has been taken to ensure the effective use of available technology; however, confidentiality during online communication cannot be guaranteed.

Benefits: The opportunity to contribute to the knowledge of the field through your participation in the study.

 


Costs and Payments

There will be no costs to you for participation in this research study. The researcher is unable to give you any payment for participating in this study.

New Information

If the researcher finds new information during this study that would reasonably change your decision about participating, you will be informed.

Confidentiality

All information obtained about you or your institution for this study is strictly confidential unless disclosure is required by law. The results of this study may be used in reports, presentations, and publications, but the researcher will not identify you or your institution.

Withdrawal Privilege

It is OK for you to say NO. Even if you say YES now, you are free to say NO later, and walk away or withdraw from the study—at any time. Your decision will not affect your relationship with the University of Georgia, or otherwise cause a loss of benefits to which you might otherwise be entitled. Your decision to participate or not participate in the research will have no effect on your employment with the University of Georgia.

Questions

If you are interested in participating or have questions about this research, please feel free to contact our team member Rita Mathew at 706-202-0935, ritmath@uga.edu.  If you have any complaints or questions about your rights as a research volunteer, contact the IRB at 706-542-3199 or by email at IRB@uga.edu.

Please keep this letter for your records.

Voluntary Consent 

I have read, understood, and received a copy of the above consent and desire of my own free will to participate in this study. 

I have been informed in advance what my task(s) will be and what procedures will be followed. 

I have been given the opportunity to ask questions and have had my questions answered to my satisfaction. 

I am aware that I have the right to withdraw consent and to discontinue participation at any time, without prejudice to my future inclusion in this project. 

I understand that participation in this usability study is voluntary and I agree to immediately raise any concerns or areas of discomfort during the session with the study administrator. 

My signature below may be taken as affirmation of all the above statements; it was given prior to my participation in this study. 

Please sign below to indicate that you have read and you understand the information on this form and that any questions you might have about the session have been answered. 

 Name (Printed): _____________  Signature:______________   Date: ____________ 

 

References

Shneiderman, B. (1998). Designing the user interface: Strategies for effective Human-Computer Interaction. 3rd Ed. Addison Wesley. (p.130).

Usability.gov (n.d.) https://www.usability.gov/how-to-and-tools/resources/templates/consent-form-adult.html

UNIVERSITY OF GEORGIA

RECRUITMENT LETTER- Student

A study to test a prototype of Rampant Strategy

University of Georgia student recruiting email

 

Dear *******,

 

This email is to invite you to participate in a survey being conducted by Dr. Michael Cotterell, faculty member, Department of Computer Science, Rita Mathew from the Department of Workforce Education and Instructional Technology at the University of Georgia, Kevin Nyquist, Eliana Gelman, William Howell, and Christian Nam to test a prototype of Rampant Strategy, a software used in large classrooms for students and faculty to communicate assignments, deadlines, quizzes and any other issue related to the course.

As a UGA student who uses the software, your perspective will be a valuable source of information for the researchers as they complete their usability test.

If you choose to participate in this study, you will be asked to participate in an interview that will last approximately 15 minutes with our research team to discuss any thoughts you would like to share regarding the software.

If you have any questions or are willing to participate in an interview, please contact our team member, Rita Mathew at ritmath@uga.edu.  

 

 


 


Form 1 - SURVEY

The link for the survey is  https://www.surveymonkey.com/r/8T5HM3W


Rampant Strategy New Improved

Rampant Strategy is used in large classrooms for students and faculty to communicate assignments, deadlines, quizzes and any other issue related to the course.


1. Overall, how would you rate Rampant Strategy NI?

2. Would you recommend Rampant Strategy to other students?

Please rate the individual features below:

3. Tab for Grades

4. Tab for Assignment feedback from instructor

5. Ease of assignment submission

6. Tab for Deadlines

7. Option to save work and return to complete later.

8. Is this course required for your major?


Form 2 – INTERVIEW PROTOCOL TOOL  for those who respond to survey to test Rampant Strategy.

The interview will use a planned script that introduces the purpose of the interview – testing a prototype of Rampant Strategy - and asks for any permission necessary related to the interview:

·         Inform that IRB approval has been obtained.

·         Obtain permission to record the interview and explain that it will be used to help you complete the report.

·         Explain that they may review the report once it is completed.

The interview will conclude with thanks to the participant for their time, and remind them about how the information will be used for research in testing a prototype of Rampant Strategy.

Points to remember during interview:

1.      Active listening.

2.      Familiarity with the interview questions.

3.      Expected length of the interview will be 15 minutes.

 [Introduction]

Can you say your name and position?

Questions for students: 

1.      What was your experience with the Rampant Strategy new version?

2.      Were the tabs easy to use?

3.      What types of challenges or concerns did you have while using Rampant Strategy?

4.      Did the software do what you anticipated it would do?

5.      Were there any issues of accessibility?




Created using Github Pages. For exclusive use of O.M.N.I. Krew and it's affiliates.