Difference between revisions of "UsabilityCluster/Usability Test Framework"

From Earth Science Information Partners (ESIP)
(Created page with "<div style="">Table of Contents</div> [https://docs.google.com/document/d/1-4eZDvknOq2r9p7BcDNUFX_wUaFJMsj3s6DvzsATgxM/edit#heading=h.ytmu6ujr63qb Inviting Users to Participa...")
 
 
(45 intermediate revisions by 2 users not shown)
Line 1: Line 1:
<div style="">Table of Contents</div>
 
  
[https://docs.google.com/document/d/1-4eZDvknOq2r9p7BcDNUFX_wUaFJMsj3s6DvzsATgxM/edit#heading=h.ytmu6ujr63qb Inviting Users to Participate in the Study]
+
=== Section 1 of 6: Inviting Users to Participate in the Study ===
 
 
[https://docs.google.com/document/d/1-4eZDvknOq2r9p7BcDNUFX_wUaFJMsj3s6DvzsATgxM/edit#heading=h.kmpivxckc8jg Test Introduction]
 
 
 
[https://docs.google.com/document/d/1-4eZDvknOq2r9p7BcDNUFX_wUaFJMsj3s6DvzsATgxM/edit#heading=h.im1bk84tgmj1 Pre-test survey (Use example questions from DataONE survey and other member’s surveys.)]
 
 
 
[https://docs.google.com/document/d/1-4eZDvknOq2r9p7BcDNUFX_wUaFJMsj3s6DvzsATgxM/edit#heading=h.km0yvuv6hx1p Usability test itself]
 
 
 
[https://docs.google.com/document/d/1-4eZDvknOq2r9p7BcDNUFX_wUaFJMsj3s6DvzsATgxM/edit#heading=h.kr5c24lal3cp Post-test reflection]
 
 
 
[https://docs.google.com/document/d/1-4eZDvknOq2r9p7BcDNUFX_wUaFJMsj3s6DvzsATgxM/edit#heading=h.q7zjeuw5j4f4 Using the Test Results]<span style=""></span>
 
 
 
 
 
=== Inviting Users to Participate in the Study ===
 
  
 
* <div style="">Purpose of this Section:</div>
 
* <div style="">Purpose of this Section:</div>
Line 22: Line 8:
  
  
 +
<div style="">
 +
 +
:::Dear [person’s name],
 +
:::
 +
 +
:::[First sentence should be something that allows the person to recognize the organization, or a particular person that is familiar to them to catch their interest and let them know that this isn’t spam.] We are conducting a study of [product] to better understand [how to organize information, how users interact with the system, etc. Why are you conducting the usability study - very broad statement]. &nbsp;[Provide and explanation of the product, if necessary (1-2 sentences that tells the person what the product is)]. Your feedback will help us to improve this product.
 +
:::
 +
 +
:::During the study you will be asked to complete a number of tasks using [the product] and provide feedback about your experience. The study will take approximately [time frame (see section further down about appropriate test times)] and will run from [date range]. The study will take place [virtually OR (insert location)]. To thank you for your time, we have [insert the incentive (e.g. $5 gift card) (This is optional)].
 +
:::
  
<div style="">Dear [person’s name],[First sentence should be something that allows the person to recognize the organization, or a particular person that is familiar to them to catch their interest and let them know that this isn’t spam.] We are conducting a study of [product] to better understand [how to organize information, how users interact with the system, etc. Why are you conducting the usability study - very broad statement]. &nbsp;[Provide and explanation of the product, if necessary (1-2 sentences that tells the person what the product is)]. Your feedback will help us to improve this product.During the study you will be asked to complete a number of tasks using [the product] and provide feedback about your experience. The study will take approximately [time frame (see section further down about appropriate test times)] and will run from [date range]. The study will take place [virtually OR (insert location)]. To thank you for your time, we have [insert the incentive (e.g. $5 gift card) (This is optional)].If you are interested in participating, please [indicate how they can reply] and we/I will provide you with more information.</div>
+
:::If you are interested in participating, please [indicate how they can reply] and we/I will provide you with more information.
  
 +
</div>
  
 
* <div style="">Tips:</div>
 
* <div style="">Tips:</div>
Line 30: Line 27:
 
** <div style="">Personalize the email and provide references that the users could relate to, such as an organization or a contact information. Name recognition could help to draw users in.</div>
 
** <div style="">Personalize the email and provide references that the users could relate to, such as an organization or a contact information. Name recognition could help to draw users in.</div>
  
 +
<br />
 +
<hr />
  
 
+
=== Section 2 of 6: Test Introduction ===
 
 
 
 
=== Test Introduction ===
 
  
 
* <div style="">Purpose of this Section:</div>
 
* <div style="">Purpose of this Section:</div>
Line 44: Line 40:
 
**** <div style="">Step 1: Verify if formal informed consent is necessary for your organization (e.g Internal Review Board - IRB)</div>
 
**** <div style="">Step 1: Verify if formal informed consent is necessary for your organization (e.g Internal Review Board - IRB)</div>
 
**** <div style="">Step 2: Document any applicable waiver for consent.</div>
 
**** <div style="">Step 2: Document any applicable waiver for consent.</div>
**** <div style="">.</span></div>
+
**** <div style=""><span style="">Step 3: If a consent is required, confirm with the IRB if a consent script would be sufficient. If yes, also ensure whether a waiver for a signed informed consent form is required. An example of consent script and can be viewed </span>[https://docs.google.com/document/d/1mZ9TEmR70OVTH2kD1MCPEqJyRg0zs9tF_pNuZUW3cu8/edit?usp=sharing here]<span style="color:#000000;">.</span></div>
**** <div style="">. </span></div>
+
**** <div style=""><span style="">Step 4: If a consent script is not sufficient and a formal consent form is required, then check with your IRB if there is an official form that should be used. In general, an example of a consent form can be viewed </span>[https://docs.google.com/document/d/1riibZc9OASCX8_B6LCyQ1ibxHPuiRkHi0Z6Yn8jKDkM/edit?usp=sharing here]<span style="color:#000000;">. </span></div>
**** <div style="">.</span></div>
+
**** <div style=""><span style="">Additional references/recommendations per the </span>[https://www.hhs.gov/ohrp/regulations-and-policy/regulations/45-cfr-46/index.html U.S. Department of Health & Human Services]<span style="color:#000000;"> has been summarized and provided by Dr. Robert Downs of the Center for International Earth Science Information Network (CIESIN) at Columbia University, which can be viewed </span>[https://docs.google.com/document/d/1CTw32GnOTBB9afXRAtSAgWJUDiHDGFIi5uqUbuzYcnI/edit?usp=sharing here]<span style="color:#000000;">.</span></div>
 
** <div style="">Additional areas to discuss with users:</div>
 
** <div style="">Additional areas to discuss with users:</div>
 
*** <div style="">Want users to be as natural as possible. </div>
 
*** <div style="">Want users to be as natural as possible. </div>
Line 57: Line 53:
 
*** <div style="">If they are on the web, bookmark the homepage, so if they do get lost they can get back to where they started. Let them know how to get back to the homepage.</div>
 
*** <div style="">If they are on the web, bookmark the homepage, so if they do get lost they can get back to where they started. Let them know how to get back to the homepage.</div>
  
 +
* <div style="">Tips:</div>
 +
** <div style="">Don’t read the test user a script here, just have a conversation. In other words, it is important to rehearse and pilot-test the introduction because it is helpful to be as natural as possible when interacting with your test users. If you read them a script, they will feel more like they are being studied. Keep things as natural and comfortable as possible.</div>
  
 +
<br />
 +
<hr />
  
*
+
=== Section 3 of 6: Pre-test survey ===
** <div style="">Tips:</div>
 
*** <div style="">Don’t read them a script here, just have a conversation. In other words, it is important to rehearse and pilot-test the introduction because it is helpful to be as natural as possible when interacting with your test users. If you read them a script, they will feel more like they are being studied. Keep things as natural and comfortable as possible.</div>
 
 
 
 
 
 
 
=== Pre-test survey (Use example questions from DataONE survey and other member’s surveys.) ===
 
  
 
* <div style="">Purpose of this Section:</div>
 
* <div style="">Purpose of this Section:</div>
Line 73: Line 67:
 
*** <div style="">Understand the differences in test results. </div>
 
*** <div style="">Understand the differences in test results. </div>
 
*** <div style="">Provide user’s context.</div>
 
*** <div style="">Provide user’s context.</div>
 
  
  
Line 80: Line 73:
 
*** <div style="">We recommend collecting information about the user’s demographic information and experience with the system-under-test.</div>
 
*** <div style="">We recommend collecting information about the user’s demographic information and experience with the system-under-test.</div>
 
*** <div style="">Sample questions that could be used are shown below:</div>
 
*** <div style="">Sample questions that could be used are shown below:</div>
**** <div style="">What is your field of study? (fill in the blank) (Possibly include link to RDA vocabulary for the suggested values)</div>
+
<br />
  
 +
<div>============================================================</div>
 +
:*<div style="">What is your field of study? (fill in the blank)</div>
  
 +
:<div style="">Examples for presenting the above questions with predefined values:</div>
 +
::<div><span style="">☐ </span><span style="color:#000000;">research</span></div>
 +
::<div><span style="color:#000000;">☐ </span><span style="color:#000000;">program management/project management</span></div>
 +
::<div><span style="color:#000000;">☐ </span><span style="color:#000000;">data management</span></div>
 +
::<div><span style="color:#000000;">☐ </span><span style="color:#000000;">metadata creating</span></div>
 +
::<div><span style="color:#000000;">☐ </span><span style="color:#000000;">database administration</span></div>
 +
::<div><span style="color:#000000;">☐ </span><span style="color:#000000;">ecological modeling</span></div>
 +
::<div><span style="color:#000000;">☐ </span><span style="color:#000000;">policy making</span></div>
 +
::<div><span style="color:#000000;">☐ </span><span style="color:#000000;">field work</span></div>
 +
::<div><span style="color:#000000;">☐ </span><span style="color:#000000;">application development</span></div>
 +
::<div>☐ Other, please specify here: </div>
  
<div style=""></div>
+
:* <div style="">How often do you [use a system that is presented by the system-under-test]?</div>
 +
:** <div style="">1 to 5 (Likert scale) 1 being least 5 being most.</div>
 +
:* <div style="">How would you rate your experience level with [the system that is presented by the system-under-test]?</div>
 +
:** <div style="">1 to 5 (Likert scale) 1 being least 5 being most.</div>
 +
:* <div style="">How often do you [perform the function/task that is being tested]?</div>
 +
:** <div style="">1 to 5 (Likert scale) 1 being least 5 being most.</div>
 +
:* <div style="">How would you rate your experience level with [the function/task]?</div>
 +
:** <div style="">1 to 5 (Likert scale) 1 being least 5 being most.</div>
 +
:* <div style="">Optional: </div>
 +
:** <div style="">Additional questions for collecting user’s reasons/interests in and expectation with the system-under-test could be considered. </div>
 +
:** <div style="">Sample questions that could be used are shown below:</div>
 +
:*** <div style="">What are your key job duties?</div>
 +
:*** <div style="">Why are you interested in participating in this test?</div>
 +
:*** <div style="">Is there anything you would like to learn from the experience?</div>
 +
:*** <div style="">Have you used other systems that are similar to the system-under-test?</div>
 +
<div>============================================================</div>
  
<div style="">Examples for presenting the above questions with predefined values:</div>
 
  
*<div style="">application development</div>
+
* <div style="">Tips: </div>
 +
** <div style="">Please note that free text questions can be time consuming for users to answer, so you should make sure that the questions add value to the survey.</div>
 +
** <div style="">Additionally, it is more difficult and time consuming to analyze free text questions, so free text questions should be considered carefully before including in the pre-test survey. </div>
  
*<div style="">Other, please specify here: </div>
+
<br />
 +
* <div style="">Additional References:</div>
 +
** <div style="">[http://www.d.umn.edu/~lcarlson/testing/planning/pre_questionnaire.html http://www.d.umn.edu/~lcarlson/testing/planning/pre_questionnaire.html]</div>
 +
** <div style="">[http://www.ablongman.com/html/barnum/pdf/pretestquestionnaire.PDF http://www.ablongman.com/html/barnum/pdf/pretestquestionnaire.PDF]<span style="color:#000000;"> </span></div>
 +
** <div style="">[https://www.webpages.uidaho.edu/ART370S15-ande8188/Syllabus%20and%20Profile/UsabilityTests.pdf https://www.webpages.uidaho.edu/ART370S15-ande8188/Syllabus%20and%20Profile/UsabilityTests.pdf]<span style="color:#000000;"> </span></div>
  
 +
<br />
 +
<hr />
  
**** <div style="">How often do you [use a system that is presented by the system-under-test]?</div>
+
=== Section 4 of 6: Usability test itself ===
***** <div style="">1 to 5 (Likert scale) 1 being least 5 being most.</div>
 
**** <div style="">How would you rate your experience level with [the system that is presented by the system-under-test]?</div>
 
***** <div style="">1 to 5 (Likert scale) 1 being least 5 being most.</div>
 
**** <div style="">How often do you [perform the function/task that is being tested]?</div>
 
***** <div style="">1 to 5 (Likert scale) 1 being least 5 being most.</div>
 
**** <div style="">How would you rate your experience level with [the function/task]?</div>
 
***** <div style="">1 to 5 (Likert scale) 1 being least 5 being most.</div>
 
** <div style="">Optional: </div>
 
*** <div style="">Additional questions for collecting user’s reasons/interests in and expectation with the system-under-test could be considered. </div>
 
*** <div style="">Sample questions that could be used are shown below:</div>
 
**** <div style="">What are your key job duties?</div>
 
**** <div style="">Why are you interested in participating in this test?</div>
 
**** <div style="">Is there anything you would like to learn from the experience?</div>
 
**** <div style="">Have you used other systems that are similar to the system-under-test?</div>
 
  
 +
* <div style="">Overview</div>
 +
** <div style="">[https://www.usability.gov/how-to-and-tools/methods/planning-usability-testing.html https://www.usability.gov/how-to-and-tools/methods/planning-usability-testing.html]<span style="color:#000000;"> </span></div>
  
 +
<br />
 +
* <div style="">Length of the Test</div>
 +
** <div style="">[https://www.nngroup.com/articles/time-budgets-for-usability-sessions/ https://www.nngroup.com/articles/time-budgets-for-usability-sessions/]<span style="color:#000000;"> (typically 60 - 90 minutes)</span></div>
 +
** <div style="">[https://ux.stackexchange.com/questions/58542/how-long-does-a-usability-study-take-from-start-to-finish https://ux.stackexchange.com/questions/58542/how-long-does-a-usability-study-take-from-start-to-finish]<span style="color:#000000;"> (guerrilla mode is about 30 to 60 minutes)</span></div>
 +
** <div style="">Typical user vs “Super” user</div>
 +
*** <div style="">Typical user: Someone who is not motivated to be part of the process. A sample from the general user pool. These people are volunteering time for this project. (30 - 60 minutes).</div>
 +
*** <div style="">Super user: Someone who is not biased, but knows about the value of the project and is motivated (incentivized or tasked) to provide time and resources to participate. (up to 90 minutes).</div>
 +
<br />
  
** <div style="">Tips: </div>
 
*** <div style="">Please note that free text questions can be time consuming for users to answer, so you should make sure that the questions add value to the survey.</div>
 
*** <div style="">Additionally, it is more difficult and time consuming to analyze free text questions, so free text questions should be considered carefully before including in the pre-test survey. </div>
 
 
 
 
** <div style="">Additional References:</div>
 
*** <div style="">[http://www.d.umn.edu/~lcarlson/testing/planning/pre_questionnaire.html http://www.d.umn.edu/~lcarlson/testing/planning/pre_questionnaire.html]</div>
 
*** <div style=""> </span></div>
 
*** <div style=""> </span></div>
 
 
 
 
<div style=""><nowiki>*Note: We will need to add references here. Do people already have recommendations?</nowiki></div>
 
 
=== Usability test itself ===
 
 
* <div style="">Overview</div>
 
** <div style="">https://www.usability.gov/how-to-and-tools/methods/planning-usability-testing.html</div>
 
* <div style="">Length of the test</div>
 
** <div style=""> https://www.nngroup.com/articles/time-budgets-for-usability-sessions/ (typically 60 - 90 minutes)</div>
 
** <div style=""> https://ux.stackexchange.com/questions/58542/how-long-does-a-usability-study-take-from-start-to-finish (guerrilla mode is about 30 to 60 minutes)</div>
 
** <div style="">Typical user vs “Super” user</div>
 
*** <div style="">Typical user: Someone who is not motivated to be part of the process. A sample from the general user pool. These people are volunteering time for this project. (30 - 60 minutes)</div>
 
*** <div style="">Super user: Someone who is not biased, but knows about the value of the project and is motivated (incentivized or tasked) to provide time and resources to participate. (up to 90 minutes)</div>
 
 
* <div style="">Tasks (appropriate to the length of the test)</div>
 
* <div style="">Tasks (appropriate to the length of the test)</div>
 
** <div style="">Using other methods (cognitive walkthrough) to determine the baseline for the length of a task. If the cognitive walkthrough is best case scenario (completing tasks perfectly), you know that average users will require substantially more time to complete the task and provide feedback. </div>
 
** <div style="">Using other methods (cognitive walkthrough) to determine the baseline for the length of a task. If the cognitive walkthrough is best case scenario (completing tasks perfectly), you know that average users will require substantially more time to complete the task and provide feedback. </div>
Line 141: Line 140:
 
** <div style="">Based on tests that have been performed by usability labs, it is typically better to design the tasks using an overall time frame (aim for 15 to 30 minutes), then to restrict to a specific set or number of tasks. &nbsp;</div>
 
** <div style="">Based on tests that have been performed by usability labs, it is typically better to design the tasks using an overall time frame (aim for 15 to 30 minutes), then to restrict to a specific set or number of tasks. &nbsp;</div>
 
** <div style="">[https://www.quora.com/In-usability-testing-how-many-task-scenarios-should-you-ask-users-to-complete-and-what-is-the-rationale-for-this-number https://www.quora.com/In-usability-testing-how-many-task-scenarios-should-you-ask-users-to-complete-and-what-is-the-rationale-for-this-number]</div>
 
** <div style="">[https://www.quora.com/In-usability-testing-how-many-task-scenarios-should-you-ask-users-to-complete-and-what-is-the-rationale-for-this-number https://www.quora.com/In-usability-testing-how-many-task-scenarios-should-you-ask-users-to-complete-and-what-is-the-rationale-for-this-number]</div>
* <div style="">Definitions of tasks:</div>
+
<br />
 +
 
 +
* <div style="">Definitions of Tasks:</div>
 
** <div style="">Guidelines/recommendations regarding how to develop the tasks.</div>
 
** <div style="">Guidelines/recommendations regarding how to develop the tasks.</div>
 
*** <div style="">References:</div>
 
*** <div style="">References:</div>
**** <div style=""> </span></div>
+
**** <div style="">[https://www.nngroup.com/articles/better-usability-tasks/ https://www.nngroup.com/articles/better-usability-tasks/]<span style="color:#000000;"> </span></div>
**** <div style=""> </span></div>
+
**** <div style="">[https://measuringu.com/task-tips/ https://measuringu.com/task-tips/]<span style="color:#000000;"> </span></div>
**** <div style=""> </span></div>
+
**** <div style="">[https://userbrain.net/blog/write-better-tasks-to-improve-usability-testing https://userbrain.net/blog/write-better-tasks-to-improve-usability-testing]<span style="color:#000000;"> </span></div>
 
** <div style="">Guidelines/recommendations regarding how the test administrators could interact with the user.</div>
 
** <div style="">Guidelines/recommendations regarding how the test administrators could interact with the user.</div>
 
*** <div style="">E.g. staying unbiased, help asking questions to help users in thinking out loud, try not to distract users.</div>
 
*** <div style="">E.g. staying unbiased, help asking questions to help users in thinking out loud, try not to distract users.</div>
*** <div style=""> </span></div>
+
*** <div style="">[https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html]<span style="color:#000000;"> </span></div>
*** <div style=""> (see the “Take on the role of moderator” section)</span></div>
+
*** <div style="">[https://uxmastery.com/beginners-guide-to-usability-testing/ https://uxmastery.com/beginners-guide-to-usability-testing/]<span style="color:#000000;"> (see the “Take on the role of moderator” section)</span></div>
 
** <div style="">Would be determined based on the system-under-test and the areas needing evaluations.</div>
 
** <div style="">Would be determined based on the system-under-test and the areas needing evaluations.</div>
 +
<br />
 +
 
* <div style="">Test Implementation:</div>
 
* <div style="">Test Implementation:</div>
 
** <div style="">Will it be in person or remote?</div>
 
** <div style="">Will it be in person or remote?</div>
Line 157: Line 160:
 
** <div style="">Will you record the usability test session?</div>
 
** <div style="">Will you record the usability test session?</div>
 
** <div style="">How will you use the information that you get from the test?</div>
 
** <div style="">How will you use the information that you get from the test?</div>
** <div style="">, (split into free vs. paid), (online vs. software application) </span></div>
+
** <div style=""><span style="color:#000000;">Tools: </span>[https://www.techsmith.com/morae.html Morae]<span style="color:#000000;">, </span>[https://www.loop11.com/ Loop11]<span style="color:#000000;">, </span>[https://marvelapp.com/user-testing/ Marvel]<span style="color:#000000;">, (split into free vs. paid), (online vs. software application) </span></div>
 
 
  
 +
<br />
 +
<hr />
  
=== Post-test reflection ===
+
=== Section 5 of 6: Post-test reflection ===
  
 
* <div style="">Choosing when to ask questions after each task versus at the end of the test</div>
 
* <div style="">Choosing when to ask questions after each task versus at the end of the test</div>
Line 167: Line 171:
 
** <div style="">How much time do you have to evaluate the results? This will affect how many questions you want to ask.</div>
 
** <div style="">How much time do you have to evaluate the results? This will affect how many questions you want to ask.</div>
 
** <div style="">Are you more interested in the individual tasks or the test as a whole?</div>
 
** <div style="">Are you more interested in the individual tasks or the test as a whole?</div>
 +
 +
<br/>
 
* <div style="">Guidelines/recommendations regarding how to “interview” the users.</div>
 
* <div style="">Guidelines/recommendations regarding how to “interview” the users.</div>
 
** <div style="">Quantitative Methods:</div>
 
** <div style="">Quantitative Methods:</div>
*** <div style=""> questions could be applied here (To get quantifiable results, you need to ask all 10 questions; however, these could be used for more generic feedback as well if you can’t get through all of them) (scale is 1 - strongly disagree to 5 - strongly agree)</span></div>
+
*** <div style="">[https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html System Usability Scale]<span style="color:#000000;"> questions could be applied here (To get quantifiable results, you need to ask all 10 questions; however, these could be used for more generic feedback as well if you can’t get through all of them) (scale is 1 - strongly disagree to 5 - strongly agree)</span></div>
 
**** <div style="">1. I think that I would like to use this system frequently. </div>
 
**** <div style="">1. I think that I would like to use this system frequently. </div>
 
**** <div style="">2. I found the system unnecessarily complex. </div>
 
**** <div style="">2. I found the system unnecessarily complex. </div>
Line 183: Line 189:
 
**** <div style="">Overall, how difficult or easy did you find this task?</div>
 
**** <div style="">Overall, how difficult or easy did you find this task?</div>
 
***** <div style="">It uses a scale of 1 to 7 (1 - very difficult; 7 - very easy)</div>
 
***** <div style="">It uses a scale of 1 to 7 (1 - very difficult; 7 - very easy)</div>
*** <div style="">Reference from Nielsen Norman Group: </span>[https://www.nngroup.com/articles/measuring-perceived-usability/ Beyond the NPS: Measuring Perceived Usability with the SUS, NASA-TLX, and the Single Ease Question After Tasks and Usability Tests]</div>
+
*** <div style=""><span style="">Reference from Nielsen Norman Group: </span>[https://www.nngroup.com/articles/measuring-perceived-usability/ Beyond the NPS: Measuring Perceived Usability with the SUS, NASA-TLX, and the Single Ease Question After Tasks and Usability Tests]</div>
 
** <div style="">Qualitative:</div>
 
** <div style="">Qualitative:</div>
 
*** <div style="">Interview Questions (could be better to ask these questions in person than with a written survey):</div>
 
*** <div style="">Interview Questions (could be better to ask these questions in person than with a written survey):</div>
Line 191: Line 197:
 
**** <div style="">The questions from System Usability Scale could also be used as prompts for getting qualitative feedback. </div>
 
**** <div style="">The questions from System Usability Scale could also be used as prompts for getting qualitative feedback. </div>
 
* <div style="">Information to invite users to participate in further activities/follow ups and to receive updates.</div>
 
* <div style="">Information to invite users to participate in further activities/follow ups and to receive updates.</div>
 +
 +
<br />
 
* <div style="">Additional information that might be useful to document (if applicable):</div>
 
* <div style="">Additional information that might be useful to document (if applicable):</div>
 
** <div style="">Information to improve the way the test is administered:</div>
 
** <div style="">Information to improve the way the test is administered:</div>
Line 198: Line 206:
 
*** <div style="">Additional audiences that the user would recommend.</div>
 
*** <div style="">Additional audiences that the user would recommend.</div>
  
 +
<br />
 +
<hr />
  
 +
=== Section 6 of 6: Using the Test Results ===
  
=== Using the Test Results ===
+
* <div style="">Prioritization of the issues found:</div>
 
 
* <div style="">Prioritization of the issues found -</div>
 
 
** <div style="">Generate summary report</div>
 
** <div style="">Generate summary report</div>
 
** <div style="">Present findings to stakeholders</div>
 
** <div style="">Present findings to stakeholders</div>
Line 208: Line 217:
 
** <div style="">Review available resources (i.e. time, financial support, staff members, alignment with institutional goals, etc)</div>
 
** <div style="">Review available resources (i.e. time, financial support, staff members, alignment with institutional goals, etc)</div>
 
** <div style="">Select top issues</div>
 
** <div style="">Select top issues</div>
* <div style="">Education - </div>
+
 
 +
<br />
 +
* <div style="">Education:</div>
 
** <div style="">Training the team</div>
 
** <div style="">Training the team</div>
 
*** <div style="">Be able to recognize potential Issues </div>
 
*** <div style="">Be able to recognize potential Issues </div>
Line 215: Line 226:
 
*** <div style="">Promote understanding of how to be more user aware</div>
 
*** <div style="">Promote understanding of how to be more user aware</div>
 
** <div style="">Create/update design practices or guides</div>
 
** <div style="">Create/update design practices or guides</div>
* <div style="">Verification/Validation of the Usability Improvement - </div>
+
 
 +
<br />
 +
* <div style="">Verification/Validation of the Usability Improvement: </div>
 
** <div style="">Reassess usability</div>
 
** <div style="">Reassess usability</div>
 
** <div style="">Measure performance/effectiveness by keeping metrics (number of clicks, amount of time people stay on a page, amount of time people take to find what they are looking for, etc.)</div>
 
** <div style="">Measure performance/effectiveness by keeping metrics (number of clicks, amount of time people stay on a page, amount of time people take to find what they are looking for, etc.)</div>
 
*** <div style="">Examples of techniques that can be used: System Usability Scale, A/B test</div>
 
*** <div style="">Examples of techniques that can be used: System Usability Scale, A/B test</div>
* <div style="">Roadmap Development - </div>
+
 
 +
<br />
 +
* <div style="">Roadmap Development: </div>
 
** <div style="">Make an action plan / improvement plan</div>
 
** <div style="">Make an action plan / improvement plan</div>
 
** <div style="">Continue to:</div>
 
** <div style="">Continue to:</div>
Line 226: Line 241:
 
*** <div style="">Adapt to changing design standards</div>
 
*** <div style="">Adapt to changing design standards</div>
 
*** <div style="">Integrate usability principles in design cycles </div>
 
*** <div style="">Integrate usability principles in design cycles </div>
 
 
 
 
 
<div style=""> </span></div>
 

Latest revision as of 18:37, November 6, 2018

Section 1 of 6: Inviting Users to Participate in the Study

  • Purpose of this Section:
    • The information from this section could be used for identifying/locating users for the user studies.
  • Information Applicable for this Section:
    • Sample message for soliciting users:


Dear [person’s name],
[First sentence should be something that allows the person to recognize the organization, or a particular person that is familiar to them to catch their interest and let them know that this isn’t spam.] We are conducting a study of [product] to better understand [how to organize information, how users interact with the system, etc. Why are you conducting the usability study - very broad statement].  [Provide and explanation of the product, if necessary (1-2 sentences that tells the person what the product is)]. Your feedback will help us to improve this product.
During the study you will be asked to complete a number of tasks using [the product] and provide feedback about your experience. The study will take approximately [time frame (see section further down about appropriate test times)] and will run from [date range]. The study will take place [virtually OR (insert location)]. To thank you for your time, we have [insert the incentive (e.g. $5 gift card) (This is optional)].
If you are interested in participating, please [indicate how they can reply] and we/I will provide you with more information.
  • Tips:
    • Keep the introductory email short.
    • Personalize the email and provide references that the users could relate to, such as an organization or a contact information. Name recognition could help to draw users in.



Section 2 of 6: Test Introduction

  • Purpose of this Section:
    • The information from this section could be used for the following:
      • To explain the process of the user study on the day of meeting with the users.
  • Information Applicable for this Section:
    • Consent to participate in the testing:
      • Possible steps:
        • Step 1: Verify if formal informed consent is necessary for your organization (e.g Internal Review Board - IRB)
        • Step 2: Document any applicable waiver for consent.
        • Step 3: If a consent is required, confirm with the IRB if a consent script would be sufficient. If yes, also ensure whether a waiver for a signed informed consent form is required. An example of consent script and can be viewed here.
        • Step 4: If a consent script is not sufficient and a formal consent form is required, then check with your IRB if there is an official form that should be used. In general, an example of a consent form can be viewed here.
        • Additional references/recommendations per the U.S. Department of Health & Human Services has been summarized and provided by Dr. Robert Downs of the Center for International Earth Science Information Network (CIESIN) at Columbia University, which can be viewed here.
    • Additional areas to discuss with users:
      • Want users to be as natural as possible.
      • Repeat what we’re testing and more information about the product, if necessary.
      • Stress that there are no right or wrong answers. We are testing the system not the user. The user can’t do anything wrong.
      • Go with users’ first reaction and don’t think through things too much. Have some sort of time limit to keep moving things along. Tell them “you will have [time] to complete each task and at the end of that time, I will ask you to move on” “You are welcome to end each task at anytime.” “Go with your first reaction and don’t think through things too much”
      • Users can stop the usability test at any time.
      • If you are recording, to let them know and make sure they are okay with it.
      • Tell them how/where the responses will be used. (e.g. if you will be using responses in a publication, giving them to developers, etc.)
      • If they are on the web, bookmark the homepage, so if they do get lost they can get back to where they started. Let them know how to get back to the homepage.
  • Tips:
    • Don’t read the test user a script here, just have a conversation. In other words, it is important to rehearse and pilot-test the introduction because it is helpful to be as natural as possible when interacting with your test users. If you read them a script, they will feel more like they are being studied. Keep things as natural and comfortable as possible.



Section 3 of 6: Pre-test survey

  • Purpose of this Section:
    • The information from this section would be used to collect information about the test user and establish benchmark for the user’s experience with the system-under-test or with similar systems.
    • The information from this section will be able to help:
      • Prioritize the usability issues found during tests.
      • Understand the differences in test results.
      • Provide user’s context.


  • Information Applicable for this Section:
    • Recommended:
      • We recommend collecting information about the user’s demographic information and experience with the system-under-test.
      • Sample questions that could be used are shown below:


============================================================
  • What is your field of study? (fill in the blank)
Examples for presenting the above questions with predefined values:
research
program management/project management
data management
metadata creating
database administration
ecological modeling
policy making
field work
application development
☐ Other, please specify here:
  • How often do you [use a system that is presented by the system-under-test]?
    • 1 to 5 (Likert scale) 1 being least 5 being most.
  • How would you rate your experience level with [the system that is presented by the system-under-test]?
    • 1 to 5 (Likert scale) 1 being least 5 being most.
  • How often do you [perform the function/task that is being tested]?
    • 1 to 5 (Likert scale) 1 being least 5 being most.
  • How would you rate your experience level with [the function/task]?
    • 1 to 5 (Likert scale) 1 being least 5 being most.
  • Optional:
    • Additional questions for collecting user’s reasons/interests in and expectation with the system-under-test could be considered.
    • Sample questions that could be used are shown below:
      • What are your key job duties?
      • Why are you interested in participating in this test?
      • Is there anything you would like to learn from the experience?
      • Have you used other systems that are similar to the system-under-test?
============================================================


  • Tips:
    • Please note that free text questions can be time consuming for users to answer, so you should make sure that the questions add value to the survey.
    • Additionally, it is more difficult and time consuming to analyze free text questions, so free text questions should be considered carefully before including in the pre-test survey.




Section 4 of 6: Usability test itself



  • Tasks (appropriate to the length of the test)
    • Using other methods (cognitive walkthrough) to determine the baseline for the length of a task. If the cognitive walkthrough is best case scenario (completing tasks perfectly), you know that average users will require substantially more time to complete the task and provide feedback.
    • Doing a pilot test is important for ensuring that the time you estimate for completion is realistic.
    • Based on tests that have been performed by usability labs, it is typically better to design the tasks using an overall time frame (aim for 15 to 30 minutes), then to restrict to a specific set or number of tasks.  



  • Test Implementation:
    • Will it be in person or remote?
    • How will tasks be presented to users?
    • Will you record the usability test session?
    • How will you use the information that you get from the test?
    • Tools: Morae, Loop11, Marvel, (split into free vs. paid), (online vs. software application)



Section 5 of 6: Post-test reflection

  • Choosing when to ask questions after each task versus at the end of the test
    • How much time do you have for the test? You need more time to ask questions after each task.
    • How much time do you have to evaluate the results? This will affect how many questions you want to ask.
    • Are you more interested in the individual tasks or the test as a whole?


  • Guidelines/recommendations regarding how to “interview” the users.
    • Quantitative Methods:
      • System Usability Scale questions could be applied here (To get quantifiable results, you need to ask all 10 questions; however, these could be used for more generic feedback as well if you can’t get through all of them) (scale is 1 - strongly disagree to 5 - strongly agree)
        • 1. I think that I would like to use this system frequently.
        • 2. I found the system unnecessarily complex.
        • 3. I thought the system was easy to use.
        • 4. I think that I would need the support of a technical person to be able to use this system.
        • 5. I found the various functions in this system were well integrated.
        • 6. I thought there was too much inconsistency in this system.
        • 7. I would imagine that most people would learn to use this system very quickly.
        • 8. I found the system very cumbersome to use.
        • 9. I felt very confident using the system.
        • 10. I needed to learn a lot of things before I could get going with this system.
      • If there is time, single-ease question can also be used after each task:
        • Overall, how difficult or easy did you find this task?
          • It uses a scale of 1 to 7 (1 - very difficult; 7 - very easy)
    • Qualitative:
      • Interview Questions (could be better to ask these questions in person than with a written survey):
        • How was your overall experience with the system?
        • Is there anything that you would like to comment on other than what you have already thought aloud?
        • Anything else that you would like to share about your interaction with the system?
        • The questions from System Usability Scale could also be used as prompts for getting qualitative feedback.
  • Information to invite users to participate in further activities/follow ups and to receive updates.


  • Additional information that might be useful to document (if applicable):
    • Information to improve the way the test is administered:
      • What did users think about the use of their time? Why or why not?
      • Was 15 minutes too long? Too short? Efficient use of time?
      • How could the test administrator improve the test experience?
      • Additional audiences that the user would recommend.



Section 6 of 6: Using the Test Results

  • Prioritization of the issues found:
    • Generate summary report
    • Present findings to stakeholders
    • Review goals/mission statements of the project
    • Review available resources (i.e. time, financial support, staff members, alignment with institutional goals, etc)
    • Select top issues


  • Education:
    • Training the team
      • Be able to recognize potential Issues
      • Be able to provide possible solutions
    • Build user-centric culture
      • Promote understanding of how to be more user aware
    • Create/update design practices or guides


  • Verification/Validation of the Usability Improvement:
    • Reassess usability
    • Measure performance/effectiveness by keeping metrics (number of clicks, amount of time people stay on a page, amount of time people take to find what they are looking for, etc.)
      • Examples of techniques that can be used: System Usability Scale, A/B test


  • Roadmap Development:
    • Make an action plan / improvement plan
    • Continue to:
      • Involve stakeholders in discussions
      • Educate team members
      • Adapt to changing design standards
      • Integrate usability principles in design cycles