Usability Testing Update! OLT Innovation & Development Grant ID13-2658

We have been working on an OLT innovation and development grant led by Associate Professor Denise Wood titled, ‘An evidence-based approach to the design and redevelopment of inclusive technology enhanced learning environments‘.

We nominated our first-year course ‘Introduction to Programming‘ to be involved in usability testing and redesign. Effective design for any course is important but we wanted to focus on this course as these are first-year students and retaining CS students is one of our key priority areas. First year students are already overwhelmed with information, are learning to navigate university learning management systems, and in CS learning to program for the first time, so we want to ensure this process is as smooth as possible.

At some stage, we have all encountered a website which can feel like a “where’s Wally” hunt just to find the information we are looking for and we can only imagine what this might feel like as a student navigating an online learning environment for information with the pressures of a deadline, working in a second language, or those with a disability! Often as educators, we will put materials online thinking that students will find them, but students may not always think to look in the places that we think are obvious! Materials and information that are difficult and time consuming to locate can cause a frustrating experience for students. The goal of this OLT project is to use evidence-based research to evaluate online learning designs so that we can improve the online experience for all students.

The research process involved students (who volunteered) to perform a series of 10 tasks, such as ‘locate the readings for week 4’ or ‘find the course coordinator’s contact details’. Students worked through the tasks using Morae software and a ‘think-aloud’ approach; providing feedback on ease of task completion. The goal was to use this feedback to determine if content and materials were easy to locate, access and use for students.

As a team, involving the course-coordinator (A/Prof Katrina Falkner), project leader (A/Prof Denise Wood), project manager (Dr Sheila Scutter), online learning designer (Dr Tracey Johnson) and myself, we met to discuss the findings from the usability testing while also viewing the course website. Together, we devised some key recommendations for the redesign. Further, Tracey showed us examples of online learning designs that could provide inspiration for how we might consider the course layout and structure of materials. Using the recommendations, Katrina and myself were able to consider what the new course design might look like.

Katrina redesigned the course based on the feedback and recommendations. The new course has a more simple, clean design. A large part was to remove all unnecessary materials – keeping the core required materials and essential course information. In the new design, additional and ‘interesting’ articles were now shared with students via news updates and in the ‘additional’ materials section. We structured the course around weeks and had a clear tab-section for ‘assessment’ which included weekly workshop tasks. The links between the course and the official course profile and learning outcomes has been strengthened. Katrina also placed the discussion forum and news forum at the top of the page for easy student access – to assist those seeking and providing support, as needed.

We are currently running usability testing on the new cohort with the new design so that we can make comparisons and determine if the new layout and structure of course materials works for students. We will use these findings to make any further amendments. All the feedback and recommendations will be going toward an overall set of guidelines that will be developed for academics based on all of the courses involved in the OLT project.

The results were interesting and the process of carefully critiquing the course layout and presentation of materials has proven to be a worthwhile experience so far.  One suggestion that we can make based on our process of analysis and redesign is to separate all required and optional materials so that it is clearer for students what is expected to be completed – as well as making it easier for students to navigate course materials.

Another valuable lesson we have taken away from this project is the value in simply critiquing and checking one’s own website before the course begins: putting yourself in your students’ shoes. It is difficult when lecturers are pressed for time but this time taken to examine course layout (e.g. Do links work? Are materials accessible? Are your contact details clear?) can improve the student experience and potentially save you time in the long-run. Having a critical friend provide feedback on your course design may help when preparing your course for the new cohort. Often it is easy to overlook these things when we are viewing our own design – it’s like when editing your own work. Sometimes we don’t always see something like a missed word – because we know our own work so well and have seen it so many times before! But to have someone else read over our work can be so useful in picking up little things we did not notice ourselves. You could even give a colleague some basic tasks to do or swap and/or critique each other’s courses!

We are not certain if the new design is the right design but it seems to suit our particular context and our re-testing will help us to further improve the current design. It will be interesting to see the designs others involved in the OLT project have created based on the usability testing. Another aspect of this project involves usability testing with students with disabilities and the development of an open source software that students with disabilities can use to create personalised learning environments (to automatically customise font, volume, colour, etc). We look forward to analysing the findings about the usability testing on the new design and we will keep you posted about our future work in this space as well as progress of the overall OLT project.

To view the presentation we gave based on our process so far, you can visit our Prezi here and the full project website here. We will continue to update the Prezi as we progress through the project!

In the meantime, let us know in the comments below of your experiences using online environments or any tips that you might have for fellow colleagues designing a course for the upcoming semester!


This entry was posted in CS Analytics, News, Projects. Bookmark the permalink.

Comments are closed.