Role
Project Manager
Researcher & Designer
Date
October – December, 2020
With
Yiting Zhao, Xuekun Ma, Feijia Sun
Where
Design Informatics, University of Edinburgh
Motivation and Research Questions
The Social Isolation caused by Covid-19 has presented unprecedented challenges to students’ learning. At the University of Edinburgh, we had to adopt a mode of learning called Hybrid Learning: attending classes through online videos, experimenting with various flawed online learning tools, and collaborating with classmates remotely on assignments. As students, we were acutely aware of low levels of engagement among our peers and ourselves in the online learning environment: hardly any classmates would turn on their cameras, few would respond to the teacher's questions, and group assignments often encountered difficulties that were hard to overcome ... We wondered: if online learning will become the norm in the post-pandemic era, how can these issues be addressed?
Thus, our research question is:
Can the current teaching services offered by Design Informatics meet their engagement needs?
What factors affect their engagement in online learning, and what can we do to improve it?
Design Process
We learned about service design theory and methods throughout the project. In the book This Is Service Design Thinking, Mark Stickdorn (2011) outlined four stages of service design as Exploration, Creation, Reflection and Implementation. According to it, we clarified our design process, considering the objectives and timeline of the project.
User Research
Our research methods included culture probe, observation and semi-structured interviews.
When designing the questions for our culture probe and interviews, we adopted three dimensions of measuring engagement in online learning proposed by Singh et al: cognitive, behavioral and emotional engagement (2018). We were also inspired by Piki’s integrated framework of mobile computer-supported collaborative (mCSCL), which discussed six external factors that influence learners’ engagement in three similar dimensions (2017). Thus, our research objectives were divided into two parts, one understanding the DI students’ engagement in three dimensions, the other exploring the impact of different factors on it, including personalities, tools, interactions, etc.
Culture Probe
To get access to students’ real performance and feelings in the learning process, we adopted the cultural probe. The probe is a diary allowing participants to document their learning activities and feelings during a week. Specifically, we give each participant some emoji stickers to help them express their feelings.
The culture probe.
The probe is divided into two parts—online class and after-class activities. In each part, participants need to document their performance, feelings, and self-evaluation.
To make the findings more diverse and inclusive, we recruited participants from different native language and culture backgrounds, as well as various locations.
Observation
We also observed and documented the interactions in classes during the same week. Almost all teachers said they wanted students to turn on the camera to get their real-time feedback and told them they could ask questions at any time and not to be afraid to interrupt the class. However, almost no students turned on the camera and few questions were asked although teachers always encouraged them to do so.
Interview
After collecting probes, we did in-depth semi-structured interviews based on participants’ probes to learn more. In this process, we acquired their learning preference, reasons for specific feelings and behaviors as well as potential difficulties they have met during the online learning. The questions are based on three dimensions of engagement and influencing factors.
Analysis
We analysed the result of interviews through coding and thematic analysis. The analysis result is divided into two sections, in-class and after-class, while each section has both positive and negative parts. Influencing factors mentioned by participants are categorized into many aspects including personal background, personality, support from teachers and tools, etc.
Then a journey map was made based on participants’ goals, touchpoints, channels, and feelings (emotions) in 4 phases of online learning.
The journey map.
One of the biggest findings in the research was that we identified three different types of learners — proactive, neutral, and passive. Of these, passive learners with strong needs became our primary focus in the subsequent design.
Personas of three different types of learners in our research.
Insights from User Research
Key insights from user research forming the basis of our artefact are as follows. TL;DR? Just skip it :)
Cognitive
1. Reading relevant material before class will increase the number of times students ask and answer questions in class.
2. Passive students hardly ask questions, but often look at the questions asked by others.
3. Current questions about study materials are usually asked individually but students want to discuss them with others.
4. Asking questions noticed and answered by teachers can greatly increase students’ engagement.
Emotional
1. Students are reluctant to express negative emotions directly to the teacher during lessons.
2. Courses that have clear structure, tasks and guidance are more popular.
Behavioral
1. Using a real-time handwritten board can improve student focus.
2. Students are easily distracted in online courses, partly because some courses are too long or they lack interest in the content.
3. Courses that are fast-paced or carry out some activities will increase student focus.
4. Passive students show a preference for video recordings. Although they do not often ask teachers questions, they usually think deeply when reviewing the recordings.
Camera
1. Students are reluctant to turn on the camera mainly because they are not confident about their appearance or the surroundings.
2. Peer pressure makes some willing students hesitant to open their cameras.
3. Seeing students’ facial expressions and body language can help teachers understand if students have questions or if they are interested.
Group Work
1. The more students contribute to the group and the more positive comments they receive, the more engaged they feel.
2. Different roles naturally arise in the group. The leader and discussion moderator usually feel more engaged than followers.
3. Tasks with division that allow each member to have a clear contribution can increase their engagement.
Tools and Language
1. Course materials and teacher notifications are always found in different platforms.
2. There is a strong connection between students’ sense of engagement and their familiarity with and interest in the course content.
3. Language barriers may increase engagement in group work as it allows participants to listen and express themselves more attentively.
Prototype
The next task is to distill these findings into concrete requirements and translate them into design ideas.
As we found so many needs, it seemed that only a one-stop online learning platform could solve so many problems. We envisaged its framework and 5 main functions, based on the channels we had previously analysed: Video Conferencing, Learning Materials, Announcements, Chat Tools, Assignments Submission.
However, due to time constraints, we decided to focus first on the pain points related to Video Conferencing and Learning Materials — as the tools we currently use (e.g. Zoom, Teams) cannot meet students’ needs in both areas.
Some of the special features in our design include virtual animated avatars (Memoji), setting class breaks, public platform for discussion of study materials, etc. Please click to view the images below to see exactly how we have responded to our insights. It is worth noting that during the design process we kept the three different types of learners in persona in mind and prioritised the needs of passive learners — as they are the most vulnerable.
Distill our findings, seek solutions, and translate them into design ideas.
Prototypes: Video Conferencing.
Prototypes: Leaning Materials.
Privacy risks are considered in our design. Research by Dima Kagan et al. (2020) has shown that easily leaked screenshots of video conferences can seriously damage the privacy of participants. So we proposed the use of virtual animated avatars to represent expressions, which can avoid this problem to some extent.
You can try out this prototype by clicking here or find out more in the video below.
The presentation video of the final prototype of the project.
User Testing
We tested the prototype by assuming several different usage scenarios: pre-course reading, video conferencing, and post-course review. Participants covered all three personas. We presented and allowed them to try our prototype and recorded their feedback. Some representative and problem-related feedback are shown below.
Reflection
On the whole process
First, there are more factors affecting engagement than we thought, and the service design itself is very complex. Time was limited and we were unable to design the entire service. We should have narrowed down the research question at the outset and focused on one aspect, such as engagement in video conferences.
Second, we did not consider students’ personality differences at the beginning, whereas in the research process we found it to be an important influencing factor and reflected this finding when creating personas. We are particularly concerned about the needs of negative learners, as they are the most vulnerable and have the most urgent needs. However, it is unclear whether the negativity exhibited by these learners is a result of the online learning modality itself.
On the prototype
Reading study materials and discussing openly
Beneficiaries: 1. Active students who want to think about questions when reading study materials; 2. Passive students who want to gain from reading other people's questions and discussions; 3. Students who want to remain anonymous in discussion.
Maleficiaries: Students who have known the knowledge or feel uninterested towards reading study materials.
Virtual animated avatars
Beneficiaries: 1. Students who feel that turning on the camera will increase their engagement, but are reluctant to do so for various reasons; 2. Users who care about privacy; 3. Teachers who want to know students’ response through their facial expressions.
Maleficiaries: 1. Users without the relevant device, as Memoji is currently only available on Apple devices equipped with the TrueDepth camera; 2. Users with internet connection problems; 3. Users who want to see ‘real people’.
Technology Perspective
Some subtle expression changes, such as confusion or lack of concentration, are difficult to be recognized and presented by Memoji because of the high inter-class similarity of facial expression recognition and classification (Li, S., & Deng, W. 2020). We need to acquire more facial features to capture more subtle changes in expression, but current computational efficiency of most devices is limited.
