Intelligent web app to support Personalized, Efficient and Self-regulated math learning experience
DURATION
8 Months
Jan.2018 - Aug. 2018
@Capstone Project CMU
CLIENT
(Chinese Leading Edu Tech Company with 10M+ users)
MY ROLE
-
The sole designer on the team leading the UX process
-
Designed and conducted user research and user testing
-
Created and iterated wireframes and hi-fi prototypes
-
Organized in-team brainstorm activities and product critiques
-
Engaged in algorithm design
TEAM
Siyu Chen, Tianmi Fang, Karen Ning, Lu Sun, Yeyu Wang
INTRODUCTION
WHAT IS THE PROBLEM?
Many Chinese high school students fail to achieve their full academic potential in learning math, because of low student-teacher ratios, intense curriculum, and high learner differences.
WHY IT IS IMPORTANT?
Earning good grades in the Math Exam of Chinese National Higher Education Entrance Examination is crucial for students to be admitted to universities, however, many students struggle to learn math and lose their chances to attend good universities.
HOW DOES OUR PRODUCT HELP ?
PERSONALIZED
EFFICIENT
SELF-REGULATED
A self-study web app that creates a personalized, efficient, and self-regulated learning environment to prepare students for the Math Exam.
WHO ARE OUR USERS
1
Wei Li
-
Wants to enter TOP 100 universities in China
-
Doing well in the current academic performance but still has growing potential
She values ...
Practicing and reflecting efficiently
Improving math fluency
Developing independent thinking
2
3
1
ADVANCED LEARNER
Ming Chen
-
On the edge of being admitted to universities
-
Does not receive enough help from teachers when struggling with learning math
He values ...
Developing knowledge comprehension
Timely and detailed feedback
Becoming more aware of mastery
2
3
1
UNDERSERVED LEARNER
HIGHLIGHTED FEATURES
MINIMAL VIABLE PRODUCT
Question Recommendation
The adaptive learning helps students learn efficiently by recommending personalized learning paths based on evaluations of mastery levels of knowledge.
01 How might we optimize students' effort in learning math?
Hover on the dots to read relevant learning phases
*We tested the MVP with 19 Chinese high school students. Each student attended 4-5 days of on-site observation and completed about 50-60 questions on EdThor web
31.9%
Students gained 31.9% learning improvement after 2-hour practice separated in 4-5 days.
100%
100% of students preferred the adaptive practice than their traditional practice methods.
Traditional Learning:
Mindless receptions
Adaptive Learning:
Efficient practice
Students can practice question types and content that they will encounter in the examination.
Practice Report
Give timely feedback on students’ learning progress and knowledge mastery changes at the end of each practice.
21.6%
In 21.6% of practice, students were motivated to "Try More" because they thought their learning performance was below expectation.
02 How might we empower students to monitor on their strengths and weaknesses?
Dashboard
Provide visualizations of knowledge mastery levels and organization of knowledge components, as well as the summary of recent learning performance.
100%
of students valued visualizations and reflected on their performance.
03 How might we help students learn to apply formulas?
Formula Pool
When practicing, students can refer to formulas pools to search for formulas and memorize by applying them to solve problems.
81.3%
In 81.3% of practice, students used Formula Pool at least once, and memorized formulas better by applying. They got 87.5% average accuracy of recalling formulas they checked in tests.
04 How might we help students develop deep understanding?
Expert Answers
Provide detailed solution steps aligned with assessed knowledge.
11.4%
In 11.4% of questions that students got correctly, they still checked expert answers to learn deeper.
SUPPORTIVE FEATURES
We also develop 5 supportive features in our product to support the other three learning phases.Please click the navigation button below and enjoy the features.
The system provides worked examples and knowledge component instructions to help students transfer knowledge from instructions to solve the problem.
85%
of students reviewed learning materials in practice
05 How might we provide detailed feedback while facilitating independent thinking?
DESIGN PROCESS
EXPLORATION
It's always exciting to start a new project from the beginning. The client reached out to us in need of a new adaptive learning tool for Chinese high school students. After discussing goals and expectations with clients, we began to define the project following the three research questions:
-
How does adaptive learning work?
-
Who are the most important stakeholders and How do they influence?
-
Why are popular e-learning products successful in Chinese or the U.S. education technology market, especially products for learning math?
Literature
Review
40+ peer-reviewed papers about adaptive learning
Stakeholder
Analysis
Based on prior research conducted by Cross-Domain
Competitive
Analysis
14 popular e-learning products for K12 in China and U.S
Research Insights & Design Decisions
01 Defined users, stakeholder and user cases
We defined advanced students and underserved students as our initial users, who are a representative set of the whole population. We also designed school teachers as our major stakeholders. Both users and stakeholders were highly-interested in after school support.
02 Identified the design opportunity and business value
Adaptivity is missing from Chinese online-learning platforms. We decided to select one module in Chinese high school math education as a starting point. Therefore the solution could be generalized easily in the future.
03 Started Data-driven design
From literature, we identified different inputs from the students and outputs from the system to support adaptive learning and started to build student modeling using prior student data.
RESEARCH & SYNTHESIS
Before jumping to the design, it's crucial to understand users in context.
-
What is an appropriate starting point to develop adaptive learning and Why?
-
How do teachers support different student groups in learning math effectively?
-
What do students value or find frustrations in learning math?
-
How do we transfer research findings into design opportunities?
Learning Data
Analysis
800+ students data of 132 questions in exams
Teacher
Interviews
Interviewed 3 high school teachers
Student
Interviews
4 advanced students and 3 underserved students
Research Insights & Design Decisions
01 Selected learning materials for MVP design
We selected the "Trigonometric" module as the learning materials in our product. We made this decision because student performance varies largely on different knowledge components in this module, which has the most potential to be improved by adaptive learning
Average Correctness and Deviation Among Knowledge Components in Trigonometric
02 Adopted strengths of teachers in our system
Experienced teachers focus on students' understanding and application on problem-solving strategies and provide targeted instructions based on over-time assessments. However, because of the low teacher-to-student ratio, there is not enough attention for each individual student. It inspired us to adopt those strengths in our system and scale up to serve more students.
03 Consolidated the Sequence Model to reveal
learning steps and breakdowns
Learners follow different paths based on different triggers and motivations, which shows the great potential of adaptive learning.
04 Identified design opportunities by the
Affinity Diagram and Sequence Model
Based on affinities and the sequence model, we synthesized user needs and identifies design opportunities with 7 How Might We questions, which served as a guideline for brainstorming solutions in the next stage.
HOW
MIGHT
WE
Provide personalized feedback which can help different types of learners in practice?
Help students improve problem-solving skills more efficiently?
Provide reliable, timely and targeted feedback in practice?
Help students explicitly align questions with assessed KCs and develop deep understanding?
Help students optimize their effort in learning math?
Empower students to monitor their strengths and weaknesses?on
Prompt students to learn from mistakes effectively and efficiently?
IDEATION
150+ Ideas
From Brainstorming
23 Scenarios
Synthesized ideas across all learning phases
7 Features
3 MVP Features
4 Supportive Features
Brainstorming
In order to bounce off ideas, I organized a 4-hour in-team brainstorming activity to walk through all the personas, affinities and sequence model to ideate design solutions. Everyone was encouraged to put post-it near the needs to show their design ideas. Finally, 150+ design solutions have germinated.
Ideation Metrics
In the "Ideation Metrics", we synthesized the original solutions into 23 ideas that across all learning phases and target on different needs. The rows are divided by needs generated from user research, and the columns are divided by learning steps we adopt from sequence model and literature.
Elaborating Ideas
We crystallized all ideas into Scenarios to elaborate solutions with use cases. I also visualized them into Storyboards and presented to the clients (CTO/CEO/VP) and subject-matter experts. To better evaluate our ideas, I created a three-dimension evaluation metrics including
Desirable
Take learner difference into account
Construct knowledge organization
Prompt goal-directed practice
Provide targeted feedback
Develop positive social climate
Scaffold self-directed learning
Viable
High-business value
High-reward innovation
Feasible
Easy to measure success
Low technology limitations
Generalizable in the future
Examples of storyboards
Defining MVP features and supportive features
The process of selecting solutions to be developed in the next phase was poignant yet exciting. We were gradually developing a more clear vision of the product. To support a complete learning cycle, finally, we picked and elaborated 7 ideas based on the in-team evaluation, and suggestions from our client and professors.
At this point, we also defined our MVP features. The decision was driven by three major reasons
-
The four ideas compose a minimal yet complete practicing experience
-
The solution is unique
-
The data we got from testing MVP can benefit the future product design
ITERATION & TESTING
The whole iteration process lasted 12 weeks and all the steps are visualized in the graphic. It's not surprising that we changed our ideas so much since we learned new things from each round of testing.
In this phase, I conveyed ideas into visible and useful artifacts and collaborated with the researcher to design all test protocols. Generally, the success matrix was developed based on three dimensions: functionality (focusing more on learning effects), usability, user experience.
We teste MVP features firstly and iterated it with Qualtrics prototype, high-fi prototype and finally developed our responsive website to conduct on-site test and observation. We also designed and iterated supportive features to build a complete learning cycle in our product with MVP Features.
The followings would go through the iteration process for MVP features. Though sometimes we got negative feedback from users, instead of just giving features up, we regarded them as opportunities to improve design and tested iterations with users.
Recommend questions based on the assessment of students' mastery level
As the UX designer, I believe the elaborated UX design and advanced algorithm are equally important to ensure the success of question recommendation. I iterated the relevant features in three rounds of user tests with mid-fi and high-fi prototypes.
Empower students to be aware of the knowledge organization and their mastery levels
Scaffold students with hints to understand problem-solving strategies
Prompt deep reflections on mistaken steps in practice
Site Map
Based on feedback generated from user testings, we redefined the user flow of the whole learning journey.
Validate the design
Our testing results for the MVP and supportive features are highlighted at the beginning with the final design. We are proud and confident to claim, our minimal viable product is successful, and has great potential in the future development.
MOVING FORWARD
Large-scale Testing
To gain more convincing results, we recommend a large-scale comparative experiment in the future. We sent a comprehensive test proposal and prepared all test materials for our client.
Generalizability
Based on effective learning impact we got in testing, our process of adaptive learning system design can be generalized to other modules
Business Value
When scaling up education business by applying the adaptive learning design, it will help business such as Cross Domain to lower marginal cost without sacrificing effective learning results.
REFLECTION
I have experienced and learned so much from this eight-month project in a team of five creative and innovative people.
The balance of learning and user experience
Learning takes effort, learning was born with frustrations and failures. Sometimes learning makes user experience unpleasant and intuitive, and great user experience cannot always guarantee learning gains. During the whole design process, I was playing with a seesaw to find the best balance for an effective yet delightful learning. The most useful strategy I used is motivating students to take efforts by visualizing the outcome. For example, arousing their intrinsic motivation of practicing by showing the change of knowledge mastery.
Drive the design process by questions
My courses at CMU always started with "learning objectives", and my design process in this project is driven by "research questions". It serves as a clear guideline for me to articulate what I expect and choose suitable UX tools and methodologies.
Discover new design opportunities from failures
A successful design requires strict incubation conditions: identifying true needs, reliable functionality, and delightful user experience and so on. It's not surprised to receive negative feedback in user testings, but it's awful to stop when meeting failures. Because of my background in psychology, I'm always curious about the deep reasons behind human behaviors. I like observing, not only looking at, users in the testing and dig out insightful results than like or dislike.
