top of page

Bilingual E-book Project

Develop and evaluate an interactive e-book's usability and learning effectiveness to inform the design of English learning technology for EFL children

Product

Chinese-English Bilingual Interactive E-book

Time

August 2020 - August 2021

My Role

UX Researcher & Designer, Project Manager

Team

I worked with a team of 1 principle investigator, 3 designers, 30+ student research assistants, 4 engineers, and 200+ participants.

Methods

Literature Reviews, Contextual Inquiry, Usability Testing, Controlled Study, Generative Research, Evaluative Research, Quantitative Analysis, Qualitative Analysis, Presentation, Publication

Ebook_firstpage.png

Overview

Through literature review, contextual inquiry, participatory design, and usability testing, I led the design team and collaborated with engineers to create a touchscreen storybook with 5 interactive features. In addition, I led a team of 30 student research assistants to evaluate the product with 100+ families from China. Our findings from this study reduced the knowledge gaps in the HCI and EFL literature and informed the designs, user experiences, and educational values of e-books for children and parents from linguistically diverse backgrounds.

Problem

How might we promote children's reading engagement and learning effectiveness during English e-book reading?

Research Goals:

  1. Understand the challenges and expectations of Chinese families during English e-storybook reading

  2. Design a multimedia interactive storybook to meet the user needs

  3. Evaluate the usability and effectiveness of the innovation to inform future designs

Generative Design Research

Exploring the E-book Design Field - Literature Review

  • Interactive features add entertainment but distract learning

  • Children quickly lose focus and couldn't process information independently

  • E-books encouraged independent reading but took away the benefits of parent-child shared reading

  • Design for L2 or bilingual settings is dismissed

Understanding the Users

Recognizing the crucial roles of stakeholders and their valuable insights, we adopted key principles and techniques from participatory design and contextual inquiry into our design research. Below is a graph of our entire research and development procedure. 

Participatory Design.png
E-book Development Procedure

Understanding User Behaviors and Demands - Contextual Inquiry

We interviewed 3 families (children and mothers) from medium-sized cities in China to identify user demands and the design challenges for this particular population. We inquired about 1) the families' digital English literacy practices, 2) their impression and expectation of e-books on the market, and 3) their challenges and needs in children's English learning. 

Insights from Contextual Inquiry

English storybook apps are widely used in Chinese Families​

Themes

Shared reading is a popular and cost-effective home literacy practice

Children preferred reading e-books with tablets than printed books

Parent's limited Englsih proficiency prevented parent's shared reading engagement and ability to initiate discussion​

User Problems

E-books for EFL children on the market focused on building vocabulary and have simple and unengaging story plots

Embedded dictionaries of these E-books often lack sample sentences and developmentally appropriate explanations, so children quickly lose interest

Design Challenges

Involving parents with limited skills in the English story reading process

Create developmentally appropriate and engaging content

We transformed these insights to create our storybook draft, which has an engaging plot, interesting characters, and developmentally and culturally appropriate themes for children in our target population.

User-centered Lo-fi Prototype - Participatory Design

We conducted lo-fi prototyping with the 3 families, using the Bags of Stuff technique.

 

We narrated the story to the participants, as the children and parents contributed to the mock-ups by visualizing and drawing with art supplies.

 

Participants also discussed their expectations of the e-book and the reading experiences.

Parent and Child Codesign.jpg
Parent and Child Co-design Page

User Persona

Results from the contextual inquiry and participatory design revealed the two personas which we levearged into our design.

Google UX Design Certificate - Persona [Template].png
Google UX Design Certificate - Persona [Template] (1).png

Hi-fi prototyping

​With the preliminary findings, I led a team to build the e-book HTML5 file using Adobe Captivate and collaborated with the engineers to turn our product into a web-based system that used React as the basic framework. Based on the research insights, we designed 5 e-book features:  1) English Narration,, 2) Bilingual Support 3) English Dictionary,  4) Character Statement, and 5) Discussion Prompt.

Demonstration of Character Statements and Dictionary Hotspots
Demonstration of Discussion Prompts

Testing the Hi-fi Prototype - Usability Study and A/B Testing

At this stage, our team had 3 overarching goals:

  1. Observe how users interact with our E-book website and make a final iteration

  2. Decide whether we should include a bilingual support feature because some stakeholders held concerns

  3. Pilot test for our large-scale evaluation study and address potential problems in the research plan.

​

To meet these goals, we combined A/B testing and usability study with a larger sample size of 14 families. We assigned 8 families to the bilingual support version and 8 others to the monolingual version.

Parents
Bilingual E-book Persona Mom picture.png
  • Most were satisfied with the reading experience
  • Parent-Child shared reading was more effective than having children read alone
  • L1 support is necessary for promoting parental involvement and optimizing children's learning
  • 3 families from the monolingual version group dropped out because it was too challenging 
  • Frequently activate vocabulary hotspots and repeatedly play the English words and sentences
Children
Bilingual E-book Persona Kid picture.png
  • Most were satisfied with the reading experience
  • Greater interest in the character statement where they simulate "talking with the characters."
  • Parents' frequent activation of vocabulary hotspots and narration diminish their user experience and motivation for reading

Turning insights to design iterations

From the usability study, we revealed several challenges and tensions for the e-book design and hold meetings within the research team to discuss solutions. To balance the power between the two stakeholders and boost children's motivation, we adjusted the number of dictionaries and character statement hotspots in our next version. 

 

In addition, we also found opportunities to improve the visual layout, e-book content, and user flow. With the research insights, I led the design team and collaborate with the engineering team to adjust the e-book content, fix bugs, and run internal trials for the final e-book version to increase the system usability score by 42.6%.

Key Updates:

  • Bilingual support as a standard feature

  • More character statements

  • Reduce dictionary links

  • Replace ambiguous dictionary illustrations

  • Adjust font size and text position

  • Upgrade website bandwidth

  • Minimize file size

  • Add an animated pre-loading page

  • Establish consistent code names

  • Eliminate possible glitches

Testing the improvements:

After the modifications, I led the design team and asked 2 additional student research assistants to run internal tests on the ebook and make sure that the problems have been addressed. We also tested our products with 3 participating families and found that the system usability score significantly increased from the previous usability study.

Aquablue and Black & White Photos Marketing Report.png

Maintaining Research Rigor:

I also reviewed the study videos/logs and held meetings with research assistants to discuss problems they encounter during the study. We observed some ambiguous photos in the test material and some research assistants' tendency to help the child participants. Collaborating with the essential stakeholders, I refined the study materials and protocols and provided additional research training to the student research assistants.

Evaluative Research

Research Questions
 

  • What is the educational value of the designed e-book app?

  • How do children's characteristics and parents' English proficiency level affect learning outcomes?

  • What do parents and children think about the e-book reading experiences and the e-book features?

To answer these questions, we recruited 83 families who met the inclusion criteria: 1) Children are 3 - 7 years of age, 2) First language is Chinese, 3) No severe disabilities which might hinder reading, 4) Accessible to touch-screen tablets or computers

Procedure

Ebook_Evaluation_Process.png

Step 1: To better understand the user and gain information for later analysis, I collaborated with the principal investigator and faculty advisors to design a survey that inquired about the families' basic demographics, English proficiency, and home literacy practice. In addition, children's English ability and storybook-related knowledge were assessed with a pre-test.

Step 2: I led the research assistants to communicate with participants and coordinate the experiments. The reading sessions were recorded via Zoom and the human-computer interactions were captured by our website.

Step 3: To evaluate the e-books' educational value and usability, I also designed the post-test, interviews, and surveys which combined to reveal children's learning outcomes, the user experience, and the e-book's usability.

Step 4: In addition to managing and documenting the research progress, I also work to identify and prevent abnormal data or research misconduct by holding meetings and maintaining research rigor with additional team training. In addition to this evaluation study, I also trained research assistants for observational coding and led several teams to answer other research questions with our data and present the research insights in the form of publications and presentations. 

demonstration_readingsession.png

Qualitative Measures

User Experience

  • Overall Reading Experience

  • Perceptions of E-book Features

readingsession_demonstration.png

Quantitative Measures

English Story Vocabulary (Cronbach Alpha: 0.97)

  • Expressive Vocabulary

  • Receptive Vocabulary

Findings

Learning Effectiveness

We conducted paired t-test to examine the learning effectiveness of our e-book.

 

Our analysis revealed that children significantly improved in both receptive (t (82) = 12.60, p < 0.001, Cohen's d = 1.38) and expressive vocabulary (t (82) = 9.98, p < 0.001, Cohen's d = 1.09) after reading the e-book twice with their parents.

unnamed (1).png
User Demographics and Learning

To better understand how children's initial English skills and demographic characteristics relate to their learning, we also conducted a regression analysis.

 

Results showed that children's baseline vocabulary scores had a quadratic relationship with story vocabulary gain (b = -0.1, p < 0.001).

 

This suggests that children with either low or high vocabulary can experience lower vocabulary gain because of limited ability or the ceiling effects.

Overall User Experience

From the interviews, we found that 94% of children enjoyed reading the e-book and 92% of parents considered our e-book as an effective and enjoyable English learning tool.

We also interviewed children and parents about their perceptions of each of the e-book design features. The graph below shows the percentage of participants who like each of the features.

Brown Modern Data Growth Bar Pie Chart Graph (2).png
User Perception of Story Narration

92% of parents and 82% of children liked the English story narration.

 

Parents:

  • Narration provides an authentic and appealing story-reading experience

  • Narration maintained children’s attention

​

Children:

  • Characters' dramatic voices are attractive

  • Less appealing to children with limited English proficiency

User Quotes:
“I would not be able to help my kid pronounce well or emerge him in the story without the narration.”
“I like that the black bird has a husky and scary voice and the Little Oak sounds cute.”
User Perception of Multimedia Dictionary

92% of parents and 88% of children liked the Multimedia Dictionary

 

Parents:

  • Combined education with recreation

  • Overuse can cause cognitive overload

​

Children:

  • Illustrations, animations, and sample sentences are attractive

  • Empowered with the agency provided by dictionary hotspots

User Quotes:
“It is fun and at the same time, helps reinforce vocabulary learning”
“The fun pictures enabled me to have a deeper impression on the new words.”.
“When I don’t understand something, I would simply click on it.”
User Perception of Character Statement

Children’s (90%) and parents' (96%) favorite design feature

​

Parents:

  • Sustain children's interest and attention

  • Children can be too occupied with this function to actually learn from the story plot if they read alone

​

Children:

  • Fun and engaging

  • Simulate verbal interactions with story characters

User Quotes:
“It’s great in that it brought the story to life”
“My child listens to the story more carefully with these character statements.”
“... I can talk to them and make friends with them”
“It’s just so much fun to speak with them.”
User Perception of Discussion Prompts

Discussion prompts receive the most divergent opinions from the participants, with 82% of children and 98% of parents liking the feature

​

Parents:

  • Encourage story engagement

  • Promote story understanding

​

Children:

  • Elicit contingency with the character

  • Rewarded by the positive feedback

  • Some reported that the feature interrupted their reading flow

User Quotes:
“It triggers him to think logically,”
“The feedback is like a reward that makes her happy.”
“I love it! It helps my brain run really fast!”
“I love it when the Little Oak raises a question because I could learn more about him,”
User Perception of Bilingual Supports

90% of parents appreciated bilingual support. No feedback from the children because they can't comprehend the Chinese translations. ​

​

Parents:

  • Boost confidence in explaining the story to their children

  • Could be optional because older children can read Chinese and interfere with English learning

User Quotes:
“It’s the best! I love the Chinese translation because I have limited English skills. I would not know what to do without it,”

Impact

  • Revealed the special design needs of the EFL population

  • Created an effective and enjoyable English learning tool for children in the EFL setting

  • Informed E-book designers and researchers with presentations and publications

Additional studies from our project challenged the gold standard of e-book design established in the relevant literature. We also analyzed the parent-child-computer interactions during their shared e-book reading activity to inform more effective shared-reading practices.

Self-Reflection

Taking a 3-fold responsibility and following a project from end-to-end allowed me to practice the full spectrum of a product development cycle. I realized that the behaviors, needs, and challenges can be very different across different user populations. For multi-user products especially, it is crucial that we identify the user dynamics and balance the power between them.

Other Studies from the Project

bottom of page