Language Learning Visualization Platform

January 2022-May 2023
‍
Project Title: Effectiveness in Improving Foreign Language
Pronunciation through Visualizations

Project Overview

This is an HCI-focused research project I am working on with PhD student, Charlotte Yoder, as part of the Social Spaces group at the University of Illinois at Urbana-Champaign’s Department of Computer Science. This project is co-advised by Professor Karrie Karahalios and Professor Mark Hasegawa-Johnson.

In this research project, we are developing a language learning platform to measure the effect of visualization and teaching methodology on pronunciation gains. This platform’s goal is to help learners of a second language improve their pronunciation through the use of visualizations as improved pronunciation has a positive impact on language use and relationships formed as a result.

My Role

Undergraduate Research Assistant
‍Data Extraction, Systematic Literature Review, Qualitative Coding, Data Analysis, Interactive Visualization, Data Visualization

I contribute to this project as part of the Illinois CS Student Ambassadors/Research Scholars (CS STARS) program. This program funds undergraduate students to research in their chosen area of interest while also serving as departmental leaders to recruit and empower female and non-binary students to pursue computing majors.

For the purpose of this portfolio, I will be highlighting areas of the project which I lead and contributed to. As the project is ongoing, certain aspects such as the pronunciation-measuring algorithm will be omitted.

Abstract

This project aims to utilize pronunciation visualizations to help L1 (native language) English speakers improve their pronunciation of Spanish. We seek to help individuals learn a second language by improving their pronunciation through visual representations of sounds and spoken language.

The purpose of this project is to ascertain if focusing on different linguistic elements improves pronunciation in Spanish as a second language (L2) for college-age students who speak English as a first language (L1). We are developing a visualization platform that students will utilize in order to learn Spanish through the use of various visualization techniques. We aim to quantify the short-term and long-term effects on pronunciation students experience while using the tool by measuring motivation, pronunciation, and perception through an organized study. This study recruits college-age students to use this tool over a period of six months with planned check-ins.

Research Questions

As we began this project, we posed certain research efforts around the following questions:

What are learners’ motivations for learning a language and what kinds of effects does that have on pronunciation gains?
What is the role of perception in learning pronunciation?
How does visualization focusing on specific classes of sounds (e.g. mid vowels and voicing of consonants) affect pronunciation gains?
What effect does the visualization have on perception of sounds?
How does engagement differ between the different visualizations?

Literature Review

My first role in this project was to assist with a literature review of areas relevant to this project. I found and categorized relevant research in various fields while focusing on the roles types of feedback had on language learning and the visualization methods currently used.

In terms of feedback, one of the main questions I sought to answer through this process was whether we should use delayed or immediate feedback for our platform and if students should receive implicit or explicit feedback. Further, I focused on researching the current methodology of visualization in existing language learning platforms and the justification behind them. Throughout this process, we prioritized understanding the impact of pronunciation feedback and instruction and how we will measure pronunciation gains.

At the end of this literature review, we finalized a base case and three visualization methods to be used in our platform: international phonetic vowel chart, voice onset time, and a mirror visual.

Platform Prototyping & Justification

In this portion of the project, Charlotte and I developed low-fidelity prototypes of the visual platform utilizing Adobe XD. As I include images of the prototypes of each visualization, I will explain each method as developed by Charlotte.

Vowel visualization, consonant visualization, a video camera condition, and the control/base case are the four experimental circumstances in our study. The participant hears a Spanish minimum pair delivered by a native (L1) speaker in each condition. The minimum pair's words can all be heard as many times as the participants like. Participants must eventually record each word at least once. This could happen either before or after hearing the L1 speaker. Participants can go on to the next pair of words and repeat the listening and speaking process after recording each word at least once.

There are no visualizations in the control condition. Participants are given the option to listen to a simple pair while recording.

Base Case:
‍

We map the participants' vowel positions on the International Phonetic Vowel Chart for vowel visualization. Along with the participant's recordings, the vowels in the target minimum pair are shown on the vowel chart.

Vowel Visualization:

In the second experimental setting, we use Voice Onset Time (VOT), a consonant visualization. The duration of time between the release of a stop consonant and the vibrating of the vocal cords for the vowel that follows is known as the voice onset time. The participant's VOTs are plotted above the L1 speaker's VOT diagram.

Consonant Visualization:

The last test scenario is a video camera. A video of the L1 speaker's face is shown alongside the voice recording. Before attempting it on their own, participants have the opportunity to see and hear the L1 speaker. The L1 speaker's recordings are situated between a live webcam feed of the participants' faces, which only starts recording when they push the record button. Participants may listen to and watch an attempt after it has been recorded.

Mirror Case Visualization:

In addition to the low-fidelity mockups, I utilized Figma to create high-fidelity prototypes of the visualization platform. As the development of the platform is not yet finalized, here I will include one portion I prototyped. This interactive prototype shows how a user will log in and interact with the base case visualization with the functionality to favorite certain recordings and delete others.

‍

[data-wf-bgvideo-fallback-img] { display: none; } @media (prefers-reduced-motion: reduce) { [data-wf-bgvideo-fallback-img] { position: absolute; z-index: -100; display: inline-block; height: 100%; width: 100%; object-fit: cover; } }

Vocabulary & Lesson Plan Development

As we finalized the structure of the visualization platform, I developed the initial lesson plan for students to follow. We finalized that the platform would allow students to complete five categories of vocabulary with six vocabulary words in each. Students will also receive extra vocabulary to learn after completing the initial lessons.

To develop the lesson plan, I began by exploring materials such as current beginner’s Spanish lesson plans and guides to learning Spanish as an English speaker. By cross-referencing the Oxford Spanish Wordpack and A Frequency Dictionary of Spanish: Core Vocabulary for Learners, I finalized lesson plans that would be conducive to a student’s learning. By using these texts to discover the most frequently used Spanish words and key vocabulary for first-time Spanish learners, I developed the lesson plan’s main categories and vocabulary. For example, the following is the vocabulary for a student’s first week using our platform.

Structure of Study

To measure the impact of this tool, this project includes a study in which recruited participants utilize the visualization platform in order to learn Spanish. These participants are college students between the ages of 18 and 24 whose first language is English and who have minimal experience with Spanish.

The pilot research study consists of one in-lab session. The first in-lab session of the longitudinal study is identical, but it also includes a description of how the rest of the investigation will proceed. After then, there are three further in-lab sessions: one after one month, one after two months, and one after six months. The study is structured as follows:
‍

At each lab check-in, we assess motivation, pronunciation, and perception in order to gauge learning outcomes. It is crucial to observe both the short-term and long-term effects of using the tool.

We aim to quantify the short-term effect in order to compare initial reactions and potentially explain why different language studies have certain outcomes. Measuring the short-term effects is significant because, currently, other language research only reports short-term effects. Additionally, measuring long-term effects is important in language acquisition as pronunciation naturally improves the more a learner interacts with a language; therefore, we need to determine if pronunciation improvements exceed what happens naturally.

Finally, once the interaction has ended, we want to investigate how long-term engagement with the tool has an impact on perception and pronunciation.

Key Takeaways

In this project, I had the opportunity to work directly with the Social Spaces lab as well as the CS Stars program. This experience has allowed me to explore research while serving as a leader among undergraduate students pursuing research. I have grown as a researcher due to my direct involvement and contributions to this project.

Through this project, I’ve also had the chance to present my research for the first time at UIUC’s annual Trick or Research event. This allowed me to experience presenting research as well as learning about other research currently conducted in HCI and other fields.