Developments

All the Latest App Updates in One Place

Milestone I

November 8, 2023

Explore Ideas

In the first milestone, we brainstormed a bunch of ideas.

1) Fitness - novel fitness game

2) Virtual Musical Instruments

3) Cultural Heritage Preservation - Virtual Environment / Nature sound

4) Driving simulation

5) Accessibility Project using OCR + text-to-audio

6) AR Language Learning application using semantic segmentation and OCR

Progress

In this week, we first implement the real-time image segmentation in Unity. The video shows how this works in a mobile device. The detected object is highlighted in the rectangle area, with its label name and corresponding confidence.

We evaluate our progress based on the following criteria:

1. Technical Sophistications & Novelty: implement a real-time image segmentation model embedded in an AR phone App.

2. Social Impact:

enhance the learning experience by providing real-time object labels/names based on semantic segmentation

3. XR emphasis:

object detections and label display are purely in AR spaces

4. Finanacial Sustainability:

we design a basic charge plan based on premium subscriptions for more features

Features of AR Language Learning App

We decide to make AR Language Learning application with the following potential features:

Image semantic segmentation - the App will detect surrounding objects with boundary box and text meaning, which will be translated into different languages.
Text Recognition - the App will recognize texts and translate into different languages
Enable user to learn the name of the objects by providing spell and pronunciation.

Looking Forward

In our next step, we are going to implement the translator and import a dictionary for different languages.

Milestone II

November 15, 2023

Progress

In this week, we have achieved the following progress:

1. Implement a translator that can translate the detected object labels into other languages (Spanish in the demonstrated video); the translated labels are shown above the boundary boxes

2. Design more intuitive UI buttons

Based on the playtest reflect:

To improve the accuracy of detections, we adjust the threshold of confidence level from 0.25 to 0.5 and filter our any detections with confidence level less than 0.5

Evaluation

We evaluate our progress based on the following criteria:

1. Technical Sophistications & Novelty: implement a language translator embedded in our App to provide translation on the names of real-time detected objects

2. Social Impact:

enable language learners to interact with their environment in a new and engaging way, learning the names of everyday objects in different languages

3. XR emphasis:

object name translations are displayed in the AR spaces

4. Finanacial Sustainability:

we updated our charge plan based on the number of languages provided

Looking Forward

Current Challenges:

1. Invalid characters after translation

As shown in the video, we find some of the characters (mainly non-ASCII characters) are not supported by Unity. We regard solving this problem as our highest priority.

Next Steps:

1. Displaying non-ASCII characters based on Unity's support on unicode

2. Providing more options for different languages

3. Implement more interactive features for language learning (e.g. pronunciations)

Alpha

November 22, 2023

Progress

In this week, we have achieved the following progress:

1. We solve the invalid characters after translation which means that we can show all translated languages' characters correctly

2. Design a more intuitive UI which acts as a pop-up window that help support the multi-language options and further translation

3. Fix those visual bugs occurred in the second playtest like incorrect detection & remove the showing of confidence values

Based on the playtest reflect:

To improve the interactivity, we make it possible that people can touch the bouding box of the item that they want to translate/know directly with showing the meaning in different languages.

Evaluation

We evaluate our progress based on the following criteria:

1. Technical Sophistications & Novelty:

offer more interactivity to users with pop-up windows, which appear when people directly touch objects on the screen

2. Social Impact:

enable language learners to interact with their environment in a more user-friendly way, learning the names of everyday objects in different languages

3. XR emphasis:

object name translations are displayed in the AR spaces and people can interact with those detected objects via touching

4. Finanacial Sustainability:

we updated our charge plan to specify the detailed language options that we can provide with

Looking Forward

Current Challenges:

1. Realization of Pronunciation & detailed meaning

As shown in the video, we find that we also need to support the pronunciationa and detialed meaning of those words that people want to learn on the pop-up window, which ask us to dig deeper with the Google Cloud Translation API.

Next Steps:

1. Improve the UI

2. Add the pronunciation of those words in different language via text-to-speech

3. Add the detailed meaning and explanation for those words in English through Cloud Translation API

Gold

Dec 4, 2023

Progress

In this week, we have achieved the following progress:

1. Add prononciation for translated words that helps users to learn the correct pronunciation and further improve their language speaking skills.

2. The users can now review what they learn with the app - add a mini shooting games where users shoot at randomly generated 3D models of daily objects. Each bullet bears with a translated name of the object typed in by the user. The user succeeds if the bullet hits the right object.

Based on the playtest reflect:

Add more language options into the app and move the language options to the main page of the app.

Evaluation

We evaluate our progress based on the following criteria:

1. Technical Sophistications & Novelty:

offer better learning experience by enabling pronunciation of translated words and adding mini game for knowledge review

2. Social Impact:

provide a more completed learning experience - users can now practice their speaking skill following the pronunciation and review their knowledge in the shooting mini game.

3. XR emphasis:

the mini shooting game is displayed in AR spaces

4. Finanacial Sustainability:

update the charging plan of the app including more translation per day and detailed language options