Bilingual AI voice assistant:
an inclusive conversational experience for Portuguese and Spanish

Client

Personal project

Industry

AI Assistant

Service

Conversational design and UX writing

Period

2022

Context and problem

While using voice assistants in Brazil, I identified a recurring challenge: commands containing Spanish names and expressions failed, even when requests in English were recognized correctly.

The conceptual research identified four critical issues:

• Language mixing: difficulty interpreting sentences that alternate between Portuguese and Spanish.

• Accents: recognition errors caused by pronunciations influenced by Spanish.

• False cognates: incorrect interpretations of words that look or sound similar across the two languages.

• Insufficient recovery: repeated error messages without alternatives for continuing.

These errors shift the responsibility for technological limitations onto the user, causing frustration and loss of trust.

Objective

Design a conversational flow that reduces interpretation errors, preserves user intent, and provides clear paths for recovering the interaction.

My role and process

I led the project from research through conceptual prototyping, including:

• Research into speech recognition in Brazilian Portuguese and Latin American Spanish.

• Investigation of technological limitations existing in 2022.

• Definition of bilingual interaction principles.

• Development of the decision tree and dialogue prototyping.

• Heuristic evaluation of voice interfaces, with adaptations of Nielsen's heuristics.

I used proto-personas to represent different profiles of Spanish-speaking users who interact with assistants configured for Portuguese.

Bilingual PT-ES proto-personas

Raul Gonzáles

Age: 25 · Education: university degree.
"Tengo que repetir mis palabras. ​​El asistente simplemente no entiende mi acento, ya que está configurado en portugués”.

Background: A Spanish-speaking user who recently moved to Brazil. Uses voice assistants configured for Portuguese to become more familiar with the language. Uses work time to practice conversations with Brazilian colleagues.

Frustration: Still does not have a strong command of Portuguese and needs to improve fluency. Digitally savvy and comfortable with technology.

Mariela Aguirre

Age: 40 · Education: secondary education.

“No se entendió porque tengo acento, por lo que no completé la tarea indicada”.

Background: A Spanish-speaking user who has lived in Brazil for several years but still struggles with the nasalized sounds characteristic of Portuguese. Married to a Brazilian and has a child. Lives in a bilingual Spanish-Portuguese household because she wants her child to grow up connected to both parents' cultures.

Frustration: Struggles with Portuguese phonemes that do not exist in Spanish. Uses technology for everyday tasks. Is not interested in the latest market innovations, but rather in products that can help solve her problems.

Four principles for a more inclusive interaction

1. Detect language mixing: identify when a sentence combines Portuguese and Spanish and ask which language the user prefers to continue in.

2. Clarify ambiguities: confirm the meaning of ambiguous terms before executing an action, avoiding incorrect interpretations.

3. Navigate based on intent: when recognition repeatedly fails, offer keywords and short alternatives to help complete the task.

4. Recover the interaction without blaming the user: replace repetitive error messages with supportive responses that acknowledge the failure and explain how to proceed.

Flowchart illustrating the experience's decision tree.

Conversational flow example

Failure example

Success example: When the conversation adapts to the user.

Failure exemple: when recognition compromises user intent.

Success example

Since the project was conceived, advances in AI have expanded the possibilities for implementing more context-aware conversational flows. Opportunities include:

• Contextual speech recognition: reassessing ambiguous segments based on the conversation's context.

• Accent and hybrid pronunciation recognition: improving the interpretation of phonetic variations.

• Code-switching detection: identifying language changes within a sentence.

• Intent classification: understanding the purpose of a command even when some words are not recognized correctly.

• Personalization: adapting recognition to the user's vocabulary and preferences.

• Adaptive tone: adjusting responses based on signals of difficulty or frustration, where technically feasible.

These capabilities open up new possibilities for the proposal, but their application would depend on technical validation and user testing.

Technology evolution: 2022–2026

Design for intent, not just words: a voice interface must account for context, linguistic variations, and ambiguity to understand what the user wants to accomplish.

Recovery is part of the experience: when a failure occurs, offering clear alternatives is more useful than repeating the same error message.

Linguistic inclusion is a design decision: accents and language switching should not be treated merely as noise. The experience must account for linguistic diversity from the outset.

Design and technology must work together: improving a conversational experience requires combining UX decisions, language understanding, and technical feasibility, with particular attention to error scenarios.

Conclusions and key learnings

Scope

Conceptual project, with no implementation or quantitative user-testing results.