Transformer-Encoder Trees for Efficient Multilingual Machine Translation and Speech Translation

Technical Deep Dive: Transformer-Encoder Trees for Efficient Multilingual Machine Translation and Speech Translation

Overview

This work introduces Transformer Encoder Tree (TET), a novel hierarchical, non-autoregressive encoder-only architecture for efficient multilingual machine translation and speech translation. TET capitalizes on linguistic similarity between target languages to improve accuracy, especially for low-resource languages, while significantly reducing computational redundancy and enabling parallel decoding.

Problem & Context

• Multilingual translation suffers from computational redundancy when translating into multiple languages independently
• Low-resource languages can struggle with poor translation quality
• Autoregressive encoder-decoder models offer high accuracy but suffer from high latency and limited parallelism
• Non-autoregressive models can speed up translation but often lag behind autoregressive models in quality

Methodology

• TET is a hierarchical model that mimics the language tree structure to translate from a single source language into multiple targets
• Each node in the tree is a Transformer encoder layer, with shared layers at the top and branching out to language-specific layers at the bottom
• This allows linguistically similar languages to share intermediate representations, improving accuracy for low-resource languages and enabling parallel generation of all target languages
• TET is trained using Connectionist Temporal Classification (CTC) loss, which enables simultaneous generation of all target language tokens

Data & Experimental Setup

• Evaluated on two datasets: Multi30K (9 languages) and Tatoeba (11 languages)
• Compared to baselines:
  • TET-Rnd: TET with randomly assigned languages
  • TEnc/lang: One Transformer encoder per language
  • TEnc/all: Single Transformer encoder for all languages
• For speech translation, used Wav2Vec2 and Whisper as ASR components

Results

Machine Translation:

• On Multi30K, TET achieves translation quality within 1.6 BLEU (-3.7% relative) of the best baseline, while reducing total parameters by 66% and computation by 60%
• On Tatoeba, TET outperforms the one-model-for-all baseline by 3.5 BLEU (+19%), while being more computationally efficient

Speech Translation:

• Combining TET with the non-autoregressive Wav2Vec2 ASR model achieves comparable translation quality to the autoregressive Whisper model, but is 7-14x faster

Interpretation

• Leveraging linguistic similarity via the tree structure enables TET to:
  • Improve accuracy for low-resource languages
  • Reduce computational redundancy compared to independently translating each language
  • Enable parallel generation of all target languages
• The tree topology itself is important, with the linguistically-motivated TET outperforming randomly assigned tree structures

Limitations

• Evaluation focused on a specific set of language families and controlled datasets
• Speech translation experiments used synthesized speech, not real-world data
• Highly autoregressive models like T5 could still outperform TET in translation quality

What Comes Next

• Evaluate TET on larger, more diverse machine translation and speech translation datasets
• Explore integrating TET with other non-autoregressive or CTC-based translation techniques to further boost performance
• Investigate automated methods for optimizing the tree topology