IMPROVING HYBRID TRANSLATION

TOOL

FULL-TEXT SEARCH ENGINE APPROACH

Lianet Sepulveda, Alexander Raginsky

Pangeanic

▪ Improving translation memory matching

▪ Natural Language Processing (NLP)

▪ Full-text search engine

▪ TM database

Agenda

Translation Memory (TM)

▪ Pre-translations stored in a database and offered as suggestions

▪ Implemented matching algorithm to propose a relevant translations

▪ exact match and fuzzy match

▪ segments similarities based on characters or tokens

PLN to improve matching algorithm

Approach

➢ Statistical Machine Translation (SMT)

➢ Computer-Aided Translation (CAT) environment

Run maintenance

● Search and replace

● Update TM entries

● Imports & Export entries

Translation

Memory

Improving TM entries

ElasticTM

Full-text search engine

+

NLP techniques

Improving TM Matching

perfect match by substitution

fuzzy match

{ “source_TM” : “I have 3 cats”, “target_TM” : “Yo tengo 3 gatos”, “score” : “80%” }

{ “source_TM” : “I have <number> cats”, “target_TM” : “Yo tengo <number> gatos”, “score” : “100%” }

Original TM

{ “input_source”: “I have 2 cats”, “output_target”: “ ” }

TM after preprocessing

● URLs

● Emails

● Dates

● Units

Improving TM Matching

fuzzy match

{ “source_TM” : “I have a cat and I am very happy”, “target_TM” : “Yo tengo un gato y estoy muy feliz”, “score” : “44%” }

{ “target_TM” : “Yo tengo un gato y estoy muy feliz”, “source_TM” : “I have a cat”, “target_TM” : “Yo tengo un gato”, “source_TM” : “I am very happy”, “target_TM” : “Estoy muy feliz”, “score” : “100%” }

Original TM

{ “input_source”: “I have a cat”, “output_target”: “ ” }

TM after preprocessing

perfect match by substitution

Improving TM Matching

▪ Several language → Maximise the reuse of existing human translation

▪ Linguistic feature → improving fuzzy matching

▪ string transformation

▪ segmentation rules

▪ pos tagger

▪ tokenizer

EN

ES

PT

JA

.

.

.

FR

EN

ES

PT

JA

.

.

.

FR

Improving TM Matching

Linguistic approach to improve match

● Segment the text by sentence

○ Using delimiters like . ? ! , - :

○ Limited the total of words

● Intra-sentence segmentation

○ Using conjunctions, adverbs, determiners,

pronouns

○ Others approaches

● Replace segments

○ Numbers, dates, proper nouns and identifiers,

URLs, e-mail address, punctuation marks,

acronyms, variables.

● POS pattern string

● Named entity recognition

ElasticTM

TMX

files

source text

(Puscasu, 2004; Eriksson and

Myhrman, 2010; Orasan, 2000)

▪ TM database built from TMX files

▪ Based on the state-of-the-art full-text search engine

▪ Fast indexing, search and retrieval

▪ Supports advanced text retrieval techniques (fuzzy match,

regular expressions)

▪ Easily scalable

▪ Role-based security

ElasticTM

▪ PangeaMT

▪ As a preprocessing step before Moses SMT

▪ PangeaCrawler

▪ Automatic website translation

▪ Plugins for CAT tools

▪ As an auxiliary tool for human translators

ElasticTM - Intended Usage

ElasticTM - Design

EN ES FR ... NL

Search Engine

EN

<->

ES

FR

<->

ES

FR

<->

NL

...

Map DB

ElasticTM

ElasticTM - Design (cont’d)

▪ Monolingual índices

▪ Memory-effective

▪ Implicit transitive language pairs

▪ Bilingual mappings

▪ Fast bidirectional id <-> id mapping

▪ Role-based security system

▪ Admin, project admin, user etc.

▪ Considered Lucene-based search engines:

▪ Solr and ElasticSearch

▪ Mature open source projects

▪ Have similar capabilities & performance

▪ ElasticSearch was picked mainly because of:

▪ Out-of-the-box scalability

▪ Powerful Query DSL (query language)

▪ Role-based security (via plugin)

ElasticTM - Search Engine

▪ Mapping source language segments to a target language

▪ Bidirectional map (id to id)

▪ Supports quick bulk incremental updates

ElasticTM - Map

▪ NoSQL key-value databases

▪ MongoDB

▪ CouchDB

▪ Redis

▪ ElasticSearch

▪ … many others …

▪ SQL databases

▪ MySQL

▪ PostgreSQL

ElasticTM - Map - Alternatives

Lack of upsert support for bulk updates

Handling duplicate entries

Scalability

ElasticTM - Map - Benchmarks

* The lower, the better

Time, s

Memory, MB

ElasticTM - Map - Benchmarks

ElasticSearch MongoDB CouchDB Redis

Add (47K) 83s 432s 67s 458s

Add (440K)

858s 6112s 644s 621s

Query (10K) 51s 187s 458s 72s

Query (440K) 1400s 6451 19647 1210s

Memory 252M 549M 771M 148M

ElasticTM - Scaling

ghg

Cluster

EN ...

1)

EN-ES1

EN-ES2

ghg

Cluster

EN1

ES1

ES2 EN2 ... 2)

EN-ES1

EN-ES2

ES

▪ Status

▪ Benchmarked alternatives

▪ Implemented and tested prototype

▪ Analyzed feasibility of linguistic methods

▪ Plans

▪ Build & scale ElasticTM to cover all available TMs in Pangeanic

▪ Implement plugins for CAT tools

▪ Develop linguistic processing for major language pairs

Status & Plans