Smartphone users are beyond two billion worldwide. Heavy users of the texting application rely on input prediction to reduce typing effort. In languages based on the Roman alphabet, many techniques are available. However, Japanese text is based on multiple character sets such as Kanji (Chinese-like word symbols), Hiragana and Katakana syllable sets. For its time/labor intensive input, next word prediction is crucial. It is still an open challenge. To tackle this, a hybrid language model is proposed. It integrates a Recurrent Neural Network (RNN) with an n-gram model. RNNs are powerful models for learning long sequences for next word prediction. N-gram models are best at current word completion. Our RNN language model (RNN-LM) predicts the next words. According the “price” of the performance gain paid by a higher time complexity, our model best deploys on a client-server architecture. Heavily-loaded RNN-LM deploys on the server while the n-gram model on the client. Our RNN-LM consists of an input layer equipped with word embedding, an output layer, and hidden layers connected with LSTMs (Long Short-Term Memories). Training is done via BPTT (Back Propagation Through Time). For robust training, BPTT is elaborated by learning rate refinement and gradient norm scaling. To avoid overfitting, the dropout technique is applied except for LSTM. Our novel model is compact (2 LSTMs, 650 units per layer), indeed. Due to synergetic elaboration, it shows 10% lower perplexity than Zaremba's excellent conventional models in our Japanese text prediction experiment. Our model has been incorporated into IME (Input Method Editor) we call Flick. On the Japanese text input experiment, Flick outperforms Mozc (Google Japanese Input) by 16% in time and 34% in the number of keystrokes.

Fast ML-based Next-Word Prediction for Hybrid Languages / Y. Ikegami, S. Tsuruta, A. Kutics, E. Damiani, R. Knauf. - In: INTERNET OF THINGS. - ISSN 2542-6605. - (2024). [Epub ahead of print] [10.1016/j.iot.2024.101064]

Fast ML-based Next-Word Prediction for Hybrid Languages

E. Damiani
Penultimo
;
2024

Abstract

Smartphone users are beyond two billion worldwide. Heavy users of the texting application rely on input prediction to reduce typing effort. In languages based on the Roman alphabet, many techniques are available. However, Japanese text is based on multiple character sets such as Kanji (Chinese-like word symbols), Hiragana and Katakana syllable sets. For its time/labor intensive input, next word prediction is crucial. It is still an open challenge. To tackle this, a hybrid language model is proposed. It integrates a Recurrent Neural Network (RNN) with an n-gram model. RNNs are powerful models for learning long sequences for next word prediction. N-gram models are best at current word completion. Our RNN language model (RNN-LM) predicts the next words. According the “price” of the performance gain paid by a higher time complexity, our model best deploys on a client-server architecture. Heavily-loaded RNN-LM deploys on the server while the n-gram model on the client. Our RNN-LM consists of an input layer equipped with word embedding, an output layer, and hidden layers connected with LSTMs (Long Short-Term Memories). Training is done via BPTT (Back Propagation Through Time). For robust training, BPTT is elaborated by learning rate refinement and gradient norm scaling. To avoid overfitting, the dropout technique is applied except for LSTM. Our novel model is compact (2 LSTMs, 650 units per layer), indeed. Due to synergetic elaboration, it shows 10% lower perplexity than Zaremba's excellent conventional models in our Japanese text prediction experiment. Our model has been incorporated into IME (Input Method Editor) we call Flick. On the Japanese text input experiment, Flick outperforms Mozc (Google Japanese Input) by 16% in time and 34% in the number of keystrokes.
Input Method Editor; word prediction; hybrid language model; client-server model; Recurrent Neural Networks; Back Propagation Through Time
Settore INF/01 - Informatica
   TrustwOrthy model-awaRE Analytics Data platfORm
   TOREADOR
   EUROPEAN COMMISSION
   H2020
   688797
2024
9-gen-2024
Article (author)
File in questo prodotto:
File Dimensione Formato  
1-s2.0-S2542660524000064-main.pdf

accesso aperto

Descrizione: Article
Tipologia: Post-print, accepted manuscript ecc. (versione accettata dall'editore)
Dimensione 1.07 MB
Formato Adobe PDF
1.07 MB Adobe PDF Visualizza/Apri
Pubblicazioni consigliate

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/2434/1024149
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 0
  • ???jsp.display-item.citation.isi??? 0
social impact