Automatic word spacing of erroneous sentences in mobile devices with limited hardware resources
作者:
Highlights:
•
摘要
With the rapid evolution of the mobile environment, the demand for natural language applications on mobile devices is increasing. This paper proposes an automatic word spacing system, the first step module of natural language processing (NLP) for many languages with their own word spacing rules, that is designed for mobile devices with limited hardware resources. The proposed system uses two stages. In the first stage, it preliminarily corrects word spacing errors by using a modified hidden Markov model based on character unigrams. In the second stage, the proposed system re-corrects the miscorrected word spaces by using lexical rules based on character bigrams or longer combinations. By using this hybrid method, the proposed system improves the robustness against unknown word patterns, reduces memory usage, and increases accuracy. To evaluate the proposed system in a realistic mobile environment, we constructed a mobile-style colloquial corpus using a simple simulation method. In experiments with a commercial mobile phone, the proposed system showed good performances (a response time of 0.20 s per sentence, a memory usage of 2.04 MB, and an accuracy of 92–95%) in the various evaluation measures.
论文关键词:Automatic word spacing,Modified hidden Markov model,Mobile-style colloquial corpus,Mobile device
论文评审过程:Received 20 May 2009, Revised 3 November 2010, Accepted 4 November 2010, Available online 25 May 2011.
论文官网地址:https://doi.org/10.1016/j.ipm.2010.11.004