Improving the training and evaluation efficiency of recurrent neural network language modelsReport as inadecuate


Improving the training and evaluation efficiency of recurrent neural network language models


Improving the training and evaluation efficiency of recurrent neural network language models - Download this document for free, or read online. Document in PDF available to download.

Publication Date: 2015-04-22

Journal Title: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Pages: 5401-5405

Language: English

Type: Article

Metadata: Show full item record

Citation: Chen, X., Liu, X., Gales, M. J. F., & Woodland, P. C. (2015). Improving the training and evaluation efficiency of recurrent neural network language models. 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 5401-5405.

Description: This is the author accepted manuscript. The final version is available from IEEE via http://dx.doi.org/10.1109/ICASSP.2015.7179003

Abstract: Recurrent neural network language models (RNNLMs) are becoming increasingly popular for speech recognition. Previously, we have shown that RNNLMs with a full (non-classed) output layer (F-RNNLMs) can be trained efficiently using a GPU giving a large reduction in training time over conventional class-based models (C-RNNLMs) on a standard CPU. However, since test-time RNNLM evaluation is often performed entirely on a CPU, standard F-RNNLMs are inefficient since the entire output layer needs to be calculated for normalisation. In this paper, it is demonstrated that C-RNNLMs can be efficiently trained on a GPU, using our spliced sentence bunch technique which allows good CPU test-time performance (42x speedup over F-RNNLM). Furthermore, the performance of different classing approaches is investigated. We also examine the use of variance regularisation of the softmax denominator for F-RNNLMs and show that it allows F-RNNLMs to be efficiently used in test (56x speedup on CPU). Finally the use of two GPUs for F-RNNLM training using pipelining is described and shown to give a reduction in training time over a single GPU by a factor of 1.6.

Keywords: language models, recurrent neural network, GPU, speech recognition

Sponsorship: Xie Chen is supported by Toshiba Research Europe Ltd, Cambridge Research Lab. The research leading to these results was also supported by EPSRC grant EP/I031022/1 (Natural Speech Technology) and DARPA under the Broad Operational Language Translation (BOLT) and RATS programs. The paper does not necessarily reflect the position or the policy of US Government and no official endorsement should be inferred.

Identifiers:

This record's URL: http://www.repository.cam.ac.uk/handle/1810/247435http://dx.doi.org/10.1109/ICASSP.2015.7179003





Author: Chen, X.Liu, X.Gales, M. J. F.Woodland, P. C.

Source: https://www.repository.cam.ac.uk/handle/1810/247435



DOWNLOAD PDF




Related documents