Cross-lingual Word Segmentation and Morpheme Segmentation as Sequence Labelling

Yan Shao
9/12/2017
cs.CL

Official code: yanshao9798/segmenter (35 stars, Apache-2.0, last commit 2019-04-03).

Abstract

This paper presents our segmentation system developed for the MLP 2017 shared tasks on cross-lingual word segmentation and morpheme segmentation. We model both word and morpheme segmentation as character-level sequence labelling tasks. The prevalent bidirectional recurrent neural network with conditional random fields as the output interface is adapted as the baseline system, which is further improved via ensemble decoding. Our universal system is applied to and extensively evaluated on all the official data sets without any language-specific adjustment. The official evaluation results indicate that the proposed model achieves outstanding accuracies both for word and morpheme segmentation on all the languages in various types when compared to the other participating systems.

View on arXivView PDF

Code Implementations(1)

yanshao9798/segmenterOfficial100%
Apache-2.0

Universal segmenter based on the Universal Dependency framework, written by Y. Shao, Uppsala University

3515PythonJun 21, 20177 years agoApache-2.0

Similar Papers

Using Contextual Information for Sentence-level Morpheme Segmentation

83% match2024

Learning Beyond Limits: Multitask Learning and Synthetic Data for Low-Resource Canonical Morpheme Segmentation

81% match2025

TAMS: Translation-Assisted Morphological Segmentation

81% match2024

Massively Multilingual Joint Segmentation and Glossing

80% match2026

A Character-Level Neural Approach to Sinhala Sandhi Splitting

80% match2026

A Truly Joint Neural Architecture for Segmentation and Parsing

80% match2024

Cite this paper

@article{shao2017crosslingual,
  title  = {Cross-lingual Word Segmentation and Morpheme Segmentation as Sequence Labelling},
  author = {Yan Shao},
  year   = {2017},
  eprint = {1709.03756},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1709.03756}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1709.03756.svg)](https://codeofpaper.com/paper/1709.03756)

Discussion