LLMs' morphological analyses of complex FST-generated Finnish words

Anssi Moisio, Mathias Creutz, Mikko Kurimo
7/11/2024
cs.CL

No official code found. Best implementation: aalto-speech/llm-morph-tests (0 stars, last commit 2025-01-14).

Abstract

Rule-based language processing systems have been overshadowed by neural systems in terms of utility, but it remains unclear whether neural NLP systems, in practice, learn the grammar rules that humans use. This work aims to shed light on the issue by evaluating state-of-the-art LLMs in a task of morphological analysis of complex Finnish noun forms. We generate the forms using an FST tool, and they are unlikely to have occurred in the training sets of the LLMs, therefore requiring morphological generalisation capacity. We find that GPT-4-turbo has some difficulties in the task while GPT-3.5-turbo struggles and smaller models Llama2-70B and Poro-34B fail nearly completely.

View on arXivView PDF

Code Implementations(1)

aalto-speech/llm-morph-testsCommunity100%

A test suite to assess the knowledge of morphology in LLMs

00Shell, PythonFeb 2, 20241 years ago

Similar Papers

Evaluating Morphological Compositional Generalization in Large Language Models

84% match2 repos602024

Recent advancements in computational morphology : A comprehensive survey

80% match2024

Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs

79% match2026

Neural Morphological Tagging for Nguni Languages

78% match2025

Hybrid Neural-LLM Pipeline for Morphological Glossing in Endangered Language Documentation: A Case Study of Jungar Tuvan

78% match2026

Analyzing Finnish Inflectional Classes through Discriminative Lexicon and Deep Learning Models

78% match2025

Cite this paper

@article{moisio2024llms,
  title  = {LLMs' morphological analyses of complex FST-generated Finnish words},
  author = {Anssi Moisio and Mathias Creutz and Mikko Kurimo},
  year   = {2024},
  eprint = {2407.08269},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.08269v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.08269.svg)](https://codeofpaper.com/paper/2407.08269)

Discussion