Enhanced Transformer architecture for in-context learning of dynamical systems

Matteo Rufolo, Dario Piga, Gabriele Maroni, Marco Forgione
10/4/2024
cs.LGcs.AIeess.SY

No official code found. Best implementation: mattrufolo/sysid-prob-transformer (2 stars, last commit 2025-03-07).

Abstract

Recently introduced by some of the authors, the in-context identification paradigm aims at estimating, offline and based on synthetic data, a meta-model that describes the behavior of a whole class of systems. Once trained, this meta-model is fed with an observed input/output sequence (context) generated by a real system to predict its behavior in a zero-shot learning fashion. In this paper, we enhance the original meta-modeling framework through three key innovations: by formulating the learning task within a probabilistic framework; by managing non-contiguous context and query windows; and by adopting recurrent patching to effectively handle long context sequences. The efficacy of these modifications is demonstrated through a numerical example focusing on the Wiener-Hammerstein system class, highlighting the model's enhanced performance and scalability.

View on arXivView PDF

Code Implementations(1)

mattrufolo/sysid-prob-transformerCommunity100%
20Shell, PythonSep 16, 20241 years ago

Similar Papers

Probing In-Context Learning: Impact of Task Complexity and Model Architecture on Generalization and Efficiency

83% match1 repo12025

Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence

81% match2026

Meta-in-context learning in large language models

81% match1 repo62026

In-Context In-Context Learning with Transformer Neural Processes

81% match2024

Transformers as Algorithms: Generalization and Stability in In-context Learning

81% match1 repo82023

In-Context Learning Dynamics with Random Binary Sequences

81% match2026

Cite this paper

@article{rufolo2024enhanced,
  title  = {Enhanced Transformer architecture for in-context learning of dynamical systems},
  author = {Matteo Rufolo and Dario Piga and Gabriele Maroni and Marco Forgione},
  year   = {2024},
  eprint = {2410.03291},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.03291v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.03291.svg)](https://codeofpaper.com/paper/2410.03291)

Discussion