Gradient Ascent Post-training Enhances Language Model Generalization

Dongkeun Yoon, Joel Jang, Sungdong Kim, Minjoon Seo
6/12/2023
cs.CLcs.AI

No official code found. Best implementation: kaistAI/GAP (29 stars, Apache-2.0, last commit 2024-09-12).

Abstract

In this work, we empirically show that updating pretrained LMs (350M, 1.3B, 2.7B) with just a few steps of Gradient Ascent Post-training (GAP) on random, unlabeled text corpora enhances its zero-shot generalization capabilities across diverse NLP tasks. Specifically, we show that GAP can allow LMs to become comparable to 2-3x times larger LMs across 12 different NLP tasks. We also show that applying GAP on out-of-distribution corpora leads to the most reliable performance improvements. Our findings indicate that GAP can be a promising method for improving the generalization capability of LMs without any task-specific fine-tuning.

View on arXivView PDF

Code Implementations(1)

kaistAI/GAPCommunity100%
Apache-2.0

[ACL 2023] Gradient Ascent Post-training Enhances Language Model Generalization

294Shell, PythonMay 23, 20232 years agoApache-2.0

Similar Papers

Source Prompt: Coordinated Pre-training of Language Models on Diverse Corpora from Multiple Sources

86% match2023

Improving the Reusability of Pre-trained Language Models in Real-world Applications

85% match2023

Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner

85% match1 repo762023

Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner

85% match2 repos1022026

Class-Aware Prototype Learning with Negative Contrast for Test-Time Adaptation of Vision-Language Models

84% match2025

Generation-driven Contrastive Self-training for Zero-shot Text Classification with Instruction-following LLM

84% match1 repo122023

Cite this paper

@article{yoon2023gradient,
  title  = {Gradient Ascent Post-training Enhances Language Model Generalization},
  author = {Dongkeun Yoon and Joel Jang and Sungdong Kim and Minjoon Seo},
  year   = {2023},
  eprint = {2306.07052},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.07052v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.07052.svg)](https://codeofpaper.com/paper/2306.07052)

Discussion