SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research

Dimosthenis Antypas, Asahi Ushio, Francesco Barbieri, Leonardo Neves, Kiamehr Rezaee +3 more
10/23/2023
cs.CL

No confident code implementation found for this paper.

Abstract

Despite its relevance, the maturity of NLP for social media pales in comparison with general-purpose models, metrics and benchmarks. This fragmented landscape makes it hard for the community to know, for instance, given a task, which is the best performing model and how it compares with others. To alleviate this issue, we introduce a unified benchmark for NLP evaluation in social media, SuperTweetEval, which includes a heterogeneous set of tasks and datasets combined, adapted and constructed from scratch. We benchmarked the performance of a wide range of models on SuperTweetEval and our results suggest that, despite the recent advances in language modelling, social media remains challenging.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

SMILE: Evaluation and Domain Adaptation for Social Media Language Understanding

88% match2023

Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark

87% match1 repo242023

L3Cube-MahaSocialNER: A Social Media based Marathi NER Dataset and BERT models

86% match1 repo1602023

NorBench -- A Benchmark for Norwegian Language Models

85% match1 repo142023

BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXams

85% match1 repo202023

Catwalk: A Unified Language Model Evaluation Framework for Many Datasets

84% match2 repos2.8k2023

Cite this paper

@article{antypas2023supertweeteval,
  title  = {SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research},
  author = {Dimosthenis Antypas and Asahi Ushio and Francesco Barbieri and Leonardo Neves and Kiamehr Rezaee and Luis Espinosa-Anke and Jiaxin Pei and Jose Camacho-Collados},
  year   = {2023},
  eprint = {2310.14757},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.14757v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.14757.svg)](https://codeofpaper.com/paper/2310.14757)

Discussion