DocBERT: BERT for Document Classification

Ashutosh Adhikari, Achyudh Ram, Raphael Tang, Jimmy Lin
4/17/2019
cs.CL

Official code: castorini/hedwig (597 stars, Apache-2.0, PyTorch, last commit 2023-07-21).

Abstract

We present, to our knowledge, the first application of BERT to document classification. A few characteristics of the task might lead one to think that BERT is not the most appropriate model: syntactic structures matter less for content categories, documents can often be longer than typical BERT input, and documents often have multiple labels. Nevertheless, we show that a straightforward classification model using BERT is able to achieve the state of the art across four popular datasets. To address the computational expense associated with BERT inference, we distill knowledge from BERT-large to small bidirectional LSTMs, reaching BERT-base parity on multiple datasets using 30x fewer parameters. The primary contribution of our paper is improved baselines that can provide the foundation for future work.

View on arXivView PDF

Code Implementations(1)

castorini/hedwigOfficial100%
PyTorchApache-2.0

PyTorch deep learning models for document classification

597128PythonMar 10, 20193 years agoApache-2.0
deep-learningdocument-classificationpytorch

Similar Papers

HDLTex: Hierarchical Deep Learning for Text Classification

84% match1 repo2772017

Rationale-Augmented Convolutional Neural Networks for Text Classification

84% match2 repos492016

Learning Section Weights for Multi-Label Document Classification

83% match2023

Beyond Document Page Classification: Design, Datasets, and Challenges

83% match2023

Summarization-based Data Augmentation for Document Classification

83% match1 repo2023

WordVIS: A Color Worth A Thousand Words

82% match2024

Cite this paper

@article{adhikari2019docbert,
  title  = {DocBERT: BERT for Document Classification},
  author = {Ashutosh Adhikari and Achyudh Ram and Raphael Tang and Jimmy Lin},
  year   = {2019},
  eprint = {1904.08398},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1904.08398}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1904.08398.svg)](https://codeofpaper.com/paper/1904.08398)

Discussion