Which Features are Learned by CodeBert: An Empirical Study of the BERT-based Source Code Representation Learning

Lan Zhang, Chen Cao, Zhilong Wang, Peng Liu
1/20/2023
cs.CLcs.CRcs.LG

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

The Bidirectional Encoder Representations from Transformers (BERT) were proposed in the natural language process (NLP) and shows promising results. Recently researchers applied the BERT to source-code representation learning and reported some good news on several downstream tasks. However, in this paper, we illustrated that current methods cannot effectively understand the logic of source codes. The representation of source code heavily relies on the programmer-defined variable and function names. We design and implement a set of experiments to demonstrate our conjecture and provide some insights for future works.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

CodeSAM: Source Code Representation Learning by Infusing Self-Attention with Multi-Code-View Graphs

82% match2024

Seamlessly Integrating Tree-Based Positional Embeddings into Transformer Models for Source Code Representation

82% match2025

Enhancing Source Code Classification Effectiveness via Prompt Learning Incorporating Knowledge Features

82% match1 repo42024

INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers

81% match2023

What does Transformer learn about source code?

81% match2022

All Roads Lead to Rome? Exploring the Invariance of Transformers' Representations

80% match1 repo72023

Cite this paper

@article{zhang2023which,
  title  = {Which Features are Learned by CodeBert: An Empirical Study of the BERT-based Source Code Representation Learning},
  author = {Lan Zhang and Chen Cao and Zhilong Wang and Peng Liu},
  year   = {2023},
  eprint = {2301.08427},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2301.08427v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2301.08427.svg)](https://codeofpaper.com/paper/2301.08427)

Discussion