Unifying Two Types of Scaling Laws from the Perspective of Conditional Kolmogorov Complexity

Jun Wan
1/12/2025
cs.AI

No confident code implementation found for this paper.

Abstract

In 2020, OpenAI proposed the first type of Scaling Laws, describing the relationships between model loss and the scale of parameters, data, and training computation. In 2024, OpenAI proposed the second type of Scaling Laws, describing the relationship between model inference performance and inference computation. In this paper, we analyze LLMs training and inference processes from the perspective of lossless compression using conditional Kolmogorov complexity, and unify these two types of Scaling Laws. We find that both types of Scaling Laws improve approximation of conditional Kolmogorov complexity by increasing execution steps of Turing machine. The first type of Scaling Laws increases execution steps by increasing number of model parameters. The second type of Scaling Laws increases execution steps by increasing the number of intermediate tokens.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuning

84% match1 repo112026

ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression

83% match2026

TRiMS: Real-Time Tracking of Minimal Sufficient Length for Efficient Reasoning via RL

82% match2026

Continuous Approximations for Improving Quantization Aware Training of LLMs

81% match2024

DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Models

81% match2025

When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models

80% match2025

Cite this paper

@article{wan2025unifying,
  title  = {Unifying Two Types of Scaling Laws from the Perspective of Conditional Kolmogorov Complexity},
  author = {Jun Wan},
  year   = {2025},
  eprint = {2501.06802},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2501.06802v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2501.06802.svg)](https://codeofpaper.com/paper/2501.06802)

Discussion