One PlatformFor All Your Research, Reading, Papers, Notes, Questions, Answers, Sources, Findings, Discussions

Import papers, highlight exact passages, and keep every note, question, and AI reply anchored to the source. LitReviewer turns scattered reading into a connected workspace for literature reviews.

See the workflow →Learn more

Preview — scaling_laws_2020.pdfPage 3 / 32

Scaling Laws for Neural Language Models

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei

Research illustration for LitReviewer

Abstract

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude.

Simple equations govern the dependence of overfitting on model and dataset size, allowing readers to determine the optimal allocation of a fixed compute budget.

1 Introduction

Language provides a natural domain for the study of artificial intelligence. Recent work shows that performance improves smoothly and predictably as model size, data, and compute are scaled.

The existence of scaling laws implies that larger models are significantly more sample efficient than smaller models.

For models trained on sufficiently large datasets, the test loss follows a precise power-law in the number of non-embedding parameters.

L(N) = (N_c / N)^α_N ; α_N ∼ 0.076

These relationships hold across more than seven orders of magnitude and show no signs of deviating.

Google Docs — Lit Review Notes

EditingLast edit: copied from ChatGPT

Scaling Laws: key notes

The paper says loss scales as a power-law with model size, data, and compute. Need to cite exact passage.

ChatGPT: This means each multiplicative increase in compute produces a predictable reduction in loss.

Missing exact quote from page 3

Compare against Chinchilla result

Ask Maya which figure belongs in draft

ChatGPTGPT-4

New chat

Can you summarize Figure 1? I only have the caption copied.

I need the figure or surrounding text to interpret it accurately.

Send a message

ChatGPTGPT-4

New chat

You

I copied this paragraph from the PDF. What does "scales as a power-law" mean here? Also compare it to Chinchilla if relevant.

ChatGPT

It means the relationship is stable on a log-log scale: multiplying model size, data, or compute yields a predictable change in loss. The Chinchilla result later changes the compute-optimal allocation.

Send a message

Literature Review List

PaperStatusNotes

Scaling LawsReadingquote?

ChinchillaTo Readcompare

AttentionDoneintro

BERTTo Read

Finder — Downloads

Downloads

Papers

Draft

Screens

kaplan.pdf

chinchilla.pdf

fig1.png

notes.txt

draft.docx

todo.csv

Google Search

chinchilla scaling laws compare kaplan

Training Compute-Optimal...

DeepMind paper on compute-optimal language models...

Scaling laws for neural language...

Stickies

- ask Sarah about Chinchilla

- re-read §3

- where did I put fig 1?

Ready

Attention-Based Review Systems

A. Researcher, M. Reader, and L. Reviewer

Abstract

Reading systems increasingly combine source documents, marginal discussion, and generated explanations. We describe a review interface that keeps each model response attached to the cited passage, allowing readers to inspect claims without leaving the page.

The central design claim is that annotations, discussions, and guide steps should share a common anchor model. A comment created from highlighted text should remain visible in the thread list, recoverable from the outline, and available as context for a guided walkthrough.

1 Introduction

Researchers often move between a PDF, a notes document, and a separate chat window. The separation makes it difficult to reconstruct why a question was asked or which sentence supported the answer. A review tool can reduce this friction by treating the paper as the primary workspace.

In this setting, text selection is more than decoration. The selected passage becomes an addressable object: it can collect replies, receive follow-up questions, and appear in an outline of unresolved issues.

1.1 Anchored Reading

Anchored reading preserves locality. Instead of summarizing an entire document at once, the reader asks questions from the exact paragraph that motivated them. The interface can then show the answer beside neighboring human comments.

This also improves later review. Open threads expose weak evidence, resolved threads record decisions, and guide steps provide a compact route through the paper.

2 Evidence

A usable review environment should distinguish what the paper says from what the reader infers. When a model responds to a selected passage, the response should inherit the source location and remain available for later audit.

We found that readers returned to anchored discussions when drafting literature review sections. The visible link between a sentence and its discussion reduced duplicate notes and made unresolved questions easier to triage.

The result is not simply faster reading. The important behavior is recovery: a reader can reopen a question days later and see both the original passage and the surrounding conversation.

2.1 Discussion Context

Threaded comments help preserve disagreement. One reader may flag an unsupported claim, another may attach a related source, and the assistant can summarize the local evidence without flattening the exchange into a single answer.

The interface therefore treats comment state as part of the reading process. Active threads remain prominent; resolved threads stay available as archival context.

Signal	Use
Open thread	Needs review
AI reply	Local synthesis

3 Model Architecture

The guide feature converts document structure into a sequence of situated explanations. Each step points to one passage or figure and explains why that region matters for the reader's current goal.

Outline

Threads

PDF Anchor

Comment State

Guide Step

Next Passage

Reader

Export

Because every guide step is attached to a passage, the walkthrough can advance without hiding the source. Readers can pause the guide, open the relevant discussion, and return to the same step.

This makes the assistant less like a separate chat surface and more like a structured reading layer. The model participates in context, but the paper remains the stable object.

Stay Organized

Import papers into collections, track reading status, and keep projects separated as they grow.

Annotate & Ask

Highlight the exact passage, attach a note, and ask the assistant in the same thread.

Map Your Field

Connect papers into citation paths and semantic neighborhoods to see the shape of a topic.

Your Research Landscape

Knowledge Map

Ready

Attention-Based Review Systems

A. Researcher, M. Reader, and L. Reviewer

Abstract

1 Introduction

1.1 Anchored Reading

This also improves later review. Open threads expose weak evidence, resolved threads record decisions, and guide steps provide a compact route through the paper.

2 Evidence

The result is not simply faster reading. The important behavior is recovery: a reader can reopen a question days later and see both the original passage and the surrounding conversation.

2.1 Discussion Context

The interface therefore treats comment state as part of the reading process. Active threads remain prominent; resolved threads stay available as archival context.

Signal	Use
Open thread	Needs review
AI reply	Local synthesis

3 Model Architecture

The guide feature converts document structure into a sequence of situated explanations. Each step points to one passage or figure and explains why that region matters for the reader's current goal.

Outline

Threads

PDF Anchor

Comment State

Guide Step

Next Passage

Reader

Export

Because every guide step is attached to a passage, the walkthrough can advance without hiding the source. Readers can pause the guide, open the relevant discussion, and return to the same step.

This makes the assistant less like a separate chat surface and more like a structured reading layer. The model participates in context, but the paper remains the stable object.

Usage-Based Billing

Transparent, At-Cost Pricing

LitReviewer uses credits for the two actions that create cost: importing papers and asking the assistant to reply.

Voucher Ref. LR-PRC-001

Volume	Description	Subtotal
arXiv import	LaTeX source - fast	~$0.05
PDF upload	PDF parsing - slightly slower	~$0.06

Volume	Description	Subtotal
~500 tokens	Quick passage answer	$0.003
~3K tokens	Deep multi-section reply	$0.015

One PlatformFor All Your Research, Reading, Papers, Notes, Questions, Answers, Sources, Findings, Discussions

Attention-Based Review Systems

Abstract

1 Introduction

1.1 Anchored Reading

2 Evidence

2.1 Discussion Context

3 Model Architecture

Stay Organized

Annotate & Ask

Map Your Field

Knowledge Map

Attention-Based Review Systems

Abstract

1 Introduction

1.1 Anchored Reading

2 Evidence

2.1 Discussion Context

3 Model Architecture

Transparent, At-Cost Pricing

Paper processing

Assistant reply

One PlatformFor All Your ResearchResearch, Reading, Papers, Notes, Questions, Answers, Sources, Findings, Discussions

Stay Organized

Annotate & Ask

Map Your Field

Knowledge Map

Transparent, At-Cost Pricing

Paper processing

Assistant reply

One PlatformFor All Your Research, Reading, Papers, Notes, Questions, Answers, Sources, Findings, Discussions