CRS Summary Benchmark Frontier LLMs vs. real Congressional Research Service summaries

How far behind is CRS?

CRS writes excellent summaries — but slowly, and not for everything. The headline depends on the denominator: measured against every introduced bill, coverage sits near 30% — but most introduced bills never advance, and CRS prioritizes the ones that move. Measured against bills that reached the House or Senate floor, coverage is near-complete. Below: coverage by legislative stage, and by when each bill was introduced.

Loading…

Coverage by legislative stage

The clearest cut: classify every House and Senate bill by how far it got, then ask what share CRS has summarized. Coverage rises sharply the further a bill advances — from a quarter of bills that never leave introduction to nearly all that reach the floor.

Horizontal bar chart: CRS summary coverage by furthest legislative stage reached — bills that reached the floor 95% (601 of 631), bills with committee action 35% (471 of 1,336), bills introduced only 25% (3,174 of 12,562).

Volume and coverage by month of introduction

How many bills were introduced each month, split into those that already have a CRS summary (blue) and those that don't yet (off-white). Two things stand out: far more bills are introduced early in a Congress, and the summarized share shrinks steadily toward the present — recent bills have barely been covered yet.

Stacked bar chart: bills introduced per month in the 119th Congress, each bar split into the number with a CRS summary (blue) and without one yet (off-white). Volume is highest in early 2025 and declines; the summarized share falls from most of the bar early on to almost none for recent months.

What this shows

"How behind" is really two effects, and they're entangled here:

  • A timing lag. A freshly introduced bill has almost no chance of a CRS summary — coverage stays near zero for months and only builds over a year-plus.
  • Selective coverage. CRS doesn't summarize every bill, so coverage plateaus well below 100% even for the oldest bills (many minor bills that die in committee never get one).
  • Prioritized coverage. CRS concentrates on bills that move. Against bills with floor consideration coverage is near-complete; against bills with committee action it is high; the low overall figure is dominated by introduced-only bills that never leave committee.

Read the overall number as how much of the whole legislative record is covered, and the stage numbers as how CRS actually triages. The benchmark's point stands either way: CRS summaries score ~94–98% on the rubric, but a recent bill usually has no summary yet — while the models produce a passable one instantly for a fraction of a cent.

How it's measured

The monthly chart counts the full population of House and Senate bills, not a sample. We list every bill (the Congress.gov bill endpoint) and note which have a CRS summary (the /summaries listing). Bill numbers are assigned in order of introduction, so we fetch the introducedDate for ~1,100 bills spread across the numbering range to build a number → month curve, then place every bill on the timeline through it — giving true monthly volume without fetching all ~14,500 dates. We measure whether a summary exists today rather than its exact authoring date, because Congress.gov's updateDate fields are bulk-refreshed and unreliable for timing. Reproduce with python src/analyze_lag.py then python src/plot_lag.py.

Stage figures come from a census of every House and Senate bill's latest action (the Congress.gov list endpoint), classified into introduced / committee / floor by a small pattern table (src/stages.py). Sampled bills' full action histories are classified the same way, and the agreement rate between the two classifiers ships in the data file as a validity check. "Floor" means actual floor consideration (passage votes, presidential action, and the like); "committee" means markup, reporting, hearings, or calendar placement; a mere referral still counts as introduced.