Why Earnings Press Release Tables Break Standard Scrapers

So you wrote a scraper for an earnings table. One line of a table reader, or a tidy little loop over rows, and it worked perfectly on the company you tested it against. Then you pointed it at the next filer and it returned garbage. Misaligned columns, wrong signs, values off by a factor of a thousand.

Here is the good news: nothing is wrong with your code. An earnings table is not a data structure. It is a picture of one, drawn for a human eye, and every filer draws it slightly differently. Once you internalise that single fact, everything below stops feeling like a series of bugs and starts feeling like a catalogue — the ways the picture lies to a parser, and the specific fix for each one.

The tool everyone reaches for first

Almost everyone starts in the same place, and the failure is instructive enough to be worth showing. You point a general HTML table reader at the exhibit — pandas.read_html is the usual first move, sometimes with BeautifulSoup underneath — and it cheerfully hands you back a list of dataframes:

import pandas as pd

tables = pd.read_html(earnings_html)   # returns every <table> it finds
income = tables[0]                     # ...but is [0] the income statement?

On a clean, well-behaved filer this looks like it works, and that is the trap. A general table reader is built to flatten arbitrary HTML into a rectangle; it has no concept of accounting. It keeps the blank spacer columns, so your data lands 2 columns to the right of where you expect. It flattens multi-row headers into a tuple mess or drops them. It leaves every number as a formatted string — "(1,234)", "$ 94,036" — so nothing is a float, the parentheses are still text, and the units note in the header never gets applied. And it silently picks the wrong <table> when the exhibit contains 5 of them. These libraries are genuinely useful and completely free, which is exactly why people reach for them — but they are general-purpose tools with no financial-statement logic and no stability guarantee across issuers, so on the long tail of filers they are fragile in ways you only discover in production, one wrong number at a time. What you actually need is positional logic that understands the grid, and clean input to run it on.

Get clean HTML before you parse anything

Before we fix a single parsing bug, let me remove the ones that are not parsing bugs at all. In my experience half the pain people blame on parsing is actually pain from fetching — EDGAR rate limits, SGML containers, inline-XBRL noise wrapped around the table. Clear that out first and the real problem gets much smaller. The Section Extraction API returns the 8-K Results of Operations section as clean HTML by accession, with SGML headers and inline-XBRL namespaces already stripped, so you start from the table itself instead of a filing container:

import requests

API_KEY  = "YOUR_API_KEY"
BASE_URL = "https://data.businessquant.com"

def earnings_html(accession):
    url = (f"{BASE_URL}/secfilings/{accession}/content"
           f"?item_key=Item2_02&api_key={API_KEY}")
    return requests.get(url).text          # clean HTML, not a full filing

html = earnings_html("0000320193-26-000045")

Now the only problem left is the real one: the table.

There are no tags only a grid to reconstruct

Here is the shape of what you get. Every cell’s meaning is carried by its position and its formatting, not by any attribute you can query:

<tr>
  <td>Net sales<sup>(1)</sup></td>
  <td>$&nbsp;94,036</td>
  <td></td>
  <td>(1,234)</td>
</tr>

There is no revenue attribute on that first number, no period="Q1-2026" on the column, no sign="negative" on the parenthesized value. The label is text, the scale is a sentence somewhere above, the sign is punctuation, and the column identity is a header 3 rows up that spans several cells. Each of the following will break a scraper that does not expect it — and each has a small, specific fix.

The units note that rescales everything

Start with this one, because it is the most dangerous thing on the page. A single line near the top — “in millions, except per share data” — silently sets the scale for the entire table. Miss it, and every figure you extract is off by a factor of a million while still looking completely plausible. It does not crash. It just quietly reports Apple’s revenue as $41,000 and waits for someone to notice. Read the note once, up front:

def units_scale(html):
    head = html[:3000].lower()
    if "in thousands" in head: return 1_000
    if "in millions"  in head: return 1_000_000
    if "in billions"  in head: return 1_000_000_000
    return 1

2 traps hide inside that innocent note. The phrase “except per share data” is not decoration — it means the EPS rows are not scaled, so a parser that multiplies everything by a million turns a $1.52 EPS into a nonsense number. And the note is not always in the first 3,000 characters; some filers put it in a sub-header above a specific table, or in a footnote, or repeat a different scale for a supplementary schedule. Reading it globally is a good default, but a robust parser reads scale per table, and treats per-share rows as an exception.

Parenthesized negatives commas currency and superscripts

With the scale resolved, the next layer of noise is in the numbers themselves. Accounting shows a negative as (1,234), not -1,234. Values carry commas, currency symbols and non-breaking spaces, a footnote superscript can leave a stray marker on the number, and a dash — an em-dash, en-dash or plain hyphen depending on the filer — is used to mean nil. One converter handles all of it, and the important subtlety is that it treats parentheses as a sign only when a digit is inside, so a label like “Other (expense) income” is never mistaken for a number:

import re

def to_number(text, scale=1):
    t = text.replace("\u00a0", " ").strip()
    if t in ("", "-", "\u2013", "\u2014"):                 # hyphen, en-dash, em-dash = nil
        return None
    negative = bool(re.search(r"\(\s*[\d,]", t))          # ( immediately before digits
    m = re.search(r"[\d,]+(?:\.\d+)?", t.replace(",", ""))# first numeric token only
    if not m:
        return None
    val = float(m.group(0))
    return (-val if negative else val) * scale

assert to_number("$ 94,036", 1_000_000) == 94_036_000_000
assert to_number("(1,234)")  == -1234
assert to_number("\u2014")   is None                      # em-dash means nil, not zero
assert to_number("Other (expense) income") is None        # a label, not a number

Note the choice to return None for a dash rather than 0.0. Nil and zero are different claims in a financial statement, and collapsing them loses information you may need later — keep the distinction and let the caller decide.

Colspan headers and spacer columns

You can now read a value correctly out of a cell. The harder question is which cell belongs to which period. The period columns are labeled by header cells that span several columns, with empty spacer columns inserted for visual gaps. “Which column is this quarter” is therefore not a fixed index — it depends how many spacers the filer chose to insert, and it changes between filers and sometimes between quarters for the same filer. Worse, headers are frequently 2 rows deep: a top row reading “Three Months Ended” spanning several columns, and a second row under it giving the actual period-end dates. Expand each header cell by its colspan so labels line up with data columns positionally:

def cells(row_html):
    out = []
    for m in re.finditer(r"<t[dh]([^>]*)>(.*?)</t[dh]>", row_html, re.S | re.I):
        attrs, inner = m.group(1), re.sub(r"<[^>]+>", "", m.group(2)).strip()
        span_m = re.search(r'colspan="?(\d+)', attrs)
        span = int(span_m.group(1)) if span_m else 1
        out.extend([inner] * span)                        # a colspan cell fills N columns
    return out

def period_columns(header_row_html):
    labels = cells(header_row_html)
    return [i for i, label in enumerate(labels) if label]  # non-empty = a real period col

Expanding colspan is what lets the header line up with the data rows below it. When the header is 2 rows deep, you expand both rows the same way and combine them — the top row tells you “3 months” versus “6 months,” the bottom row tells you the exact date — and only then do you know which data column is the current quarter versus the year-to-date versus the prior year.

The label is not always in the first column

Colspan fixes the column alignment, but there is still the question of which column holds the label. The compact parsers everyone writes assume cells[0] is the line-item label and everything after is a number. Real exhibits break that assumption constantly. Indentation is often done with a leading non-breaking space or a nested table, so the label sits in column zero but with invisible prefix whitespace you have to normalize before matching. Subtotal rows — “Total operating expenses” — sit at a different indent. Section headers like “Operating expenses:” are a label with no numbers at all and must be skipped, not parsed as an empty row. And a handful of filers put a blank first cell for alignment, pushing the label into column one:

def label_and_numbers(row_cells, scale):
    # find the first non-empty text cell as the label, numbers are what follow
    label = ""
    start = 0
    for i, c in enumerate(row_cells):
        if c.strip():
            label, start = c.replace("\u00a0", " ").strip(), i + 1
            break
    nums = [n for n in (to_number(x, scale) for x in row_cells[start:]) if n is not None]
    return label, nums          # section header -> label set, nums empty -> skip it

Handling this well is also what lets you tell a subtotal from a line item, which matters the moment you try to compare across filers — because 2 companies will indent and subtotal the same economics differently.

Which table is the income statement

Every fix so far assumes you are parsing the right table. Here is an assumption worth checking early, because it quietly underpins most first drafts: an earnings exhibit is rarely one table. A typical release carries a condensed income statement, headline balance-sheet lines, a cash-flow summary, one or more segment tables, and a non-GAAP reconciliation — 5 or more <table> elements with no id, no class, and no attribute you can rely on to say which is which. Picking tables[0] is a coin flip. The robust move is to identify a table by the labels it contains, not by its position:

INCOME_HINTS = ("net sales", "revenue", "net revenue", "total revenues",
                "cost of sales", "operating income", "net income")

def find_income_table(html):
    tables = re.findall(r"<table[^>]*>(.*?)</table>", html, re.S | re.I)
    best, best_score = None, 0
    for t in tables:
        text = re.sub(r"<[^>]+>", " ", t).lower()
        score = sum(hint in text for hint in INCOME_HINTS)
        if score > best_score:
            best, best_score = t, score
    return best                 # the table that reads most like an income statement

This is heuristic, and heuristics are exactly where a parser accrues its maintenance cost: a filer who calls it “Total net revenues” instead of “Net sales,” or who leads with a non-GAAP bridge that also mentions revenue, will fool a naive scorer. You harden it with more hints and tie-breakers, and you keep hardening it — indefinitely.

Putting it together

Each of the fixes above solves one layer of the problem. Combined, they give you a parser that handles the common cases across most filers. With the scale, the number converter, the column map and a way to find the right table, a parse becomes a walk over the rows — and the mixed “3 months” versus “6 months ended” column groups become a matter of selecting the right period columns rather than a silent mix-up:

def parse_table(html):
    scale = units_scale(html)
    table = find_income_table(html) or html
    rows  = re.findall(r"<tr[^>]*>(.*?)</tr>", table, re.S | re.I)
    out = []
    for row in rows:
        c = cells(row)
        if not c:
            continue
        label, nums = label_and_numbers(c, scale)
        if label and nums:
            out.append((label, nums))                     # nums[0] = latest period
    return out

for label, nums in parse_table(html):
    print(label.ljust(34), nums[0])

Why it stays broken

That parser works, and it will give you real numbers tonight. Now let me be straight with you about what we have just built, because I do not want to oversell it. It is genuinely useful and genuinely fragile at the same time. It works right up until a filer redesigns the press release, moves the scale note, adds a footnote convention, renames a line, or reorders the tables — and then you are back in the parser on a Tuesday evening. The failures are the dangerous kind: silent. There is no XBRL tag to contradict a wrong cell and no calculation linkbase to catch a subtotal that no longer adds up, so a layout change does not throw an exception — it just quietly emits a plausible wrong number into a dashboard someone trades on.

Multiply that across every issuer you cover and every quarter they report, and the parser stops being a script and becomes a system with a permanent maintenance surface. That is not a knock on your code; it is the nature of parsing a human-facing document that has no machine contract behind it. It is also precisely why the same job for foreign 6-K interims — which carry no XBRL either, in even more layout variety — took a full engineering effort of roughly 6 to 8 months to make robust, and ongoing work to keep it that way.

Or skip the parse for periodic filings

That maintenance cost is real, but it only applies to the 8-K early read. And here is the relief: You only ever need this parser for the 8-K early read. The normalized statements for the periodic filings — 10-K, 10-Q, 20-F, 40-F and 6-K — already arrive gridded, scaled, sign-corrected and mapped to standard slugs from the Financial Statements API, as one GET, with the 6-K parser you would otherwise be building already done for you:

ticker = "AAPL"
url = (f"{BASE_URL}/statements"
       f"?ticker={ticker}&statement=IS&frequency=Quarter&period=5y&api_key={API_KEY}")
data = requests.get(url).json()

for category in data["data"].values():
    for section in category["sections"].values():
        v = section["values"]
        if v:
            m = section["metadata"]
            # raw is a real signed number in base units -- no parens, commas or scale note
            print(m["slug"].ljust(20), v[0]["reportedValue"]["raw"])

Every problem the first 2/3 of this article solved by hand — the parentheses, the commas, the hidden scale, the wrong table, the 2-row header — is already handled, and handled the same way for every issuer, so you compare across companies on a stable key instead of on whatever each filer typed.

Edge cases to keep in mind

Whether you parse the 8-K yourself or read the normalized periodic filing, a few details are worth watching for.

Some tables are images. A filer occasionally embeds a rendered table as an image, which no HTML parser can read at all — you are into OCR territory, with its own error surface. Normalized statement data from the periodic filing sidesteps that failure mode entirely.

Compare on slugs, not labels. Once parsed, join and compare on the standard slug. The company’s label wording is display text and drifts between quarters; the slug is the stable key.

Confirm the section exists first. Call the Section Extraction /items endpoint to verify Item2_02 is present before fetching it, so you fail loudly on a filing that has no Results of Operations rather than parsing an empty string into silence.

Watch the non-GAAP reconciliation. The bridge table repeats revenue and income lines with adjustments, and its labels overlap the real income statement. If your table finder or label matcher wanders into it, you will pick up adjusted figures that do not tie to GAAP. Anchor on the GAAP statement.

Start Building

With the edge cases covered, here is where things stand. An earnings table breaks scrapers because it is a human-facing picture, not a data structure — merged headers, parenthesised signs, a hidden scale note, footnote noise, labels that wander between columns, and several tables with no key to tell them apart. All of it demands positional parsing, and positional parsing never stops needing maintenance.

My recommendation, in order: get the clean HTML from the Section Extraction API so you are not also fighting EDGAR. Apply the fixes above for the 8-K early read, because that read is worth having and nobody can hand it to you pre-parsed. Then read the normalized version from the Financial Statements API for every periodic filing — including the 6-K parser you would otherwise build and then babysit for years.

• Section Extraction API — the 8-K Results of Operations section as clean HTML.
• Financial Statements API — periodic filings parsed into normalized statements, by ticker.
• Documents and Exhibits API — fetch the raw EX-99.1 exhibit by type when you need the whole thing.

Get an API key, pull one earnings section, and run the parser above against a real table. You will learn more in 10 minutes than from anything else on this page.