How to Build a Financial Statements Database from SEC Filings

The plan is perfectly reasonable. A table of financial statements you can query, join and chart, built from the primary source. I have written that ticket myself.

Then you start, and you discover something slightly deflating: “build a database” was mostly “build a parser.” The database — the part you actually wanted, the part that was the whole point — turns out to be the easy piece you saved for last and never quite reached. So let me give you the map of that terrain before you set off: what each stage really costs, where the schema decisions bite, and how much of it you can simply skip.

A home-grown fundamentals ETL is 5 stages, and 4 are the parser

1. Fetch. Pull filings from EDGAR under a 10-requests-per-second-per-IP ceiling and a mandatory User-Agent, with a cache so you never fetch an immutable filing twice.
2. Parse. Turn XBRL 10-K and 10-Q instance documents into facts — contexts, units, dimensions, linkbases — and turn non-XBRL 8-K and 6-K HTML tables into numbers positionally.
3. Normalize. Map thousands of issuer-chosen tags and custom extensions to a standard line-item set, keyed so companies compare, across both us-gaap and ifrs-full.
4. Model. Design a schema that stores a time series per line item, orders the statement, marks subtotals, survives ragged templates, and does not double-count restated periods.
5. Sync. Keep it current as new filings land, incrementally, without re-pulling 20 years every night.

Only stages 4 and 5 are really about your database. Stages 1 through 3 are the entire parser project — the rate limiter, the XBRL concept map, the non-XBRL parser — each covered in its own guide, and none of them is why you set out to build a database. It is worth seeing why each of the first 3 is heavier than it looks before deciding how much to build.

Stage one is a rate-limit problem before it is a data problem

Fetching feels completely trivial right up until it meets production, and then it is the first thing to break. EDGAR’s 10-requests-per-second ceiling is per IP and shared across everything behind that address — your laptop, your CI runners, your serverless functions behind one NAT. This is why a backfill that flies on your laptop returns a wall of 403 Forbidden the moment it runs in the cloud alongside other jobs, and retrying the 403 only extends the roughly 10-minute block. Every request also needs a real User-Agent naming a contact; the library default is rejected on the first call. There is no paid SEC tier to buy your way out of this — the only fix is a global rate limiter, a durable cache of immutable filings, and careful IP hygiene.

Even if you build all of that, fetching is not one request per filing. A filing is an SGML container with many attachments and arbitrary filenames; to reach a specific exhibit you first fetch and parse the index to learn the filename, then fetch the exhibit — 2 round trips against the same shared budget. Multiply by a universe of thousands of companies and 20 years of history and stage one is an infrastructure project on its own. If you do walk this path, the SEC Filings API and Documents and Exhibits API at least retire it: they list a filing’s XBRL attachments and exhibits by direct URL and return any document clean by document type, over a mirror that is not SEC-rate-limited, so fetching stops being a rate-limit problem and leaves only the parse and normalize work.

Stage 2 hides a dimensional trap

Parsing XBRL is not just reading numbers out of tags. A single concept like revenue appears in many contexts in the same instance document — one for the consolidated total, and others for each reportable segment, geography or product line, because those are dimensional members. If you pull “revenue” without filtering to the non-dimensional consolidated context, you either grab a segment figure by accident or sum the total together with its own members and double-count. You also have to read the decimals attribute to get scale right (thousands versus millions), distinguish instant contexts (balance-sheet items) from duration contexts (income and cash-flow items), and handle the units. Open-source processors such as Arelle will parse the instance into typed facts and save you the mechanical XBRL work — but they hand you every context and every custom extension tag verbatim, and choosing the right context and mapping the tags is still yours to build and maintain. And none of it touches the 6-K and 8-K, which have no XBRL and need a positional HTML parser instead.

Stage 3 is where the tags never stop

And now the stage that never finishes. I do not say that for effect — normalization genuinely has no end state. Issuers do not tag consistently: one company’s revenue is Revenues, another’s is RevenueFromContractWithCustomerExcludingAssessedTax, a third invents a custom extension because its line item does not fit any standard concept. Foreign filers add the entire ifrs-full vocabulary on top. Mapping thousands of these onto one comparable line-item set, with the right sign, is the core of the whole exercise — and because both taxonomies revise annually and companies re-tag between years, the map is a living thing you maintain forever. This is the work that actually makes a foreign name and a domestic name comparable, and it is exactly the work a parsing library leaves undone.

Financials are ragged, so your schema must be too

Those 3 stages are the parser you did not plan to build. Now you can finally talk about the database itself — and the first thing you learn is that the obvious schema is wrong.

Right — we have finally reached the database stages, which is where you thought you were starting. Your first schema instinct will be a wide table, one column per line item. Mine was. It falls apart almost immediately, because financial statements are not one shape. A bank’s income statement has net interest income and provision for credit losses where an industrial has cost of goods sold; an insurer has premiums earned and claims incurred; a REIT has funds from operations; a utility has regulated revenue. There is no fixed column set that fits all of them, and inventing one wide table with hundreds of mostly-null columns is a schema that fights you forever.

The design that survives is the opposite: one row per reported value, with a template discriminator that says which statement shape this issuer uses (general, financials, insurance, real estate, utilities) and a stable slug that identifies the line item. Ragged statements become tall-and-narrow data, and the template column lets you pull the right presentation without special-casing every sector in code. This is why the schema below stores values as rows, not columns.

Design the schema first

The schema is the part worth thinking about, because a good one makes ragged templates, restatements and incremental updates trivial. Store one row per reported value, with a natural key that identifies a period uniquely and columns that carry the structure:

CREATE TABLE statement_line (
  ticker           TEXT    NOT NULL,
  statement        TEXT    NOT NULL,   -- IS | BS | CF | Ratios | Growth
  frequency        TEXT    NOT NULL,   -- Annual | Quarter | TTM
  slug             TEXT    NOT NULL,   -- stable line-item key, e.g. 'revenue'
  normalized_date  TEXT    NOT NULL,   -- period key, joins across statements
  raw              REAL,               -- full-precision value in base units
  itemtype         TEXT,               -- lineitem | subtotal | grandtotal
  sno_total        INTEGER,            -- global sort key -> rebuilds order
  template         TEXT,               -- general | financials | insurance | real_estate | utilities
  currency         TEXT,
  PRIMARY KEY (ticker, statement, frequency, slug, normalized_date)
);

That primary key is the whole trick. Because a period is identified by (ticker, statement, frequency, slug, normalized_date) and not by which filing it came from, a restated value simply overwrites the old one on the next load — no duplicate rows, no merge logic. The sno_total column means you rebuild statement order with an ORDER BY, not a join. And the template column carries the raggedness, so a bank and an industrial live in the same table without either one distorting the other.

Decide now whether you store history-of-history

The schema above handles ragged templates and restatements cleanly. But there is one more design choice that has to happen before you load a single row, because changing your mind later means a migration.

There is one schema decision I would make deliberately, out loud, before you load a single row — because changing your mind later means a migration. Do you keep only the latest reported value for each period, or every value that was ever reported for it? The schema above keeps the latest — a restatement overwrites in place, and a query for 2023 returns the current figure. That is what you want for analysis, screening and current-state reporting, and it is the simplest thing that is correct.

A backtest is different. If you test a strategy over 2023 using the restated 2023 numbers, you have handed your model information the market did not have at the time — look-ahead bias baked straight into the data. Avoiding that means a bitemporal design: 2 dates per row, the period it describes and the knowledge date when it became known, so you can ask “what did the books say as of this date” rather than “what do the books say now.” Bitemporality is more schema and more discipline, so build it only if you actually backtest; but decide on purpose, because retrofitting a knowledge-date axis after you have overwritten your history is painful. The restatement mechanics behind this choice get their own article.

Collapse the loader to a GET and an upsert

With the schema decided and the bitemporality question settled, the only thing left is actually loading data. And this is where the payoff of skipping stages 1 through 3 lands.

Because the Financial Statements API has already done fetch, parse and normalize — stages 1 through 3, the entire parser — your loader is the fifth stage only. It reads the normalized statement and upserts rows keyed on that primary key:

import sqlite3, requests

API_KEY  = "YOUR_API_KEY"
BASE_URL = "https://data.businessquant.com"
db = sqlite3.connect("fundamentals.db")

UPSERT = """
INSERT INTO statement_line
  (ticker, statement, frequency, slug, normalized_date,
   raw, itemtype, sno_total, template, currency)
VALUES (?,?,?,?,?,?,?,?,?,?)
ON CONFLICT(ticker, statement, frequency, slug, normalized_date)
DO UPDATE SET raw=excluded.raw, itemtype=excluded.itemtype,
              sno_total=excluded.sno_total, template=excluded.template,
              currency=excluded.currency;
"""

def load(ticker, statement="IS", frequency="Annual", period="all"):
    url = (f"{BASE_URL}/statements"
           f"?ticker={ticker}&statement={statement}"
           f"&frequency={frequency}&period={period}&api_key={API_KEY}")
    doc = requests.get(url).json()
    md = doc["metadata"]
    rows = []
    for category in doc["data"].values():
        for section in category["sections"].values():
            m = section["metadata"]
            for v in section["values"]:
                rows.append((ticker, statement, frequency, m["slug"],
                             v["normalizedDate"], v["reportedValue"]["raw"],
                             m["itemtype"], m["sno_total"],
                             md["template"], md["currency"]))
    db.executemany(UPSERT, rows)
    db.commit()
    return len(rows)

The upsert is idempotent: run it twice and nothing duplicates. Run it after a restatement and the changed periods update in place. That property comes entirely from keying on the period rather than the filing.

Backfill once, then sync incrementally

The loader handles one ticker, one view, one frequency at a time. Wrapping it into a backfill-then-sync pattern is the last piece.

Backfill with period=all to pull the full 20+ year history for each view, then keep current by polling a recent window on a schedule and letting the upsert absorb it. Filings are sparse outside earnings season, so you poll on an interval you control — daily is plenty, tightened during earnings weeks — rather than re-pulling history:

VIEWS = ["IS", "BS", "CF", "Ratios", "Growth"]

def backfill(ticker):
    for view in VIEWS:
        for freq in ("Annual", "Quarter"):
            load(ticker, view, freq, period="all")

def sync(ticker):
    # Recent window only; the upsert overwrites the latest periods,
    # picking up new quarters and any restated comparatives.
    for view in VIEWS:
        for freq in ("Annual", "Quarter"):
            load(ticker, view, freq, period="5y")

backfill("AAPL")     # once
sync("AAPL")         # on a schedule you control

You never re-pull 20 years nightly, and you never write duplicate-detection logic, because the primary key does that work. Querying the statement back out is now ordinary SQL — and because sno_total is a column, the intended line order rebuilds with a sort:

SELECT slug, normalized_date, raw
FROM statement_line
WHERE ticker='AAPL' AND statement='IS' AND frequency='Annual'
ORDER BY normalized_date DESC, sno_total ASC;

Edge cases to keep in mind

The backfill-and-sync pattern above covers the main path. A few things are worth knowing before you put it into production.

Restatements are handled by the key, and by the source. The upsert overwrites in place, and the API already returns reconciled latest values per period, so amended filings do not double-count. The mechanics get their own article.

Validate after load, not never. A quick post-load check that subtotals sum and the balance sheet balances turns a bad row into an alert. See how to validate parsed financial statements.

Store the currency and template columns. Foreign issuers report in their home currency; keeping currency per row is what lets you convert correctly later instead of silently mixing units, and template is what keeps banks, insurers and REITs from distorting a shared schema.

Batch your sync cadence to filings. Filings are sparse outside earnings season; a daily poll is plenty, with a tighter loop during earnings weeks.

Start Building

So here is what I hope you take away, and it is not really about databases. The database was never the hard part. The parser was — and it was hiding inside stages 1 through 3 of your ETL the whole time, from the shared EDGAR rate limit, through the dimensional contexts, to the tag map that never stops growing.

Once fetch, parse and normalize live behind an endpoint, a fundamentals database becomes what you originally imagined it would be: a schema keyed on the period, ragged by template, and a loader that GETs and upserts. Backfill once. Poll a recent window. Let the primary key handle restatements and duplicates while you go do the analysis you actually wanted to do.

• Financial Statements API — normalized statements, 20+ years, 5 views, by ticker.
• SEC Filings API — filings and their XBRL attachments by direct URL, over a mirror that is not SEC-rate-limited.
• Stock Screener API — enumerate the universe of tickers to backfill.

Get an API key, create the table, and load your first company with a single upsert loop.