How to Build a RAG Pipeline Over SEC Filings

You build a RAG pipeline over 10-K filings and the demo goes perfectly — “what does this company make?” returns a clean paragraph and the room nods. A week later a portfolio manager asks what changed in Apple’s supply-chain risk language, and the retriever hands back chunks from the table of contents. The model writes a confident answer anyway.

That is not a model problem or a prompt problem. It is an ingestion problem, and by the time anyone noticed, the pipeline had been serving junk for weeks.

The free route through raw EDGAR

Everything you need is technically free. The SEC publishes every filing, open-source libraries will download them, and so the first version of every SEC RAG pipeline — mine included — looks almost exactly like this:

# The version everyone writes first
filing_html = requests.get(edgar_url, headers={"User-Agent": "[email protected]"}).text
text = BeautifulSoup(filing_html, "html.parser").get_text()
chunks = splitter.split_text(text)
vectorstore.add_texts(chunks)

4 things then go wrong, and in my experience they go wrong in this order.

EDGAR throttles you before your corpus is built. The SEC caps automated access at 10 requests per second, per IP address, across every EDGAR domain, regardless of how many machines you run. Exceed it and you get a 403 Forbidden and roughly a 10-minute block — on your whole IP, which in a shared VPC or office network means everyone. Every request must also carry a declared User-Agent with a real contact address; the defaults your HTTP library ships with are rejected outright. Building a corpus over even a few hundred companies means several thousand requests, and the ceiling turns a coffee break into an afternoon.

The HTML fights back. Modern filings are inline XBRL. Financial facts are wrapped in <ix:nonFraction> elements, and <ix:hidden> blocks bury thousands of characters of machine-readable tags inside the document body. Run a naive text extractor over that and your “clean text” contains orphaned number fragments and invisible tag soup — which then gets embedded, indexed, and retrieved.

Your chunks contain things that are not content. A 10-K’s raw text includes a cover page of checkbox disclosures, a table of contents, an exhibit index, and signature blocks. These embed perfectly well. They are semantically “about” the company. Cosine similarity likes them fine.

And the maintenance never ends. Filers change their HTML. Amendments arrive. Older filings have a different structure entirely. Whatever you built starts drifting the day you ship it.

So here is the lesson I paid for. Having free access to the underlying documents is not the same as having free access to usable documents. That gap is where every SEC RAG project I have seen spends its first month, and it is not the interesting month.

The real problem is chunking a filing like a blob

Now suppose someone hands you perfect text. Clean, no XBRL, no tag soup. You would think the hard part is over. It is not — because a fixed-size splitter over a whole 10-K still produces chunks that cannot answer real questions.

Boundaries land mid-thought. A 1,000-token splitter will cut a risk factor in half and glue its second paragraph onto the start of the Properties section. That chunk is about 2 things, so it retrieves well for neither.

You throw away the metadata that matters most. A chunk reading “we face intense competition” is nearly useless without knowing which company, which fiscal year, which form — and above all, which section it came from. The same sentence in Item 1A (Risk Factors) is a legally compelled statement of downside. In Item 7 (MD&A) it is management commentary. Identical words, entirely different weight. If your chunks do not carry the item key, your retriever cannot tell them apart, and neither can your model.

Most real questions are metadata questions in disguise. “What did Apple say about supply-chain risk in FY2025?” is not primarily a similarity search. It is ticker = AAPL and fiscal_year = 2025 and item = item1a — and only then a similarity search for “supply chain,” inside those 40 chunks. If your first 3 filters are performed by cosine distance across 200,000 undifferentiated chunks, you are gambling on the embedding model reading your mind.

And stuffing the whole filing is not a workaround. Apple’s FY2025 10-K runs to roughly 100,000 tokens. Large-context models can technically swallow it, but you pay to re-read the exhibit index on every question, time-to-first-token is unacceptable in a chat product, and recall measurably degrades for facts buried in the middle of a long context. Filling the window is not the same as using it.

Sections are the correct chunk boundary

Here is the whole idea, and it is simpler than the problem suggests: let the filing’s own structure define your ingestion structure. Sections first, chunks within sections. That is it.

Do that and you get 3 things at once, for free:

• Clean boundaries. SEC items are semantically coherent units by design.
• Free metadata. Every chunk inherits ticker, form type, fiscal period, accession, and item key — precisely the fields your retrieval filters need.
• No junk. You never ingest the table of contents, the exhibit index, or the signature block, because they are not sections.

So why does almost nobody do it? Because extracting sections reliably has historically meant maintaining a 10-K parser — a system with several independent, silent failure modes and a permanent seat on someone’s backlog. The architecture was never in doubt. The cost of getting there was.

Business Quant’s Section Extraction API removes that requirement. Sections are pre-extracted, cleaned, and stored at ingestion time across 6 form types — 10-K, 10-Q, 8-K, 20-F, S-1 and F-1 — so retrieval is a sub-second lookup rather than a parse. Inline XBRL is already stripped. The section-aware architecture, which was always the right one, is now also the cheapest one.

Building the corpus with pre-extracted sections

3 calls. Let me walk through them, because the shape of this is the whole point.

Find the filings, and get the section list for free. Every record from the SEC Filings API already carries an items[] array listing the sections detected in that filing:

r = requests.get("https://data.businessquant.com/secfilings", params={
    "ticker": "AAPL", "formtype": "10-K,10-Q",
    "from_date": "01-01-2021",          # DD-MM-YYYY
    "api_key": API_KEY,
})

for filing in r.json()["data"]:
    print(filing["accession"], filing["formtype"], filing["items"])
# 0000320193-25-000079  10-K  ['item1', 'item1a', 'item1c', 'item7', ...]

Your ingestion loop now knows which sections exist before requesting a single one. No probing, no 404s, no wasted calls.

Fetch each section as clean HTML. The Section Extraction API returns exactly the text that belongs to one item — no table of contents, no exhibit index, no inline XBRL tags. You get the narrative as filed, scoped to the section you asked for, and your splitter never has to guess where one item ends and the next begins.

def section(accession, item_key):
    r = requests.get(
        f"https://data.businessquant.com/secfilings/{accession}/content",
        params={"item_key": item_key, "api_key": API_KEY},
    )
    return BeautifulSoup(r.text, "html.parser").get_text("\n", strip=True)

risk = section("0000320193-25-000079", "item1a")   # Risk Factors, nothing else

Attach the metadata that makes retrieval work. This is the step most pipelines skip, and it is the reason their retriever cannot tell Apple’s FY2025 risk factors from Tesla’s FY2022 management commentary. Every field you store here — ticker, form type, item key, fiscal period — becomes a filter your retriever can use before it computes a single embedding distance.

doc = {
    "text": risk,
    "meta": {
        "ticker": "AAPL",
        "accession": filing["accession"],
        "formtype": filing["formtype"],
        "item_key": "item1a",
        "period": filing["reportperiod"],       # fiscal period covered
        "filed": filing["filingdate"],          # when EDGAR accepted it
        "source_url": filing["url_sec"]["url_html"],
    },
}

2 details I would ask you to carry forward, because both have cost me real time. First, period and filed are different dates and you need both. A 10-K covering FY2025 was filed in October 2025, and conflating the 2 quietly introduces look-ahead bias into anything you backtest. Second, store url_sec as the citation. Download through the API, cite the SEC. You get both in the same record, so there is no reason to choose.

Then split inside each section — a standard recursive splitter at roughly 800 tokens with 120 overlap — so no chunk can ever straddle the Item 1A to Item 1B boundary, and every chunk carries its item key into the vector store.

Retrieval that filters before it searches

This is the moment a working system separates itself from a demo. Watch the difference between these 2 queries.

# Vague: six chunks drawn from every company, every year, every section
results = collection.query(query_texts=["supply chain risk"], n_results=6)

# Precise: six chunks drawn from ~40 candidates that are, by construction,
# Apple's FY2025 risk factors
results = collection.query(
    query_texts=["supply chain risk"],
    n_results=6,
    where={"$and": [
        {"ticker": "AAPL"},
        {"item_key": "item1a"},
        {"fiscal_year": 2025},
    ]},
)

The second query is not incrementally better than the first. It is a different operation entirely. You have replaced “I hope the embedding model understood that I meant Apple, and 2025, and the risk section” with “here are the only 40 chunks that could possibly contain a correct answer — now rank them.” One of those is a wish. The other is a query.

Nearly every high-value filings question decomposes the same way:

Question	Metadata filter	Semantic part
What are this company’s biggest risks this year?	ticker, item1a, fiscal_year	ranking by materiality
How did management explain the margin decline?	ticker, item7	“margin,” “gross profit”
Which holdings disclose AI regulatory risk?	item1a, fiscal_year	“AI regulation”
What is their cybersecurity posture?	item1c (10-K) or Item16K (20-F)	“incident response”
What changed in risk factors mid-year?	part2_item1a, formtype=10-Q	topic

Note the last 2 rows. Cybersecurity is item1c on a 10-K but Item16K on a 20-F. Risk Factors are item1a on a 10-K but part2_item1a on a 10-Q. Hardcode item1a across a mixed corpus and every quarterly filing silently disappears from your results. A 4-line routing map prevents it:

RISK  = {"10-K": "item1a", "10-Q": "part2_item1a", "20-F": "Item3D", "S-1": "RiskFactors"}
MDNA  = {"10-K": "item7",  "10-Q": "part1_item2",  "20-F": "Item5",  "S-1": "MDA"}
CYBER = {"10-K": "item1c", "20-F": "Item16K"}

Answers with citations back to the filing

Because every chunk carries its accession, item key and SEC URL, a citation is not a vague gesture at a company. It is a specific section of a specific filing that a portfolio manager can click, a compliance reviewer can verify, and an auditor can reconstruct.

context = "\n\n".join(
    f"[{i}] {m['ticker']} \u00b7 {m['formtype']} \u00b7 FY{m['period'][:4]} \u00b7 {m['item_key']}\n{t}"
    for i, (t, m) in enumerate(zip(docs, metas), 1)
)

system = ("Answer only from the provided filing excerpts. Cite every claim with "
          "its [n] marker. If the excerpts do not answer the question, say so.")

That, right there, is the difference between a research tool and a plausible-text generator. And notice that you did not get it from a better prompt or a bigger model — you got it from having ingested sections rather than a blob, weeks earlier.

Keeping the index current

Everything above assumes the filings are already in your store. In practice, an index over SEC filings is stale the moment a company files. Poll for new filings with a from_date window and ingest incrementally — new filings are indexed within seconds of EDGAR acceptance, so a nightly poll keeps a research assistant current, and a tighter interval covers earnings windows.

Let me be precise about a word this market abuses badly: this is polling, not streaming. You choose the interval, and nothing is pushed to you. I would rather tell you that plainly than let you discover it in production. For this use case it is entirely sufficient.

The items[] array doubles as an event router for 8-K filings, with no extra call and no classification pass:

EVENTS = {
    "Item2_02": "earnings",
    "Item5_02": "executive_departure",
    "Item1_05": "cyber_incident",
    "Item4_02": "restatement",
}

for filing in new_8ks:
    for item in filing["items"]:
        if item in EVENTS:
            ingest_and_alert(EVENTS[item], filing)

Cost and quality compared

At this point you have the architecture — section-aware chunks, metadata filters, incremental updates. The natural question is what it actually costs compared to the alternatives, per question, scoped to one company’s risk factors:

Approach	Tokens into the model	Retrieval quality
Stuff the entire 10-K	~100,000	Recall degrades mid-context; you pay to re-read the exhibit index every time
Naive RAG over raw filing text	~5,000	Chunks may be table of contents, exhibit index, or half a section; no way to scope by year or section
Section-aware RAG with metadata filters	~5,000	Candidates pre-filtered to the right company, year and section before a single distance is computed

Look at the middle row, because that is the trap. It looks exactly like the third row on a cost dashboard, and it behaves exactly like the first row the moment somebody asks a hard question. Which is why I would push back on framing this as a cost optimisation at all. What section-aware ingestion buys you is not cheaper tokens. It is the ability to constrain the candidate set to chunks that could actually be correct.

Design notes for production

The architecture above works as described. What follows are the things I wish someone had told me before I shipped it the first time.

Filter on metadata, rank semantically. This is the architecture, not a workaround. The API’s search layer is built for exactly the filters your retriever needs — ticker, CIK, form type, form group, date — and the vector store handles ranking within the candidate set they produce. Every high-value filings question decomposes this way, which is why the 2 layers compose so cleanly.

Route financial figures around the language model. Item 8 returns the statements, footnotes and table markup as filed. When you need typed numeric facts to put in a model, take them from a financials endpoint or from the raw XBRL linkbases in the Documents and Exhibits API. Narrative sections are for narrative questions; asking an LLM to read a balance-sheet table and hand you a float you will trade on is a category error, whatever the source.

Use the items[] array as your ground truth. Filers exercise real discretion — trusts sometimes retitle MD&A, a small number scatter sections unusually. Because every filing record tells you exactly which sections it contains, your pipeline can always distinguish “the filer omitted this” from “my code failed,” and record the difference. That distinction is what keeps a 3,000-name panel trustworthy 6 months after you build it.

The short version

If I had to compress everything above into one sentence, it would be this: RAG over SEC filings fails because teams chunk the wrong object. A filing is not a blob of text. It is a structured document whose sections are already the semantic units you want, and whose section labels are already the metadata filters your retriever needs. You are not imposing structure on it. You are declining to destroy the structure it came with.

The only reason anyone ever chunked blindly is that extracting sections properly used to mean owning a parser and fighting a rate limit. That is no longer the trade.

The Business Quant Section Extraction API pre-extracts sections from 10-K, 10-Q, 8-K, 20-F, S-1 and F-1, sub-second and with inline XBRL already stripped, free to use. The SEC Filings API supplies 18M+ filings since 1993 with the items[] routing array in every search result, and the Documents and Exhibits API hands you every individual file when the question is numeric rather than narrative.

Build the ingestion layer on sections. I promise you, everything downstream gets easier — and the questions your users ask stop being the ones that break it.