Download SEC Filings at Scale Without Rate Limits

Let me guess how you got here. You wrote a script that downloads 10-Ks. You tested it on 3 companies and it worked beautifully. So you pointed it at 500, went to make coffee, and came back to a terminal full of this:

HTTP/1.1 403 Forbidden

Your Request Originates from an Undeclared Automated Tool

Your IP is blocked. Not for a second — for roughly 10 minutes, on every EDGAR domain at once. And if you are on a corporate network or a NAT’d VPC, congratulations: you just blocked your colleagues too. I have done this to an entire office. Twice.

What the EDGAR rate limit actually enforces

There are 3 separate rules here, and in my experience most broken pipelines are tripping over 2 of them at the same time without realising it. Let me take them one at a time.

10 requests per second, per IP, across all of EDGAR

The SEC’s fair-access policy, in force since July 2021, limits automated access to no more than 10 requests per second, counted by source IP — not by process, script or machine. The SEC’s own announcement is explicit that the ceiling applies regardless of how many machines you spread the requests across.

3 consequences fall out of that, and I miss at least one of them every time I explain this from memory:

• The budget is shared across www.sec.gov, data.sec.gov and efts.sec.gov. Hitting the submissions JSON API and the Archives at the same time does not give you 20 req/sec. It gives you 10, split.
• The budget is shared across everyone behind your IP. Your laptop, your teammate’s laptop, your CI runner and your Lambda functions behind a NAT gateway all draw from the same 10/sec. This is the single most common cause of “it works on my machine, it 403s in production”.
• It is a ceiling, not a target. Sustained bursts at exactly 10 req/sec with no jitter look like a bot and get treated like one.

EDGAR rejects requests without a declared User-Agent

EDGAR rejects requests with no User-Agent header, and rejects the defaults your HTTP library ships with (python-requests/2.x, Go-http-client/1.1, a bare Mozilla/5.0). The expected format is an identifiable name plus a contact address:

User-Agent: Acme Research [email protected]

A missing or generic User-Agent gets you a 403 on request number one — not after 10. If you are being blocked instantly rather than after a burst, this is almost always why.

Browser JavaScript cannot call EDGAR at all

This one traps everyone building a front-end. User-Agent is a forbidden header name in the Fetch spec, so browser JavaScript is not permitted to set it. EDGAR also does not return Access-Control-Allow-Origin. A browser-only tool therefore cannot make a compliant EDGAR request, ever. There is no header trick and no mode: 'no-cors' workaround that returns readable data. You need a server in the path.

Why free SEC filing downloaders stall at scale

Before you write your own, you will almost certainly reach for an open-source EDGAR downloader. I did. They are useful, they are free, and for a handful of filings they are genuinely the right answer — I still use one for quick lookups. But every one of them is a client talking to www.sec.gov, which means none of them can move the ceiling. They can only be polite about hitting it. Here is what actually goes wrong when you scale one up.

• The rate limit is not a bug they can fix. A library throttling itself to 8 req/sec is still spending your IP’s single 10 req/sec budget. Running 2 of them in parallel does not double your throughput; it doubles your chance of a block.
• Per-thread limiters silently multiply. The most common defect in home-grown and library code alike: a limiter instantiated inside the worker rather than shared across the pool, giving you n_threads × 6 req/sec while you believe you are throttled. You find out when the 403s start.
• One filing is 4 requests, not one. Fetching a 10-K “properly” means: submissions JSON, then the filing index, then parse that index to discover filenames, then the primary document — plus one more for every exhibit you want. Your 500-company job is 2,000 requests before you have touched a single exhibit.
• Filename guessing is where they break. The primary document might be aapl-20250927.htm or a10k2025.htm. Downloaders scrape the index HTML table to find out, and that scraper is the most brittle code in the stack.
• Retries during a block extend the block. A downloader that retries a 403 immediately — and many do, because a 403 looks transient — keeps your IP in penalty. Failures compound rather than recover.
• They fail silently, and you get a corpus with holes. This is the one that costs real money. A multi-day backfill on a shared IP will hit a 403 somewhere in hour 9. If the retry logic is imperfect, the run finishes, exits zero, and hands you a dataset that is quietly missing 4% of its filings — and you will not notice until someone asks why a company disappeared from a panel.
• Nothing helps in CI or Lambda. Behind a NAT gateway you cannot even reason about who is spending your budget. Another team’s job can block yours.

None of this is a criticism of the libraries. It is a structural fact: if the origin server rate-limits you, no client can un-rate-limit you. The 10 req/sec wall is not something you engineer around. More threads do not help. More machines do not help — the limit is per IP and the SEC says so explicitly. Rotating IPs is a policy violation and a good way to get an entire ASN range throttled.

How to download SEC filings from EDGAR without getting blocked

If your volume is small, complying is entirely reasonable and I would not over-engineer it. Here is the downloader I keep around for exactly that case. It declares itself, throttles below the ceiling, backs off on failure, and caches so it never asks twice.

import time, threading, hashlib, requests
from pathlib import Path

UA = "Acme Research [email protected]"       # a real contact, or it is a 403
CACHE = Path("./edgar_cache"); CACHE.mkdir(exist_ok=True)

class RateLimiter:
    """Token bucket. ONE instance, shared by every thread."""
    def __init__(self, rate=6.0):           # deliberately under the ceiling of 10
        self.interval = 1.0 / rate
        self.lock = threading.Lock()
        self.next_slot = time.monotonic()

    def acquire(self):
        with self.lock:
            now = time.monotonic()
            wait = max(0.0, self.next_slot - now)
            self.next_slot = max(now, self.next_slot) + self.interval
        time.sleep(wait)

limiter = RateLimiter()
session = requests.Session()
session.headers.update({"User-Agent": UA})

def edgar_get(url, retries=5):
    cached = CACHE / hashlib.sha256(url.encode()).hexdigest()
    if cached.exists():
        return cached.read_bytes()          # filings are immutable: never fetch twice

    delay = 2.0
    for _ in range(retries):
        limiter.acquire()
        r = session.get(url, timeout=30)
        if r.status_code == 200:
            cached.write_bytes(r.content)
            return r.content
        if r.status_code in (403, 429, 503):
            time.sleep(delay)               # never retry a 403 immediately
            delay *= 2                      # 2, 4, 8, 16, 32
            continue
        r.raise_for_status()
    raise RuntimeError(f"gave up on {url}")

There are 4 rules baked into that snippet. I have personally got every one of them wrong at some point, so let me spell them out:

1. Throttle at 6 req/sec, not 10. The ceiling is enforced server-side with its own clock. Running at exactly the limit means normal network jitter pushes you over it.
2. Never retry a 403 immediately. Retrying during a block extends the block. Wait, and wait longer each time.
3. Cache on disk, keyed by URL. Filings are immutable — an accession never changes. Any request you make twice is a request you did not need to make.
4. One limiter object, shared by every thread. Per-thread limiters are the exact bug that gets teams blocked while they believe they are throttled.

Where compliant EDGAR downloading stops scaling

Now let me walk through the arithmetic, because this is where the plan quietly dies. A single filing is not a single request.

Job	Filings	Requests each	Total requests	Wall clock at 6 req/sec
1 company’s 10-K history	20	~4	80	13 seconds
S&P 500, latest 10-K	500	~4	2,000	5.5 minutes
S&P 500, 10 years of 10-K + 10-Q	500	~4 × 50	100,000	4.6 hours
Russell 3000, 10 years, all periodic filings	3,000	~4 × 50	600,000	27.7 hours

Look at that bottom row for a second. It is the honest number, and it still assumes zero failures, zero retries, nobody else on your IP, and no backfill re-runs. None of which will be true. Once your corpus is large enough, the 10 req/sec wall stops being an annoyance and becomes your architecture.

Download SEC filings in bulk from a mirror instead

So here is the shift that fixed it for me. Stop treating www.sec.gov as your bulk download endpoint. Keep it as the authoritative source of record — that part matters and I will come back to it — but use a mirror as the pipe.

The Business Quant SEC Filings API does this automatically. Every filing record carries 2 complete URL bundles — one pointing at the SEC, one at the Business Quant EDGAR mirror:

{
  "accession": "0000320193-25-000079",
  "formtype": "10-K",
  "items": ["item1", "item1a", "item1c", "item7", "item8"],
  "url_sec": {"url_html": "https://www.sec.gov/Archives/edgar/data/320193/000032019325000079/aapl-20250927.htm"},
  "url_bq":  {"url_html": "https://edgar.businessquant.com/Archives/edgar/data/320193/000032019325000079/aapl-20250927.htm"}
}

Look at those 2 URLs closely. The path is byte-for-byte identical. Only the host differs. That is on purpose, because it means the swap is a one-line change and every downstream parser, path regex and archive layout you already wrote keeps working exactly as it did.

The same dual URLs appear on every individual file in the filing_exhibits[] and xbrl_attachments[] arrays, so you get direct download links for the primary document, every EX- exhibit, every XBRL linkbase and every embedded graphic — without fetching and parsing the EDGAR index page first. 3 requests collapse into one, and the one that remains is not fighting a ceiling.

A bulk SEC filing downloader that actually scales

Same job as the compliant version above — download a company’s filing history in parallel. Notice how much smaller it gets once the constraint is gone: no rate limiter, no retry logic, no cache keyed by URL, and no shared lock across threads. The mirror absorbs all of that.

import requests
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path

API_KEY, BASE = "YOUR_API_KEY", "https://data.businessquant.com"
session = requests.Session()

def list_filings(ticker, formtype=None, from_date=None):
    """Page through a company's filings. Full metadata per filing."""
    out, page = [], 1
    while True:
        params = {"ticker": ticker, "api_key": API_KEY, "page": page, "page_size": 1000}
        if formtype:  params["formtype"] = formtype     # "10-K" or "10-K,10-Q"
        if from_date: params["from_date"] = from_date   # DD-MM-YYYY, not ISO
        body = session.get(f"{BASE}/secfilings", params=params, timeout=30).json()
        out += body["data"]
        if page >= body["metadata"]["total_pages"]:
            return out
        page += 1

def download(filing, outdir=Path("./filings")):
    url = filing["url_bq"]["url_html"]                  # the whole trick
    dest = outdir / filing["formtype"] / f"{filing['accession']}.htm"
    dest.parent.mkdir(parents=True, exist_ok=True)
    if not dest.exists():
        dest.write_bytes(session.get(url, timeout=60).content)
    return dest

filings = list_filings("AAPL", formtype="10-K,10-Q", from_date="01-01-2015")
with ThreadPoolExecutor(max_workers=12) as pool:
    paths = list(pool.map(download, filings))

2 things I want you to notice.

There is no rate limiter. Not because limits stop mattering as etiquette, but because the SEC’s ceiling is no longer the thing standing between you and your corpus. Keep concurrency sane — a dozen workers, not a thousand — and size it to your own bandwidth rather than to somebody else’s policy.

There is no index-parsing step. You never fetch -index.htm, never regex an HTML table to discover filenames, never guess whether the primary document is aapl-20250927.htm or a10k2025.htm. The filing record already told you.

The DD-MM-YYYY trap

One thing that will bite you, because it bit me. from_date and till_date are day-month-year, not ISO. So from_date=01-03-2025 is 1 March, not 3 January. I lost an afternoon to a backfill that looked complete and was silently 2 months short. Build the string with dt.strftime("%d-%m-%Y") and move on with your life.

Download SEC filing exhibits without parsing the index

Here is something that took me embarrassingly long to internalise: the primary document is usually not what you are after. The EX-21 subsidiary list, the EX-99.1 earnings release stapled to an 8-K, the EX-10 material contracts, the XBRL linkbases — those are the real research targets. And on raw EDGAR, each one costs you an index fetch, plus a parse, plus a hope.

The Documents & Exhibits API returns a typed manifest of every file in the submission:

curl "https://data.businessquant.com/secfilings/0000320193-25-000079/documents?api_key=YOUR_API_KEY"

{"data": [
  {"filesno": 1,  "document_type": "10-K",       "filename": "aapl-20250927.htm"},
  {"filesno": 3,  "document_type": "EX-21.1",    "filename": "a10-kexhibit21109272025.htm"},
  {"filesno": 9,  "document_type": "EX-101.CAL", "filename": "aapl-20250927_cal.xml"},
  {"filesno": 13, "document_type": "GRAPHIC",    "filename": "aapl-20250927_g1.jpg"}
]}

Filter on document_type to find the exhibits you need, then fetch any file by its filename. No index-page parsing, no filename guessing — the manifest already told you what exists and what it is called.

def manifest(accession):
    r = session.get(f"{BASE}/secfilings/{accession}/documents",
                    params={"api_key": API_KEY}, timeout=30)
    return r.json()["data"]

def fetch(accession, filename):
    """Raw bytes with the correct Content-Type. Not JSON."""
    r = session.get(f"{BASE}/secfilings/{accession}/content",
                    params={"filename": filename, "api_key": API_KEY}, timeout=60)
    return r.content

ACC = "0000320193-25-000079"
subs = {d["document_type"]: fetch(ACC, d["filename"])
        for d in manifest(ACC) if d["document_type"].startswith("EX-21")}

print(subs.keys())      # dict_keys(['EX-21.1'])

A bonus you get for free: the primary HTML comes back with inline XBRL stripped. Modern filings embed <ix:nonFraction> and friends directly in the body, and they shatter text extraction — a single dollar figure ends up split across 3 nested elements, and a naive .get_text() produces 3 orphaned strings with no reliable spacing. The documents endpoint strips those namespaces before delivery, so BeautifulSoup behaves the way you expected it to the first time you tried.

Monitor new SEC filings without a scraper

The other half of downloading at scale is keeping up, and this is where I see the most fragile code of all. It is almost always the same thing: a scraper pointed at the EDGAR full-index files, running every minute, quietly eating the same 10 req/sec budget as everything else the team owns. Then someone kicks off a backfill and both jobs die at once.

Poll the filings endpoint with a from_date window instead. New filings are indexed within seconds of EDGAR acceptance, and publishdatetime gives you minute-level precision to dedupe against.

from datetime import datetime

seen = set()   # persist this: Redis, sqlite, anything durable

def poll(tickers, formtypes="8-K,10-K,10-Q"):
    today, fresh = datetime.utcnow().strftime("%d-%m-%Y"), []
    for ticker in tickers:
        r = session.get(f"{BASE}/secfilings", params={
            "ticker": ticker, "formtype": formtypes, "from_date": today,
            "page_size": 100, "api_key": API_KEY}, timeout=30)
        for filing in r.json()["data"]:
            if filing["accession"] not in seen:
                seen.add(filing["accession"])
                fresh.append(filing)
    return fresh

A note on honesty, since the market is full of vendors who are careless with this word: this is polling, not streaming. You choose the interval; the API does not push. For an 8-K alerting workflow a 30–60 second poll is well inside the useful window, because the filing is indexed within seconds of hitting EDGAR — but call it what it is.

Because each filing record already includes an items[] array listing the sections detected in it, you can route on event type without a second call. An 8-K with Item5_02 is an executive departure. Item4_02 is a non-reliance restatement. Item2_02 is earnings.

EDGAR edge cases that break any SEC filing pipeline

What follows are properties of EDGAR itself, not of any API sitting on top of it — ours included. I am telling you about them because anyone selling you a filings product who does not warn you about these has not actually run a large backfill. I learned each of these the expensive way.

Amended filings carry only what changed. A 10-K/A gets a brand-new accession, and its document manifest contains only the files that were updated or added. It does not re-list the original exhibits. To get the complete post-amendment file set you fetch both manifests and merge them yourself. Pipelines that assume the amendment supersedes the original end up with a subsidiary list that vanished.

Pre-2003 filings are often one monolithic .txt. Before HTML adoption was widespread, filers submitted a single text blob with no discrete attached exhibits. A manifest returning exactly one record for a 1998 10-K is correct behaviour, not a coverage gap.

Graphics are binary. Requesting a GRAPHIC filename returns image/jpeg or image/gif bytes. If your HTTP client decodes everything as UTF-8, you will hit a UnicodeDecodeError a few thousand files into an otherwise clean run.

Keep the SEC URL even when you download from the mirror. For compliance workflows, audit trails or anything a regulator or client might ask you to source, cite url_sec. It is in the same response object. Download from the mirror; cite the SEC. There is no reason to choose.

Which SEC filing download approach fits your job

Everything above boils down to one question: how many filings do you actually need? Here is how I would decide.

Your situation	What to do
A handful of filings, occasionally, from a script on your laptop	Raw EDGAR, with a proper User-Agent and a 6 req/sec limiter. It is fine. Do not over-engineer.
Backfilling a corpus of thousands of filings	Mirror. The ceiling turns a 20-minute job into a multi-day one, and you will lose runs to 403s.
Anything running in CI, Lambda or a shared VPC	Mirror. A NAT’d IP shares one budget across every function you have, and you cannot reason about who is spending it.
A browser-based tool or front-end	Mirror, or any server-side proxy. Browser JS cannot set User-Agent, so it cannot call EDGAR compliantly. This is a spec-level constraint, not a bug you can fix.
Downloading exhibits (EX-21, EX-10, EX-99.1) at scale	Documents manifest. Guessing filenames from index pages is the most brittle part of any home-grown scraper.
A compliance or audit workflow where provenance matters	Mirror for retrieval, url_sec for citation. Both come back in the same record.

The short version

If you remember one thing, make it this. The SEC’s 10 requests per second is not a suggestion, it is not per-process, and you cannot thread your way around it. It is per IP, shared across every EDGAR domain and every machine behind that IP, and it fails closed with a 10-minute block. Free downloaders do not change any of that, because they are clients of the same rate-limited origin you are.

So the decision is genuinely simple. Small corpus? Comply carefully, use the limiter above, and do not over-engineer it. Large corpus? Stop making the origin server your bottleneck.

That second path is what Business Quant was built for. The SEC Filings API indexes 18M+ filings across 400+ form types back to 1993, returns them within seconds of EDGAR acceptance, and hands you a mirrored, structurally identical download URL for every file in every filing. It is free to use, with no 10 req/sec wall.

Get an API key, change one hostname, and go delete your rate limiter. I promise it feels good.