Breadth First vs. Depth First Crawling

TLDR: Breadth first crawling visits every page at one link level before going deeper. Depth first follows a single path as far as it goes before backtracking. The choice determines which pages a crawl reaches first and what it covers if stopped early.

Both strategies describe the order in which a crawler removes URLs from its URL frontier. The mechanical difference is small: breadth first uses a FIFO queue, depth first uses a LIFO stack. The practical consequences for coverage, memory, and politeness are substantial.

How Each Traverses:

Starting from a seed URL that links to pages A, B, and C, where A links to A1 and A2:

Breadth first order: seed, A, B, C, A1, A2, B1, B2, C1, C2. All of depth 1 completes before any depth 2 page is touched.

Depth first order: seed, A, A1, A1a, A1b, A2, B, B1. The crawler descends as far as it can through A before it ever looks at B.

Direct Comparison:

Implementation:

from collections import deque

def crawl(seed, max_depth, strategy="bfs"):
    queue = deque([(seed, 0)])
    seen = {seed}

    while queue:
        url, depth = queue.popleft() if strategy == "bfs" else queue.pop()

        if depth >= max_depth:
            continue

        for link in extract_links(fetch(url), url):
            if link not in seen:
                seen.add(link)
                queue.append((link, depth + 1))

One method call separates the two strategies. Using popleft takes the oldest queued URL and produces breadth first order. Using pop takes the most recently added URL and produces depth first.

Why Breadth First Dominates:

Almost every production web crawler uses breadth first traversal or a priority variant of it, for several reinforcing reasons.

Important pages are shallow. Sites link to their most valuable content from the homepage and primary navigation. Breadth first reaches that content immediately, while depth first may spend thousands of requests in an archive before touching a main category.

Partial crawls stay useful. A breadth first crawl halted at any point has complete coverage up to some depth. A depth first crawl halted early has one deeply explored branch and nothing else, which is rarely a usable dataset.

Politeness falls out naturally. Because a level typically spans many hosts, consecutive requests tend to hit different servers, which supports polite crawling without extra scheduling. Depth first does the opposite, concentrating requests on one host.

Traps are contained. A calendar generating infinite pages pulls a depth first crawler in forever. Under breadth first, those URLs sit in a queue alongside everything else at that level and consume only their share of the budget.

When Depth First Fits:

Depth first is the better choice in a narrow set of cases, usually where the target structure is known in advance.

Extracting a single documentation section, following a linear chain such as paginated results, or operating under tight memory constraints all favor depth first. Its memory profile is genuinely better: breadth first on a large site can hold millions of URLs at a single level, while depth first holds only the current path plus siblings.

For general data collection, that memory advantage rarely outweighs the coverage and politeness costs.

What Production Crawlers Actually Do:

Large scale crawlers rarely implement either strategy in pure form. They use priority queues in which breadth first is the default ordering, modified by signals such as page importance, change frequency, domain quotas, and business value.

The result behaves like breadth first with exceptions: a high value product page discovered at depth 4 can be fetched before a low value tag page at depth 2. The underlying commitment to level by level progress remains, which is what keeps coverage predictable and crawl depth limits meaningful.