My First Crawler: The Spider That Collected the World

From Tukuk-OS Wiki, the free knowledge base.

Building a web crawler of my own: from blindly following links to respecting robots.txt, handling heavy pages, and avoiding becoming a burden to others.

The crawler is the part of a search engine I feared most before building one. It sounds aggressive — a bot visiting other people's sites and copying content. But after building my first crawler I understood it is the most courteous work in the whole system: not collecting everything, but collecting properly and responsibly.

The Basic Framework: Follow Links, Do Not Forget

The simplest crawler is a queue and a set. Take a seed URL, fetch the page, extract links, add unseen ones, repeat. The structure seems simple until you notice several obstacles:

• The same page can be reached through dozens of URL variants.

• Links can take you outside the site you were exploring.

• Some sites' pages contain thousands of links (calendars, archives).

My solution was to normalise URLs (drop www, unify https, strip tracking parameters), limit depth, and filter domains. Filtering utm_ tracking parameters alone prevented hundreds of duplicates from entering my index.

Respecting robots.txt: Not Optional

My first crawler did not read robots.txt. I noticed something odd when I looked at the logs and realised I had accessed an /admin section that a site explicitly forbade. I stopped the service, wrote robots.txt handling into my crawler, and added a courteous crawl delay.

Since then the Tukuk-OS crawler reads robots.txt before requesting any page, honours Disallow, and limits request rates to a reasonable number per domain. This is not just ethics; it is responsibility. Other people's servers have real users waiting, and I do not want to be the reason they slow down.

Extraction: Finding the Needle

After downloading a page the real challenge begins: separating content from junk. Ads, scripts, navigation, and widgets fill most of HTML. I wrote an extractor that prefers <article> and <main>, then falls back to the body, and removes script, style, nav, and footer elements.

I also extract metadata: title, description, keywords, OpenGraph, publication date, and site name. That metadata then becomes the filterable fields in the Tukuk-OS index — when users filter by "Publisher" or "Language", it is this extraction work that makes it possible.

Handling Variety in Page Formats

Every site is designed differently, and my crawler has to pull useful content out of all of them. I wrote a tiered detector: first look for <article> or a main element, then fall back to <main>, then to the whole body with navigation stripped out.

I also handle pages with almost no text — image galleries, video pages, short link lists. Instead of discarding them I extract what exists (title, description, outgoing links) and tag them with a suitable type so they remain searchable without dominating results for queries that ask for deep content.

What surprised me most is how much real content hides in nearly identical templates. Hundreds of blogs across millions of pages use two or three common themes, so once I wrote extractors for a few popular themes, extraction quality improved dramatically without needing per-site rules.

I also learned to respect time: some pages take seconds to fully load because they run many scripts. Instead of waiting I set a limit and use the raw HTML text — because on most pages the actual content does exist in the HTML rather than being generated by scripts.

Managing Scale Honestly

At first I tried to crawl too much too fast. My server began running out of memory, and some sites started rejecting my requests. I learned three rules: limit concurrent interfaces, keep state so a restart does not start from zero, and log every decision so I can see problem patterns.

I also learned to value: not all pages are worth the same. Pages with reasonable content length, recent updates, and a good number of inbound links are worth more than navigation listings. This ranking stage reduced how many pages I had to store without reducing index quality.

When Content Changes or Disappears

Pages die, move, or change content. I added periodic rechecks for already-indexed pages, then update or remove them. This process — known as index maintenance — I never thought about before building a crawler, but it takes as much time as collecting new content.

Recovery matters too. When I reran an index from scratch after corruption, I wanted my seed list and crawl state stored in a human-readable format. Designing for emergencies from day one saved me hours later.

The Lesson

Building a crawler taught me to be polite at scale. The web is a shared space, and how you navigate it reflects who you are as a builder. Tukuk-OS crawls gently, respects limits, and stores only what is needed to help people find information. That is the idea behind a good search engine: serve users, never exploit other people's servers.

See also

• Building an Inverted Index by Hand: My First Search Engine Code
• Meilisearch: From a Hand-Built Index to a Fast Production Engine
• What Is Tukuk-OS? What This Search Engine Actually Does
• Image and Video Search in Tukuk-OS: How It Actually Works