Getting Data from Websites with Agents

A plain HTTP request sees nothing but an empty shell on many modern websites – JS rendering, rate limits, and robots.txt all belong in the toolkit.

The JS-SPA problem

Many modern websites are single-page apps: the first HTTP request returns an almost empty HTML skeleton, and the actual content loads afterward via JavaScript in the browser. A plain curl call or a simple fetch tool then sees only the empty shell – not the text a human sees in the browser.

Fetch tools and rendering

A fetch tool like WebFetch retrieves a server's raw response and converts HTML into Markdown – fine for classic, server-rendered pages. For JS-heavy pages you instead need a rendering service or a real (headless) browser that executes the page first and then hands back the fully rendered content.

The legal side, briefly

A robots.txt file at a website's root defines which areas automated requests may crawl – respecting it is baseline web etiquette, even though it isn't technically enforced. On top of that come a site's terms of service, which sometimes restrict scraping explicitly.

Respecting rate limits

Servers often respond to too many requests in a short time with status 429 ("Too Many Requests"). An agent that ignores this and keeps requesting can overload a site or get itself blocked – pauses and a moderate request pace are part of scraping responsibly.

EXAMPLE

Prompt: "Fetch https://example.com/products and list all product names with prices. If the page is rendered client-side and no content is visible, say so explicitly instead of returning an empty list."

QUICK QUIZ

Why does a plain HTTP request (e.g. curl) return almost no content on many modern websites?