Any time a task involves pulling information off the web, there’s a good chance you’re paying more than you need to, and getting worse results in the process. Codex and Claude can both drive a browser: click around, wait for things to load, read the DOM, retry when it doesn’t work. But that loop is expensive, and on some pages it just doesn’t succeed no matter how much you throw at it.
I’ve hit this a few times now. One was a website with a JavaScript-rendered branch menu, the kind where the actual content only shows up after a script runs and populates the page. Another was a local business with a branch-specific booking menu, different services and time slots depending on which location you picked. Codex, running Soul 5.6 at high effort, spent a large amount of token budget trying to get at that information and came up empty.
FireCrawl got the same data out in 4 credits.
Why the browser-driving approach struggles here
When an agent tries to read a page like this directly, it’s working blind. It has to load the page, guess how long to wait for scripts to finish, inspect a DOM that may still be changing, and figure out which of dozens of nested elements actually contains the branch or menu data it wants. If the site uses client-side routing, or loads content behind a dropdown interaction, the agent needs to simulate that interaction correctly before the data even exists on the page. Every one of those steps costs tokens, and every failed attempt costs more while producing nothing.
FireCrawl exists specifically to solve the “this page needs real rendering” problem. It runs the page in an actual browser environment on its end, waits for JavaScript to execute, and hands back clean, structured content: markdown, HTML, or extracted data, depending on what you ask for. Your agent doesn’t have to simulate a browser session token by token. It sends one request and gets back the answer.
The economics make this an easy call
A free FireCrawl account gives you 1,000 credits a month. Extracting the branch menu and the booking data each cost 4 credits. That’s not a rounding error, it’s close to free, especially set against an agent run that burned a meaningful chunk of a paid token budget and still didn’t get the answer.
How to set it up
- Sign up for a free account at firecrawl.dev.
- Grab your API key from the dashboard.
- Give the key to whichever model you’re using, Claude, Codex, or otherwise, either as an environment variable it can reach or through an MCP server that wraps the FireCrawl API.
- Tell it to reach for FireCrawl when a page needs JavaScript rendering, has content behind interactions like dropdowns or location pickers, or has already failed once via direct browser tools.
You don’t need a paid plan to get the benefit. The free tier is generous enough that this costs nothing to try, and the next time your agent gets stuck spinning on a page that won’t render, you’ll already have the fix in place.