List the exact fields you need, then pick a route that fits your skills: a no-code scraper for a one-off list, or an API or framework for repeat jobs. Test on a few pages, check whether the content is built with JavaScript, export the results, and read the site's terms before a full run.
Start with a field list, not a tool
Scraping sounds technical, but the hard part usually isn't the code. It's knowing exactly what you want before you start. Asking for everything on a directory gets you a messy export. Asking for the business name, town and listing URL from a known set of pages gets you a list you can use.
So write it down first: the fields, and the pages they sit on. Product name, listing URL, public business details, whatever the job needs. A short, specific field list also tells you whether a ready-made template will do or you need something custom.
Three ways in, matched to your skills
No code at all. You point a desktop app or browser tool at a page, it detects the data, and you refine what it picked up. This suits one-off business lists and anyone who doesn't write code.
Ready-made scrapers you run rather than write. Someone else has built a scraper for a popular site; you supply the inputs and collect the output. Good when your target is a well-known platform.
APIs and frameworks. You or your developer write a little code, and the service handles the awkward parts such as rendering pages or rotating proxies. This is the route for repeatable pipelines that run every week.
| Tool | Route | What it's good at |
|---|---|---|
| Octoparse | No code | Auto-detects data on a page in its desktop app, refined by drag-and-drop, with templates for sources such as Google Maps and Amazon and a managed cloud to run scrapers. Free to start, as of Nov 2026. |
| Browse AI | No code | Train a robot on a page to extract data or monitor changes, then send results to Google Sheets, Airtable or an API. Your first website's extraction is free, as of Nov 2026. |
| Apify | Ready-made scrapers | A store of ready-to-run scrapers, called Actors, for web data, lead generation and market research, plus anti-blocking and proxy tools. Developers can build their own in Python or JavaScript. |
| Firecrawl | API | Search, scrape, map and crawl through one API, returning markdown, HTML, screenshots or structured data. Handles JavaScript rendering automatically, is open source, and you can start for free, as of Nov 2026. |
| ScrapingBee | API | Handles proxies and headless browsers, with CSS and XPath extraction and AI data extraction. Its JavaScript scenarios click, scroll and type before scraping. Free API credits to start, as of Nov 2026. |
| Scrapy | Framework | Open-source Python framework for crawling sites and extracting structured data, extended by plugins for browser rendering, monitoring and anti-ban. BSD-licensed and free to use, as of Nov 2026. |
| Zyte | API or managed service | Its API picks an access strategy per site and escalates only when needed, or a managed team builds and runs collection and delivers data to your warehouse. |
Six steps for a clean first scrape
- Write the field list. Fields plus the pages they live on. If Octoparse already has a template for the site, you may be done with setup before you start.
- Choose your route. No-code tools suit one-off lists; APIs and frameworks suit jobs you'll repeat. Use the table above to match a tool to your skills.
- Check whether the page is built with JavaScript. Compare what the plain page source holds with what you see in the browser. If the content only appears in the browser, a plain request returns an empty shell and your data will be missing, so choose a route that renders pages. Firecrawl and ScrapingBee both describe browser rendering.
- Test on a handful of pages. Run the scraper on a small sample, then compare the output against the page by eye. Layout quirks and missing fields show up on small samples, long before you run thousands of pages.
- Export, then schedule repeat runs if the data changes. Send the result to a spreadsheet or database. Browse AI can keep a live spreadsheet updated, and Zyte's managed service delivers straight to a warehouse. Data goes stale, so a repeat schedule keeps the list usable.
- Before the full run, read robots.txt and the site's terms, and keep your request rate low. This step has no tool attached, and it's the one people skip.
Robots.txt is a request, not a permission slip
Here's a common misunderstanding. People treat robots.txt as a yes or no on scraping. It isn't. RFC 9309, the IETF standard that defines the Robots Exclusion Protocol, says its rules are requests that crawlers are asked to honour and aren't a form of access authorisation.
So an allowed line in robots.txt doesn't mean you may take the data, and a missing file doesn't either. Read the site's terms of use as well, treat robots.txt as one signal rather than a legal answer, and if personal data is involved, check what the regulator says where you operate, because the rules vary by country. Nothing here is legal advice.
If you're on the other side and want to limit who scrapes your own site, I've written about stopping AI bots from scraping your content.
Want AI doing the heavy lifting in your marketing?
I build the systems that handle the boring 80 percent, so you get your week back. Done properly, with the human kept in.
My advice for your first scraping job
Keep the first job small and boring: one site, a short field list, a test on a handful of pages. If you don't code, start with a no-code tool and only move to an API once you're running the same job every week. If you do code, the open-source framework gives you the most control, and an API saves you the rendering headaches. Either way, read the terms before the big run, not after. For deeper comparisons, see my lists of web scraping APIs and scraper extensions for Chrome.
Want your tool on this list? Featured spots and reviews here are paid partnerships. Here's how to partner with me.
Frequently asked questions
Do I need to code to scrape website data?
No. Octoparse and Browse AI both describe no-code scraping, and Octoparse offers ready-made templates. If you write Python or JavaScript, Scrapy and the API tools give you more control.
Why does my scraper return empty results on some pages?
The page may build its content with JavaScript. Firecrawl, ScrapingBee and Zyte describe browser rendering, and ScrapingBee can click, scroll and type before it scrapes.
Does robots.txt make scraping allowed or forbidden?
Neither on its own. RFC 9309 says robots.txt rules are requests crawlers are asked to honour and aren't a form of access authorisation. Read the site's terms too.
Is there a free or open-source way to scrape?
Scrapy is an open-source Python framework, and Firecrawl also describes itself as open source. Several other tools describe a free start, as of Nov 2026.
Can I send scraped data straight to a spreadsheet?
Yes. Browse AI states that it can turn websites into live spreadsheets in Google Sheets or Airtable.
