The short version: Yes, you can legally watch almost everything a competitor does on their public website, from pricing changes to hiring plans to backlink strategy, using free and paid tools that scrape only what’s already public. The line you cannot cross is logging into their systems, scraping password-protected content, or breaching a site’s terms of service in a way that causes damage. Everything else, the Wayback Machine, SimilarWeb, job boards, Companies House filings, is fair game and most businesses aren’t using even half of it.
Why “spying” is the wrong word (and the right one at the same time)
I don’t love the word spying. It sounds like something you’d get sued for. But when a client says “I want to spy on my competitor’s website,” what they mean is: I want to know what they’re doing before my customers tell me about it. That’s not sneaky, that’s just competent business.
I had a client in the recruitment software space in 2023 whose biggest rival went quiet for six weeks. No blog posts, no LinkedIn activity, nothing. Everyone on the team assumed they’d stalled. I pulled up their careers page and found they’d posted eleven new roles in a fortnight, mostly sales and customer success, all tagged “remote UK.” Then I checked Companies House and saw they’d filed a change of registered office to a bigger address in Manchester. They weren’t quiet. They were building a sales team and moving offices ahead of a funding round. My client accelerated their own hiring plan by four months because of two free, entirely legal checks that took maybe twenty minutes.
That’s the whole game. None of what follows requires a hacker, a lawyer, or a subscription to something shady. It requires patience and knowing where to look.
Start with what’s already been published: the Wayback Machine
The Internet Archive’s Wayback Machine has snapshotted billions of pages since 1996, and it’s free. Go to web.archive.org, drop in a competitor’s URL, and you can see exactly how their homepage, pricing page, or about page looked six months or six years ago.
This is how you catch pricing changes without paying for a monitoring tool. Pull up their pricing page snapshot from January, then compare it to today. I’ve used this to spot a competitor quietly removing their “starting from £29” tier and repositioning as a premium-only product, months before they announced it publicly. If you want something that does this automatically going forward rather than retroactively, I’ve written a full walkthrough on how to track what a competitor changes on their website, including the specific tools that alert you the moment a page updates.
Traffic and audience data: SimilarWeb, BuiltWith, and what they tell you
SimilarWeb gives you an estimate of a competitor’s monthly traffic, top traffic sources, and even which other sites their visitors also browse. The free version is rough (treat the numbers as directional, not gospel) but the paid tier, starting around $199 a month, gets specific enough to show you whether their growth is coming from paid search, organic, or referral traffic. If a competitor’s traffic jumps 40% in a month and SimilarWeb shows it’s almost entirely paid, that tells you they’ve opened a new ad budget, not that their content suddenly went viral.
BuiltWith is the one people forget. Type in any URL and it tells you every piece of tech that site runs, its CMS, its email platform, its live chat widget, its analytics stack, even its A/B testing tool. I use this constantly to see what marketing stack a competitor has invested in. If they’ve just installed a £15,000-a-year enterprise chatbot, that’s a signal about where their budget is going and what they think matters right now.
What their backlinks and keywords say about their next move
Ahrefs and SEMrush both let you plug in a competitor’s domain and see every keyword they rank for, every page driving their organic traffic, and who’s linking to them. This is less “spying” and more “reading their homework.”
If a competitor suddenly ranks for fifteen new keywords around, say, “AI content approval,” that’s not an accident, it’s a deliberate content push, and it usually means they’ve hired someone or built a process internally to support it. I’ve written before about building an AI content approval system that protects your brand, and half the reason I built mine was watching three competitors quietly shift their content operations the same way within a single quarter, all visible through their new backlink profiles and keyword gains.
The public paper trail: Companies House, SEC filings, and job postings
This is the bit most marketing blogs skip entirely, and it’s the most reliable of all of it.
In the UK, Companies House is free and gives you a competitor’s filed accounts, director changes, registered office moves, and charges against the company (which can tell you if they’ve taken on debt or secured investment). US public companies file with the SEC, and even private US companies often leave a trail through state business registries.
Job postings are an underused goldmine. A competitor posting for a “Head of Partnerships, Europe” role tells you they’re expanding into Europe, months before it’s public. A competitor posting for three data engineers tells you they’re building something that needs serious infrastructure, possibly an AI product. Check their careers page directly, and check LinkedIn Jobs filtered by company name. It costs nothing and it’s entirely legal because they published it themselves.
Social listening and newsletter subscriptions
Sign up for your competitor’s email newsletter using a spare address. It sounds obvious and almost nobody does it consistently. I keep a Gmail account whose sole purpose is subscribing to competitor newsletters, and I read every one. You learn their offer cadence, their discount patterns, their tone shifts, and often their upcoming launches days before the press release.
Tools like Brand24 or Mention will alert you whenever a competitor is mentioned anywhere online, blogs, forums, Reddit, news sites. Set a free Google Alert for their brand name and their founder’s name too. Founders leak more strategy in podcast interviews and LinkedIn posts than in any official channel.
A simple weekly process (the one I run)
Here’s the actual routine I use for my three closest competitors, roughly 45 minutes a week total:
- Monday: check Wayback Machine snapshots of their homepage and pricing page against last month’s
- Monday: scan their careers page for new roles
- Wednesday: check SimilarWeb for traffic trend direction, not exact numbers
- Wednesday: skim my inbox for their newsletter
- Friday: check Ahrefs for new ranking keywords and new backlinks
- Monthly: check Companies House for filing changes
- Monthly: run BuiltWith to see if they’ve added new tech
None of this requires a big team or a big budget. Some of it is free entirely. The paid tools, SimilarWeb, Ahrefs, together run you maybe £250 to £400 a month depending on tier, which is less than most businesses spend on a single Facebook ad campaign that doesn’t work.
Where you’d cross the line (and why almost everyone gets close to it anyway)
Legal doesn’t mean you can do anything you can technically do. The line sits roughly here: viewing, reading, and analysing public pages is legal. Automated scraping at scale that ignores a robots.txt file or hammers a server hard enough to cause damage can breach computer misuse laws, and in the US the Computer Fraud and Abuse Act has been used (unsuccessfully, mostly, see the LinkedIn vs hiQ Labs case) to try to stop scraping of public profile data. Logging into a competitor’s platform with a fake account to access gated content, pricing, or client-only areas is where you move from research into something a court would take seriously.
Here’s the bit most articles on this topic won’t say plainly: almost every serious company scrapes its competitors’ public pages in some form, price monitoring tools, review aggregators, comparison sites, all of them are built on scraping data that technically sits behind a “no scraping” clause in someone’s terms of service. Enforcement against small and medium businesses doing this quietly and reasonably is close to nonexistent, because proving damage and justifying the legal cost rarely makes sense for anyone but the very largest platforms. That doesn’t make ignoring terms of service a good idea, but it does mean the real risk most businesses face isn’t a lawsuit, it’s getting your IP address blocked or your account flagged, which is inconvenient, not illegal.
Building this into something bigger
Once you’ve got a handle on manual competitor research, the next step for most of my clients is bringing in tools built for structured data extraction rather than doing it by hand every week. I’ve reviewed several in 7 data extraction tools that make product research a breeze, which is useful if you’re tracking product catalogues, prices, or reviews across dozens of competitor pages rather than three.
If you’re running this kind of intelligence gathering alongside AI tools for the first time, it’s worth spending a bit of time learning the basics rather than bolting AI onto a process you don’t fully understand yet. I put together a full path for that in how to learn AI from scratch in 90 days, and it’s the same approach I’d recommend to anyone trying to turn scattered competitor research into an actual repeatable system rather than a once-a-year panic.
And if you’d rather someone build the whole monitoring and analysis system for you, an AI consultant for small business can set up automated tracking across pricing, content, and traffic in a few weeks instead of you doing it manually every Monday morning for the rest of your career.
A few underused sources worth trying once
Most of what I’ve mentioned is standard for anyone who’s done competitor research before. But there are obscure corners of the internet that most people never think to check, review sites, forums, niche directories, archived press kits. I collected a batch of these in 50 websites you didn’t know existed (but really should), several of which are exactly the kind of place a competitor’s older announcements, old reviews, or forgotten product pages still sit, untouched, waiting for someone to bother looking.
Frequently asked questions
Is it legal to use tools like SimilarWeb or SEMrush to check a competitor’s traffic?
Yes. These tools estimate traffic using publicly available data, panel data, and DNS records, none of which requires accessing anything private or password-protected. This is standard, legal market research that most agencies and marketing teams use daily.
Can I get in trouble for scraping a competitor’s website?
Basic scraping of public pages, like using the Wayback Machine or a keyword tool, is not illegal. Large-scale automated scraping that ignores robots.txt, overloads their server, or breaches their published terms of service carries more legal risk, though enforcement against small businesses doing this reasonably is rare. The safest approach is to only pull publicly visible data and avoid anything that requires logging in or bypassing access controls.
What’s the fastest way to spot a competitor’s price changes?
Set up a free change-tracking tool like Visualping or Distill.io on their pricing page, or manually compare Wayback Machine snapshots month to month. Both are free, both are legal, and both will catch a pricing change within days of it happening.
Where can I find out if a competitor is expanding or hiring?
Check their careers page directly and filter LinkedIn Jobs by company name. New senior roles, especially in sales, partnerships, or a new region, are one of the most reliable public signals of where a competitor is about to invest next, and they’re publishing this information themselves for free.
Related reading: Why Your LinkedIn Account Got Restricted (and How to Fix It) and What You Can Legally Learn About a Competitor’s Marketing.
