If you run a website with traffic or a growing SaaS, you've probably already run into a quiet but costly problem: bots. We're not just talking about the classic brute-force attack, but scrapers that copy your catalog, tools that abuse your endpoints, and crawlers that inflate your cloud bill without giving anything back. In this article we'll walk you through concrete measures to control that abusive traffic and keep your infrastructure costs in check.
Why scraping and bots cost you money
Automated traffic isn't always malicious, but when it is, it hits your bottom line directly. Every request consumes resources: bandwidth, CPU time, database queries and, in many cases, calls to third-party APIs you pay for by usage.
The most common effects are:
- Inflated cloud bills from traffic spikes that don't generate any revenue.
- Degraded performance for real users during attacks.
- Stolen content and pricing that competitors reuse.
- Fraud in sign-ups and forms through fake accounts.
A well-built bot can generate the equivalent of thousands of real users' worth of traffic without you noticing — until the bill arrives.
Identify the traffic you want to block
Before rolling out technical measures, it's worth distinguishing between types of bots. Not all of them are enemies.
Good, bad and ambiguous bots
- Good: Googlebot, Bingbot and other legitimate crawlers you need for your SEO.
- Ambiguous: SEO tools, aggregators or monitors that use up resources but can still be useful.
- Bad: content scrapers, credential-stuffing bots and crawlers that ignore your
robots.txt.
Signs to spot them
Check your logs and metrics for suspicious patterns:
- Lots of requests from the same IP or range in a short period.
- Generic, empty or spoofed user-agents.
- Browsing without loading resources (CSS, JS, images).
- Sequential access to product URLs or paginated results.
- Total absence of cookies or JavaScript execution.
Practical measures to control bots and scraping
This is where we get into action. The ideal approach is to combine several layers, because no single measure is foolproof on its own.
Network and CDN layer
Your first line of defense should sit as far away from your origin server as possible, so abusive traffic doesn't even get the chance to consume your resources.
Rate limiting
Limit the number of requests per IP within a given time window. It's the measure with the best effort-to-benefit ratio. You can apply it at the CDN level, on a reverse proxy (Nginx, Cloudflare) or directly in your application.
WAF and firewall rules
A Web Application Firewall lets you block known attack patterns, filter by country or IP reputation, and apply specific rules to sensitive routes like /login or your APIs.
Application layer
When traffic makes it past the network, your application can apply finer-grained controls.
Challenges and verification
- Modern CAPTCHA (like Turnstile or hCaptcha) on sign-up and login forms, without adding unnecessary friction for legitimate users.
- JavaScript challenges that force the client to run code before content is served.
- Honeypots: hidden form fields that only bots fill in.
Authentication and data access control
Protect endpoints that return valuable data by requiring authentication, expiring tokens and per-user quotas. If your most expensive-to-generate content sits behind a login, you've already shrunk the scraping surface a lot.
Defensive backend design
Cutting the cost per request matters just as much as blocking requests outright. An efficient endpoint holds up much better under spikes.
- Cache public content aggressively with sensible TTLs.
- Paginate and cap the amount of data each response returns.
- Avoid expensive queries that lack mandatory filters.
- Set up cost and traffic alerts so you can react before things escalate.
Technique comparison
Here's a quick overview of the trade-off between effectiveness, user friction and implementation effort:
| Technique | Effectiveness | User friction | Effort |
|---|---|---|---|
| Rate limiting | High | Low | Low |
| WAF / network rules | High | Low | Medium |
| Modern CAPTCHA | Medium-High | Medium | Low |
| Honeypots | Medium | None | Low |
| Authentication + quotas | High | Medium | Medium |
| Cache + optimization | Indirect | None | Medium |
How to prioritize without losing your mind
You don't need to roll everything out at once. If you have to start somewhere, this is a sensible order:
- Rate limiting at the edge: the quickest win.
- Cost alerts and traffic monitoring to get visibility.
- Caching for the most-requested public content.
- Form protection with honeypots and CAPTCHA.
- WAF and advanced rules based on the patterns you detect.
One important tip: measure before and after every change. Without data, there's no way to know whether you're blocking bots or driving away paying customers.
Mistakes worth avoiding
- Blocking too much: aggressive rules that catch legitimate users or Googlebot and hurt your SEO.
- Relying solely on
robots.txt: malicious bots ignore it; it's a request, not a wall. - Relying only on IP: attackers rotate IPs easily using residential proxies.
- Not reviewing false positives: regularly check what you're actually blocking.
Conclusion
Protecting your website and SaaS from abusive scraping isn't about finding a silver bullet — it's about stacking layers of defense that, combined, make the attack not worth it. The good news is that many of these measures (rate limiting, cache, honeypots) are cheap to implement and pay off immediately on your infrastructure bill.
At FlowITeam we help studios, stores and platforms audit their traffic, detect bots and design architectures that withstand spikes without driving up costs. If you notice your cloud bill climbing without your real user base growing, you probably already have a bot problem worth tackling.
Frequently asked questions
Does the robots.txt file help stop abusive scraping?
Not reliably. robots.txt is a recommendation that legitimate bots like Googlebot respect, but malicious scrapers ignore it completely. It's there to guide good crawlers, not to protect you.
Can rate limiting hurt my real users?
If you set it up with sensible thresholds per IP or session, it barely affects normal use. The key is to measure your users' typical behaviour before setting the limits and to review any false positives.
How do I know if my cloud bill is going up because of bots?
Check your logs and metrics for spikes in traffic from just a handful of IPs, suspicious user-agents, and requests that don't load resources or run JavaScript. Setting up cost and traffic alerts gives you early visibility.