01
Source review
Each source checked for robots.txt rules, terms, structure and how often it changes, before we quote.
FIG 01 · Web Scraping & Data Extraction
Web scraping services built on public data and clear rules
Public web data, collected politely and delivered clean: prices, listings, catalogues and public records as structured feeds, with a record of where every field came from.
See how it works
FIG 02 · Quick answer
What are web scraping services?
Web scraping services collect public data from websites and deliver it as clean, structured files or feeds. Elgrith checks for an official API first, respects robots.txt and site terms, avoids personal data unless there is a lawful basis, and delivers deduplicated JSON, CSV or database feeds with monitoring for site changes.
- Official API or license checked before any crawler
- API first
- Projects delivered since 2015
- 1,000+
- Clients across the USA, UK and India
- 250+
- Your code, repository and IP
- 100%
FIG 03 · Why it matters
Data you can defend starts with how you collect it.
From the IETF, CNIL and Cloudflare, checked 24 Oct 2026.
01
RFC 9309
Robots.txt is a standard, not a license
The Robots Exclusion Protocol became a standard in September 2022. It says its rules “are not a form of access authorization”, so respecting robots.txt is polite, and still not permission.
Source: IETF, RFC 9309 Robots Exclusion Protocol (Sep 2022)(opens in a new tab)02
19 Jun 2025
CNIL’s scraping guidance
France’s data regulator says scrapers relying on legitimate interest should limit collection to freely accessible data and exclude sites that clearly object to scraping, for example through robots.txt.
Source: CNIL, legitimate interest and web scraping (19 Jun 2025, in French)(opens in a new tab)03
1 Jul 2025
AI crawlers blocked by default
Cloudflare began blocking AI crawlers by default on new domains unless they have permission. Honest crawler identification now decides both ethics and whether a feed keeps working.
Source: Cloudflare, press release on AI crawlers (1 Jul 2025)(opens in a new tab)
FIG 04 · Our rules
How we keep scraping lawful and polite
These limits are part of every scope we write. If a source can only be collected by breaking one of them, we tell you, and we don’t collect it.
| Step | What we do | What we won’t do |
|---|---|---|
| API or license | Check for an official API, data feed or licensed dataset first | Scrape a site that offers the same data through an API |
| Robots.txt | Read and follow it by policy (RFC 9309) | Treat robots.txt as permission, or ignore it |
| Site terms | Review each source’s terms before the build | Collect from a site whose terms forbid it |
| Rate limits | Crawl slowly, off-peak where possible, and back off on errors | Load a site hard enough to slow it for its users |
| Identification | Use an honest, identified user agent with a contact route | Disguise the crawler or rotate identities to avoid blocks |
| Login walls | Collect public pages only | Sign in to collect data behind a login, except your own account data where the terms allow it |
| CAPTCHAs and blocks | Stop and review when a site blocks or challenges us | Bypass a CAPTCHA or any other access control |
| Personal data | Avoid it; collect it only with a lawful basis and a DPIA where needed | Collect personal data because it happens to be visible |
General guidance from public sources, not legal advice; your counsel confirms what applies to you.
The law in detail: is web scraping legal in 2026?FIG 05 · What you get
What our web scraping services cover
From the first source review to the feed that still runs after a site redesign. You get data you can use, and a record of where it came from.
02
API and license check
If the publisher offers an API, a feed or a licensed dataset, we use that instead of a crawler.
03
Crawler build
Polite crawlers in Scrapy or Playwright, with rate limits, retries and an identified user agent.
04
Scheduling
Hourly, daily or weekly runs, timed to the source’s update rhythm, not more often than you need.
05
Parsing
Pages turned into typed fields: prices, units, dates and categories normalized to one schema.
06
QA checks
Field-level validation and sampling against the live page, so a bad run is caught before you use it.
07
Deduplication
Repeated listings and products merged by stable keys, with history kept when values change.
08
Change monitoring
Alerts when a source changes layout or a field goes empty, and fixes under terms agreed in your written scope.
Deliverables
10
deliverables, in writing
Included in every build, written into your fixed quote before work starts.
FIG 06 · Use cases
Common use cases
Public data that teams track by hand today. Every use case still goes through the source review and the rules above.
- Public web pages
- Official APIs
- PDFs and tables
- Public records
Your data feed
- JSON or CSV
- S3 or BigQuery
- Snowflake
- Google Sheets
Price and assortment monitoring
- Data
- Public prices, stock flags and ranges on retail sites
- Use
- Pricing decisions and range gaps
Product catalogues
- Data
- Public specs, images and categories, where terms allow
- Use
- Catalogue enrichment and matching
Real-estate listings
- Data
- Public listing prices, sizes and locations, without agent personal data
- Use
- Market tracking and valuations
Job postings
- Data
- Public roles, locations and skills
- Use
- Hiring-market research
Public records and research
- Data
- Registers, filings and open datasets published for reuse
- Use
- Research datasets and due diligence
AI training data
- Data
- Only sources whose license and terms allow AI use
- Use
- Retrieval or fine-tuning datasets
FIG 07 · Extraction
Data extraction services: documents and sites
Not every source is a web page. We extract structured data from the documents and tables that publishers put online, with the same source checks.
04principles
PDF reports and filings
Tables and figures pulled from public PDFs into rows, with the page reference kept.
HTML tables
Public tables and directories turned into typed, consistent columns.
Catalogues
Product and parts catalogues normalized to one schema across many sources.
Your own documents?
Invoices, forms and scans from your own business are document processing, not scraping.
AI automation
FIG 08 · Managed feeds
Web scraping as a service: managed feeds
For data you need every day or every week, we run the crawlers for you, or in your cloud, and tell you when something changes.
04principles
Schedules
Runs on a schedule matched to how often the source updates.
Alerts
Empty fields, row-count drops and layout changes raised before you rely on the data.
Schema versioning
Field changes versioned and announced, so your reports don’t break silently.
Delivery to your warehouse
Feeds into S3, BigQuery or Snowflake, or Google Sheets and an API for smaller teams.
Data warehouse consulting
FIG 09 · Quality
Quality and delivery
Every row carries its own evidence. If someone asks where a number came from, you can show them.
04principles
Field-level validation
Types, ranges and required fields checked on every run, with failures quarantined, not delivered.
Sampling
A sample of rows compared against the live page after each change to the crawler.
Source and timestamp
The source page and collection time stored on every row.
Formats
JSON, CSV or Parquet files, or tables loaded straight into your database.
FIG 10 · Law
The laws we plan around
Scraping law turns on what you collect, how you access it and what you do with it. These are the rules we design around. General guidance from public sources, not legal advice; your counsel confirms what applies to you.
04principles
CFAA (US)
The Computer Fraud and Abuse Act concerns access without authorization. The Supreme Court’s Van Buren decision (3 Jun 2021) framed it as a “gates-up-or-down” question, which is one reason we stay out of anything behind a login or a block.
GDPR and UK GDPR
Personal data needs a lawful basis, usually legitimate interests, with a balancing test. The ICO (Dec 2024) treats legitimate interests as the only realistic basis for scraping personal data to train generative AI, and expects developers to show scraping is necessary.
DPDP Act (India)
India’s Digital Personal Data Protection Act, 2023 covers digital personal data processed in India, and abroad when it relates to offering goods or services to people in India. For Indian sources and Indian clients, we treat personal data the same way: avoid it unless there is a lawful basis.
Copyright and database rights
Facts are generally free to use, but text, images and the structure of a database can be protected, and the EU and UK have database rights. We collect facts, not content to republish, unless the license allows it.
FIG 11 · UK
Web scraping for UK teams
For UK research, retail and property teams, we work through our London client-partnership centre, with delivery from our New Delhi HQ. Data dictionaries are written in UK English, and working hours are agreed in your written scope.
03principles
UK GDPR first
We design feeds to leave personal data out. Where you need it, we record the lawful basis and help you prepare a DPIA. Not legal advice.
UK formats
GBP prices, UK postcodes and UK date formats normalised in every feed.
Your organisation owns it
Crawler code, schedules and data stay in your accounts, so your team keeps control.
FIG 12 · Next
Using the data
A feed is only useful once it reaches a report, a model or an app.
04principles
FIG 13 · Process · 6 steps
Check the source, then build the crawler
A fixed written quote after a free call, then a source review, a sample, the build and a monitored feed, with a demo every week. Your crawler code, schedules and data stay in your accounts.
- Step 01
Free call
Tell us the data, the sources and how you will use it. We ask about APIs, terms and personal data first.
- Step 02
Source review
Robots.txt, terms, API and license options and page structure checked for every source.
- Step 03
Fixed written quote
A written scope and a fixed quote: sources, fields, schedule, delivery format and the rules we follow.
- Step 04
Sample data
A sample from each source in your schema, so you can check fields and quality before the full build.
- Step 05
Build and QA
Crawlers, parsers, validation and deduplication built in sprints with a demo every week.
- Step 06
Run and monitor
Scheduled runs, change alerts and fixes, under terms agreed in your written scope.
FIG 14 · What you get
Everything included
Written into your fixed quote, so you know exactly what you are paying for before work starts.
10items
- Written scope and fixed quote
- Source review for every site
- Crawlers with rate limits and an identified user agent
- Parsers to your schema
- Validation and deduplication rules
- Source and timestamp on every row
- JSON, CSV, Parquet or database delivery
- Change monitoring and alerts
- Data dictionary and runbook
- Crawler code, schedules and data in your accounts
FIG 15 · Tech & tools
The tools behind the work
Proven, well-supported tools, chosen for your project and set up in accounts your business owns.
- Python
- Scrapy
- Playwright
- BeautifulSoup
- Pandas
- PostgreSQL
- Parquet
- Amazon S3
- BigQuery
- Snowflake
- Airflow
- Docker
- Google Sheets API
FIG 16 · Who it’s for
Public data for ecommerce, research and property teams.
Data you can trace, collected in a way you can explain.
3 audiences
Ecommerce
Prices, ranges and stock across your market.
Daily feeds of public prices and assortment from retail sites whose terms allow it, matched to your own catalogue.
- Price monitoring
- Assortment gaps
- Product matching
- Daily or weekly feeds
Research
Datasets you can cite.
Public records, filings and open datasets collected with the source and time on every row, ready for analysis or a model.
- Source on every row
- Open data first
- License checks for AI use
- Reproducible runs
UK teams
Data extraction for UK organisations.
Through our London client-partnership centre, with delivery from our New Delhi HQ. UK GDPR in the design, UK formats in the feed.
- Personal data left out
- DPIA support where needed
- GBP and UK postcodes
- Fixed written quote
FIG 17 · Working with us from the USA and UK
Built in New Delhi. Run on your hours.
Every project gets a point of contact in New York or London and an engineering team in New Delhi. Here is how today’s working days line up, live.
New York
North America
--:--ET
New Delhi team Your working day- Shared working time
- ≈ 3 hours today
- New Delhi is
- 10.5 h ahead
- Live calls, your time
- 9 a.m. to 12 p.m. ET
London
UK & Europe
--:--UK
New Delhi team Your working day- Shared working time
- ≈ 4.5 hours today
- New Delhi is
- 5.5 h ahead
- Live calls, your time
- 9 a.m. to 1:30 p.m. UK
Three ways to work together
- 01
Fixed-scope project
A written scope, one fixed price and a dated plan. Suits websites, apps and MVPs.
- 02
Dedicated team
Designers and engineers on your roadmap every month, billed monthly.
- 03
White label
For US and UK agencies: our work, your brand, unbranded deliverables.
Every engagement
- Fixed quote in USD, GBP or INR after a free 30-minute call
- NDA before discovery. You own all code, designs and content
- A working demo every sprint, not status slides
- Reply within one business day, calls inside your hours
Book a call
Calls on ET or UK hours
FIG 18 · Cost
What drives web scraping cost
We don’t publish prices. These are the things that move the cost of a scraping project, so you can decide what to collect first.
| Driver | Lower cost | Higher cost |
|---|---|---|
| Sources | A few sites with a stable layout | Many sites, each with its own structure |
| Frequency | Weekly or monthly runs | Hourly runs |
| Volume | Thousands of rows | Millions of rows with history |
| Pages | Static HTML | Pages that need a headless browser to render |
| Delivery | CSV or JSON files | Loads into your warehouse with schema versioning |
Hosting and storage costs grow with volume and frequency, and sit in your cloud account. After a free call you get a written scope and a fixed quote.
FIG 19 · Rules and dates
Web scraping rules and dates (checked 24 Oct 2026)
What changed for web scraping, and when. Checked at each source on 24 Oct 2026.
| Source | Date | What it says | What it means for your feed |
|---|---|---|---|
| IETF, RFC 9309 | Sep 2022 | Robots.txt rules are a standard, and “are not a form of access authorization” | Respect robots.txt, and check terms and law separately |
| ICO, generative AI consultation response | Dec 2024 | Legitimate interests is the only realistic basis for scraping personal data to train generative AI, if the balancing test is met | Leave personal data out of AI datasets unless you can justify it |
| CNIL, legitimate interest and web scraping | 19 Jun 2025 | Limit collection to freely accessible data; exclude sites that clearly object | Honour robots.txt and blocks as objections |
| Cloudflare press release | 1 Jul 2025 | AI crawlers blocked by default on new domains without permission | Expect blocks on AI-labelled crawlers; identify honestly |
| Cloudflare blog | 15 Sep 2026 | Separate controls for search, AI training and AI agents | Site owners can now allow search while refusing AI training |
General guidance from public sources, not legal advice; your counsel confirms what applies to you. Sources: RFC Editor; ICO response to the consultation series on generative AI; CNIL; Cloudflare press release and blog.
FIG 20 · Compare
Official API, licensed dataset or web scraping?
| Official API | Licensed dataset | Web scraping | |
|---|---|---|---|
| What it is | Data the publisher offers to developers | Data you buy or license from the owner or a data vendor | Data collected from public pages |
| Permission | Clear, in the API terms | Clear, in the license | Depends on terms, robots.txt and the law |
| Reliability | High; changes are versioned | High; refreshed on the vendor’s schedule | Breaks when the page layout changes |
| We use it when | Always, if it has the fields you need | The owner sells the data you need | No API or license covers the public data you need |
FAQ
Frequently asked questions
Straight answers about Web Scraping & Data Extraction with Elgrith.
Scraping public data is not automatically illegal in the US, UK or EU, but it can still breach a site’s terms, copyright, database rights or data-protection law. Risk rises when you log in, bypass blocks, collect personal data or reuse content to train AI. Our guide on whether web scraping is legal covers the main cases. General guidance from public sources, not legal advice; your counsel confirms what applies to you.
FIG 21 · Related
Keep exploring
Data Engineering
Pipelines, warehouses & AI-ready dataData Warehouse Consulting
Snowflake, Databricks & BigQuery buildsAPI Integration
Connect your apps, CRM & paymentsAI Automation
Workflow & process automation with AILLM Integration
ChatGPT, Claude & RAG in your productData Visualization
Dashboards, Power BI & Tableau
Free 30-minute call
Get public data you can defend.
Tell us the data, the sources and how you will use it. We check for an API or license first, and after a free call you get a written scope and a fixed quote.
Pick a day for your call
OpenPick a day. You'll choose a time in your own time zone on the next step.