What it is
An automated programme, run by ShiftAi Systems Ltd, that reads public web pages. It identifies itself with this user agent:
ShiftAiResearchBot/0.1 (+https://shiftaiconsulting.co.uk/research-crawler; hello@shiftaiconsulting.co.uk)
Earlier development runs (from September 2026) used ShiftAiResearchBot/0.1 (+https://shiftaiconsulting.co.uk; hello@shiftaiconsulting.co.uk).
This is an independent project, not affiliated with or endorsed by the NHS.
What it collects
- Public web pages.
- Published board papers.
- Public procurement and transparency records.
It does not collect anything behind a login, and it does not collect from social media.
How it behaves
- It reads a site’s
robots.txtbefore fetching, and respects it. - If it cannot read or confidently interpret
robots.txt, it does not fetch. - It makes no more than one request per second to any one host.
- It follows redirects only within the same site, and applies the same checks at every step.
We make two exceptions to robots.txt, and no others. One is where a site owner has given us written permission. The other is official open-data files that their publisher documents for public reuse.
Personal data
The index records organisations, not people. Email addresses and phone numbers are redacted from the text we keep for analysis, and anything that looks like personal data in an extracted record is blocked before human review.
Original documents are held privately so that quotations can be verified. They are never published, and they are deleted after 12 months unless they back a record.
How to opt out
Add this to your robots.txt:
User-agent: ShiftAiResearchBot
Disallow: /
Or email hello@shiftaiconsulting.co.uk and we will exclude your site and remove what we hold from it.
Corrections
If something we hold about your organisation is wrong, email hello@shiftaiconsulting.co.uk with the page address and what needs correcting. We will check it against the source, then correct it or remove it.
If you are named in a document we hold, see our privacy notice for your rights, including asking us to delete it.
Contact
hello@shiftaiconsulting.co.uk
ShiftAi Systems Ltd, 71-75 Shelton Street, Covent Garden, London, WC2H 9JQ