# Data Sources & API Reference — Deal Sourcing / Vacant-Site Detection > **Status:** Reference companion to `DEAL_SOURCING_ENGINE.md`. > **Purpose:** For each signal the engine needs, this document states *what to pull*, *the specific source/endpoint*, *whether an API key exists and how to get it*, and the *licensing / GDPR reality*. > **Read this first:** Availability ≠ permission. Several sources below have keys that take five minutes to get, but terms that restrict the exact bulk/automated use this engine wants. The "Reality" column matters more than the "Key" column. Verify current terms before building — access models change (the EPC route changed in May 2026; see below). --- ## 1. How to read this Every source falls into one of five access patterns. Design around which one applies: | Pattern | Meaning | Example | |---|---|---| | **Open API + key** | Register, get a key, call programmatically. Commercial use permitted. | Companies House, EPC search | | **Bulk download** | Download whole files (CSV/zip), process locally. Often better than the API for engine-scale work. | Land Registry Price Paid, EPC bulk | | **Licensed / paid** | Commercial licence or subscription required; often includes the *right to run automated analysis*. | Aerial imagery, some OS/title data | | **Inference only** | No queryable feed exists; you *infer* the signal from other sources. | Empty-homes / vacancy | | **Partnership / manual** | Obtainable only via relationship or FOI; not a pipeline. | Council empty-homes officer, address-level council data | --- ## 2. Records signals (the confirming data) ### EPC — Energy Performance Certificates *Gives:* `total-floor-area` (m²), current/potential rating, construction age band, property type, tenure, lodgement date. Absence or age of a certificate is itself a vacancy/long-hold signal. - **⚠️ Access route changed May 2026.** The old `epc.opendatacommunities.org` site was **retired on 30 May 2026** and replaced by **`get-energy-performance-data.communities.gov.uk`**. Any older tutorial pointing at opendatacommunities is stale. - **Key / account:** You now sign in / create an account via **GOV.UK One Login** to download bulk certificate data. (The legacy API used HTTP Basic auth with an email + API key printed in the site footer; confirm the current auth model on the new service.) - **Two access modes:** - **Bulk download** — whole-register zip (~5–6 GB, all England & Wales), plus smaller per-county / per-postcode cuts. **This is what the engine wants** — pull once, process locally, refresh monthly. Data covers certificates from 2008/2012 onward, including expired/superseded. - **Search API** — query by postcode/address; historically paginated. Fine for on-demand single lookups, not for warehousing the whole set. - **Coverage:** ~22M of ~29M properties have a lodged certificate. Newer sales, new builds and rentals are well covered; long-held owner-occupied stock may have none (a useful long-hold signal). - **Reality / GDPR:** EPC data **contains personal data** — MHCLG has published a DPIA. If your processing is in scope of UK GDPR you become a **data controller** and must comply. Address data is processed against Royal Mail PAF®/OS AddressBase — using the *address* fields beyond the permitted notice needs a Royal Mail licence. Scotland is separate: `scottishepcregister.org.uk`. ### Land Registry Price Paid Data (PPD) *Gives:* Sold price, date, address components (PAON/SAON/street/postcode), property type, tenure (F/L), new-build flag. The price half of your £/m² baseline. - **Access:** **Bulk download**, free, under **Open Government Licence**. Complete file (~5 GB CSV/TXT, 1995→present), plus yearly and monthly files (115–230 MB). Updated monthly. Source: `gov.uk` "Price Paid Data downloads". - **Key:** None needed for bulk download. - **Reality:** OGL permits commercial use. **But** the address fields are processed against OS AddressBase / Royal Mail PAF® — OGL does *not* cover those third-party rights; using address data beyond the permitted use needs a Royal Mail licence (`address.management@royalmail.com`). This is the recurring UK gotcha: the *data* is open, the *addresses* are licensed. ### Companies House *Gives:* Company status (active/dissolved/liquidation), officers, registered office, filing history. Identifies corporate owners and — critically — dissolved/insolvent companies owning property (classic stalled-site cause). - **Access:** **Open REST API + key.** Free. Register at `developer.company-information.service.gov.uk`, create an application, get an API key. Streaming API also available for change feeds. - **Key:** Yes — free, instant, self-service. - **Reality:** Public statutory data, commercial use fine. Rate-limited (600 requests / 5 min historically). Company data is not personal-data-sensitive in the way EPC is, though officer personal details still warrant care. ### Land Registry — corporate & overseas ownership *Gives:* Which **companies** own which titles (name, type, reg number, address, price paid). - **"UK companies that own property in England & Wales"** (formerly **CCOD**) — free, account required, on `use-land-property-data.service.gov.uk`. This is the route to corporate-owner identity at the title level. - **"Non-UK companies…"** (formerly **OCOD**) — free, account required. Useful for absentee/overseas-owner signals. - **Full registered-title bulk data** — costs **£20,000 + VAT / year**. Not needed for a pilot. - **INSPIRE Index Polygons** — free, open; property boundary shapes (need GIS/PostGIS to use). Your spatial base layer. - **Key/account:** Account + licence acceptance on the Use Land & Property Data service; some datasets keyed via that service's API. ### Individual title register / lease length *Gives:* Registered owner name + correspondence address (→ absentee signal), lease term (→ short-lease false-positive filter). - **Access:** **Paid, per-document.** ~£3 per title register via HMLR, or via aggregators (e.g. PropertyData's title endpoint returns a `document_url`, billed per request, HMLR business hours 06:30–23:00). - **Reality:** Pay-per-title, so use it *after* the flag stage on shortlisted properties only — not across the whole area. This is where you resolve "is the 20%-below-comp flat actually a short-lease trap." ### Planning data *Gives:* Application history, approvals, lapsed/stalled consents. Explains hoarded plots (Image 2 case) and surfaces developer distress. - **Access:** **`planning.data.gov.uk`** — open national planning data platform, free, growing coverage. Individual LPA portals (often Idox-based) hold the fullest detail but have no uniform API — coverage and format vary by council. - **Key:** Not generally required for `planning.data.gov.uk`. - **Reality:** Open, but fragmented across ~300 LPAs for the deep detail. Treat national platform as the spine, LPA portals as manual deep-dives. ### Environment Agency — flood risk *Gives:* Flood-zone flags (relevant for Southend's estuary pockets; a £/m² depressant explainer). - **Access:** **Open API**, free, no key for core flood-risk services. `environment.data.gov.uk`. - **Reality:** Open, clean, commercial use fine. --- ## 3. Vacancy / empty-homes — INFERENCE, not a source **There is no open, address-level, queryable empty-homes API for England.** This is a hard wall, not a gap. Design accordingly. - **Why:** The underlying data lives in each council's **Council Tax** back-office (empty/unfurnished and long-term-empty flags, driving the empty-homes council-tax premium). It's personal data held for a statutory tax purpose; councils can't lawfully hand you address lists for marketing. ~300+ billing authorities, each siloed. - **What you CAN get:** - **Aggregate counts by council** — government Council Taxbase / empty-homes statistics. Use for **area weighting** (where empties concentrate), not pinpointing. - **FOI requests** — sometimes yield lists, routinely refused at address level on GDPR grounds. Not a pipeline. - **Empty Homes Officer partnership** — many councils *want* empties reoccupied and have an officer for it. A legitimate **partnership channel**, not a scrape. - **Engine design consequence:** Treat vacancy as a **composite inference** from signals you *can* get: - No recent / no EPC (dormancy) - Absentee owner (title correspondence address ≠ property address) - Long ownership, no transactions - Planning dormancy / lapsed consent - **Visual + temporal** signals (overgrowth, hoarding, unchanged across capture dates) The composite score exists *precisely because* the clean vacancy list doesn't. Council empty-homes data downgrades to an area-weighting input plus a possible partnership. - **Devolved note:** Wales (national empty-homes grant schemes) and Scotland (Scottish Empty Homes Partnership) have more central visibility, but still no open address-level API. --- ## 4. Imagery signals ### The rule **Do not build a pipeline that bulk-pulls and warehouses Google Street View / Google Maps imagery** — it breaches Google Maps Platform terms and Google enforces it. A human *viewing* Street View to check a lead is normal permitted use. Automated ingest at scale is not. This is the imagery equivalent of the portal-scraping problem: design so the engine does **not depend** on it. ### Licensed aerial (the base layer) — how it works You take a **commercial licence/subscription** with an imagery provider. It grants API/tile access **and the rights to run automated analysis and store your *derived outputs*** (scores, change flags) — while the raw imagery usually stays licensed rather than owned. Pricing is area- or usage-based (£/km² or tiered by area + refresh rate). Licensing one town for a pilot is a modest, known cost. | Provider | Strength | Key/access | |---|---|---| | **Nearmap** | High-frequency re-fly (several/yr) → enables **temporal change detection**; oblique views; explicitly targets property/insurance change-detection | Commercial subscription + API key | | **Bluesky** | High-res GB vertical aerial + oblique + specialist layers | Commercial licence + API/tile | | **Getmapping** | High-res GB vertical aerial, rolling refresh | Commercial licence + API/tile | | **Ordnance Survey (OS Data Hub)** | Aerial + mapping; some under Public Sector Geospatial Agreement | OS Data Hub account + API key; commercial use under OS licence | | **Satellite (Planet / Airbus / Google Earth Engine)** | Very high revisit; good for cleared-plot & large-change detection; too coarse for reading a single frontage | Platform account + key; analysis licence | **Why Nearmap specifically for this tool:** temporal change detection needs the *same place at two+ dates*. A re-fly provider gives "hoarded in March, still hoarded in September" (stalled-site signal) inside the licence. That temporal capability is worth more here than raw resolution. ### Street-level (the confirming look) | Source | Notes | |---|---| | **Google Street View Static API** | The *permitted* way to use Street View: call with lat/long, get a static image, pay per request. **Usage restrictions apply** — limited/no caching, no building a competing dataset, display within Google terms. Fine for human-triggered "show me this address." **Not** a licence to bulk-ingest a whole town and warehouse tiles. Read Maps Platform terms before designing storage. | | **Mapillary** (Meta) | Crowd-sourced street-level imagery under a **far more permissive licence** than Google, with an API. Coverage patchier than Google but **genuinely usable in a pipeline**. Test coverage in your target areas — this is often the right pipeline choice for street-level. | ### Vision model The only genuinely new component vs the deal engine. Start with a **prompted general vision model** (rubric: overgrowth obscuring windows? temporary hoarding? cleared/bare plot? boarded windows? scaffolding/tarpaulin?) rather than a custom-trained model. Refine from observed weaknesses; specialise later if needed. --- ## 5. Shortcut — commercial aggregators (pre-joined data) The hardest engineering in §2 is the **Land Registry ↔ EPC join on UPRN**. Several commercial APIs have already done it and sell it behind one key. For many builders this is faster than stitching five gov sources by hand — worth pricing against build time before committing to DIY. | Aggregator | Combines | Access | |---|---|---| | **Homedata** (`homedata.co.uk`) | Land Registry PPD + EPC + EA flood + VOA council tax + OS UPRN + planning.data.gov.uk, normalised per-UPRN JSON; ~29M properties | Free tier, instant API key, no card; paid tiers for volume | | **PropertyData** (`propertydata.co.uk`) | Land Registry + EPC-derived floor areas + proprietary quoting-rent estimates + live listing snapshots + per-title documents | Subscription + API key; per-document title billing | **Note on the rent-data wall:** these aggregators offer *estimated/quoting* rents (e.g. PropertyData derives from VOA rateable values + floor area). That partly addresses the "no Land Registry for rents" problem flagged in the engine spec — but it's an **estimate**, not observed lettings data. Treat yield outputs built on it as modelled, with a confidence flag. --- ## 6. Cross-cutting licensing & GDPR checklist Before going live, confirm each: 1. **Address data (Royal Mail PAF® / OS AddressBase).** Recurs across EPC and Land Registry. The *data* is often open; the *address fields* are licensed. Confirm your use is within the permitted notice or get a Royal Mail licence. 2. **EPC = personal data.** You are likely a **data controller**. Do a DPIA. MHCLG's own DPIA is the model. 3. **Vacancy targeting = direct marketing against personal data.** Approaching an identified owner needs a **lawful basis** under UK GDPR/PECR. Empty homes are sometimes empty for bereavement/care reasons — reputationally as well as legally exposed. Human-in-the-loop before any contact. 4. **Google imagery.** Street View Static API within its terms only; no bulk warehousing. Prefer licensed aerial + Mapillary for anything automated. 5. **Portal listings (Rightmove/Zoopla/OTM).** Scraping breaches their T&Cs. Keep the engine's core independent of it (see engine spec §4). 6. **Third-party analysis rights.** When licensing aerial, confirm the licence explicitly covers *automated analysis* and *storage of derived outputs*, not just display. --- ## 7. Quick-start priority for a pilot Get these first (all free or free-tier, all permitted for the intended use): 1. **Companies House API key** — instant, free. Corporate + dissolved-owner signals. 2. **Land Registry Price Paid bulk download** — free, OGL. Baseline price data. 3. **EPC bulk download** via `get-energy-performance-data.communities.gov.uk` (GOV.UK One Login) — floor areas + dormancy signal. 4. **planning.data.gov.uk** + **EA flood API** — free context. 5. **INSPIRE polygons** — free spatial base for PostGIS. 6. *Then decide:* DIY the LR↔EPC join, or start on a **Homedata/PropertyData free tier** to prove the funnel before investing engineering. 7. **Imagery last** — pilot one town on **Mapillary** (free-ish, pipeline-legal) + a trial of one aerial provider before committing to a paid area licence. --- *This document is a technical reference, not legal advice. Confirm current access terms with each provider before building — government access routes change (the EPC route changed in May 2026). Take professional advice on UK GDPR/PECR, Royal Mail/OS address licensing, and imagery-provider terms before going live.*