← All work

Solo Inteligente, a land scout that reads 11 million tax records.

Central São Paulo is underpopulated: large, outdated properties keep land prices high and push residents to the periphery. The 2023 Master Plan allows higher density near transit axes. I built this pipeline to find the lots that fit the new rules before the market reprices them.

Context

Research project, land prospecting in São Paulo

Status

Research

Stack

Python, GeoPandas, pandas, NumPy, SciPy, scikit-learn, FAISS, BM25, OpenAI ada-002 and GPT-4o, ReportLab, Contextily, Matplotlib, Google Drive and Sheets APIs

What it replaces

The manual version of this job belongs to perdigueiros, scouts who comb Google Maps for lots. Solo Inteligente runs the search over data instead: more than 11 million IPTU records, the zoning shapefiles and the legal documents. For each lot it flags, it produces a georeferenced PDF viability report.

It is two Jupyter notebooks run in sequence. The first does the geospatial data engineering and writes normalized parquets. The second reads them, scores lots and calls GPT-4o with hybrid retrieval to write the report.

Code on GitHub

Sources

Six sources, normalized into a single feature space at the lot level.

INCRA / SNCR

Federal rural property registry: owner, area in hectares, location.

CAR

Cadastro Ambiental Rural. Environmental restrictions per parcel: APP, legal reserve, restricted use.

SIGEF

Certified rural properties with polygon boundaries validated by INCRA.

GeoSampa

The city's cadastre: 11M+ IPTU records with zoning, floor area ratio, use, built area and lot geometry.

Shapefiles

Zoning under Lei 18177 and Lei 16402, geological risk, watershed protection areas, flood extent.

Google Drive PDFs

Deeds, environmental reports and INCRA records, indexed in FAISS for retrieval.

Data engineering

organiza_terrenos.ipynb turns the raw layers into normalized parquets, one file per layer.

  1. Load and join

    Shapefiles load in parallel with ThreadPoolExecutor and GeoPandas. Lot geometries join the IPTU records on sector, block and lot.

  2. Zoning

    The current law (Lei 18177) and the previous one (Lei 16402) are normalized side by side, covering ZEU, ZEM, ZC, ZM, ZEIS, ZDE, ZPI, ZPR and ZER.

  3. Risk

    Overlays for flood extent at 5, 25 and 100-year return periods, geological risk and watershed restrictions.

  4. Distance

    A cKDTree from scipy.spatial over five point layers: metro, train, bus stops, terminals and UDH. Each sector-block gets the Euclidean distance in meters to the nearest point of each layer, at O(log N) per lookup.

  5. Scaling

    IPTU and UDH data merge on latitude and longitude. Area, obsolescence, owner age and the UDH features go through MinMaxScaler.

  • lotes_iptu
  • lei16
  • lei18
  • mancha_inundacao
  • manancial_completo
  • uso_predominante
  • risco_geologico

Scoring and reports

rag_terrenos.ipynb reads the parquets and writes one PDF per flagged lot.

  1. Similarity

    PCA and KNN over the normalized features surface lots that resemble high-potential parcels.

  2. Retrieval

    The legal documents are embedded with OpenAI text-embedding-ada-002 into a FAISS index searched by cosine similarity. BM25 runs next to it for better recall on Brazilian zoning terms.

  3. Generation

    GPT-4o gets the lot's structured data from SQL plus the top three FAISS excerpts (k=3) and returns a structured report.

  4. Map

    GeoPandas draws the lot polygon and its surroundings, Contextily adds an OpenStreetMap basemap, and Matplotlib and PIL render the PNG.

  5. PDF

    ReportLab lays out the A4 report with the map embedded as the site plan.

Each report has these sections:

  • three-bullet summary
  • lot data
  • zoning
  • conclusions
  • regulatory inconsistencies
  • regularization paths
  • site plan