Urban Space: pricing São Paulo property from public data.

I wanted to see whether freely available public data could produce usable price estimates at the neighborhood level. The inputs are IPTU records from the city, FipeZAP, and monthly series from BCB, IBGE and FGV.

There are three parts on one ingestion layer, each running on its own: pricing (current value plus a 60-month projection), buyer recommendation, and lead mining over public records. Results land in BigQuery or Google Sheets.

Code: github.com/Guicicca90/urban-space

VivaReal listings were scraped. Everything else is public data.

Monthly macro series from BCB, IBGE, FGV and SINDUSCON go into a single feature matrix:

• Selic
• IPCA
• IGP-M
• INCC
• IBC-BR
• IGMI-R
• IVG-R
• IIE-BR
• CubSP

The target is price per m². Property features: type, zoning class, FAR, built area, lot area, rooms, suites, bathrooms, parking, IPTU, condo fee and amenities. Census and zoning data are attached to each property through spatial joins on a cKDTree.

Ten monthly macro indicators also go in: the nine above plus FipeZAP. FipeZAP is matched to each property through a fallback chain. It tries the exact city, type and bedroom count first, then the bedroom aggregate, the residential type, the state capital and finally the national average.

The model is XGBoost with 3,000 estimators, learning rate 0.01 and L1 plus L2 regularization, trained on GPU (gpu_hist). Held-out MdAPE is 14.9%. Each estimate is stored with the macro snapshot used to compute it, and its deviation from the FipeZAP reference is flagged as a pricing gap.

The 60-month projection is a separate model: SARIMAX(1,1,1)(1,1,1,12) with nine exogenous variables (IGP-M, Selic, IPCA, the dollar, INCC, IVG-R, IPAM, IIE-BR, CubSP). I split the series by bedroom count (1, 2, 3, 4+) to match the structure of the FipeZAP index. The projected index change is applied from a November 2023 base to get a price path per property. Output goes to a Google Sheet, dbPredicao, with 60 rows per property, one per month.

The recommendation engine matches buyer intent profiles, collected through forms, against active VivaReal listings.

1. Load

   Listings and buyer forms are read from BigQuery in parallel with ThreadPoolExecutor.

2. Filter

   Hard filters on type, subtype, municipality and neighborhood run before KNN.

3. Normalize

   The 80+ Portuguese amenity strings map to a fixed vocabulary (BARBECUE_GRILL, POOL, FITNESS_ROOM, HELIPAD) through unidecode and a mapping table.

4. Match

   Continuous features are scaled with StandardScaler, then KNN runs over up to 100K candidates per query.

5. Write

   One row per buyer and property in Warehouse.Recomendacoes: buyer ID, ranked property features, similarity score and a Google Maps link.

Lead mining works on the same public records. It scans the 11M+ GeoSampa IPTU records for underpriced lots by comparing each lot's fiscal value with the pricing model's estimate, and writes a CSV with lot ID, fiscal value, estimated market value and the gap, ranked by the size of the gap.