← All work

bquant, the data layer under my research.

My Python toolkit for Brazilian macro, real estate and urban data. It pulls public series from BCB, IBGE, FGV, FipeZAP, RAIS, GeoSampa and IPEA, tests them before anything gets modeled, and writes parquet files that two of my other projects read.

Context

Personal research, private code

Status

Personal research

Stack

Python, pandas, numpy, statsmodels, scipy (cKDTree), scikit-learn, geopandas, BigQuery, openpyxl, requests

Macro series

The econ/ module scrapes Brazilian macro and real estate series from BCB, FGV, IBGE and B3. economia_pipeline.py orchestrates it through two classes, DadosEconomicos and DadosFipezap, and exports everything to parquet.

Rates and activity

Selic, the policy rate, and IBC-Br as a GDP proxy.

Inflation

IPCA as the general CPI baseline, IGP-M for rent inflation, INCC for construction costs.

Real estate

IGMI-R for real estate returns by state, IVG-R for collateral value.

Uncertainty

IIE-Br, FGV's economic uncertainty index.

FipeZAP publishes price per m² by city, property type and bedroom count, spread over 50+ Excel sheets and 300+ cities. I normalize it into 49 columns covering sale and rent, residential and commercial, with monthly and 12-month variation.

When a breakdown is missing, the empty cell would reach the downstream models as a silent NaN. So each lookup falls back in a fixed order: exact city, type and bedrooms → same city, all bedrooms → residential type → state capital → national average.

The module writes three files:

  • economia.parquet
  • fipezap_latest.parquet
  • fipezap_sp_ts.parquet

Tests before models

TimeSeriesEDA, in timeseries_eda.py, runs the same diagnostics on every series before any model sees it. Each test has an explicit null hypothesis and a rejection criterion. The available transforms are log1p and first difference, and a non-stationary series never reaches a model untransformed.

ADF + KPSS

Run together, ADF with autolag='AIC'. ADF's null is a unit root, KPSS's null is stationarity. When the two disagree, I flag the series as a structural break candidate.

ACF / PACF

The autocorrelation structure sets the ARIMA (p, d, q) orders before fitting, which avoids overfitting through a search over orders.

ARCH-LM

Lagrange multiplier test for ARCH effects, via statsmodels het_arch. Series with volatility clustering are flagged for separate variance modeling, GARCH in the case of asset returns.

Rolling 6-month stats

Rolling mean and standard deviation to catch regime changes, such as IGP-M and Selic diverging before and after rate cycles.

Urban data

The geo/ module runs eight processors in sequence through pipeline_geo.py and writes 14 parquet tables. IPTU, postal codes and police districts cover São Paulo. RAIS, municipal HDI, population and establishments cover all of Brazil.

The CEP (postal code) is the join key between property data and every urban indicator. Each CEP is geocoded and matched to the nearest UDH centroid with scipy.spatial.cKDTree. UDHs are sub-municipal human development units published by ONU and IPEA. The tree answers each nearest-neighbor query in O(log N), which replaced brute-force polygon intersection for more than 300k CEP records.

IPTU São Paulo

Ten years of GeoSampa IPTU files (IPTU_*.csv): the fiscal value history of every property in the city.

RAIS

Formal labor market, 2005–2021, in three series: general, industry, services. Median age is interpolated from the age brackets.

HDI and UDH

Municipal HDI, the sub-municipal UDHs and population by district. Every neighborhood-level join goes through the UDH.

Establishments

Density and sector mix of formal businesses by municipality.

CEPs

Geocoded postal codes, matched to UDH centroids.

Police districts

District perimeters and crime occurrence rates.

RAIS reports schooling in bands. I encode them from 0 (ANALF, illiterate) to 8 (ESCOMP, complete higher education) and take a weighted average per UDH. The result, Score_Edu, is one of the socioeconomic features the downstream models use.

Composite geo scores go through scikit-learn: MinMaxScaler to normalize the features, then PCA to reduce dimensions.

Some of the output tables:

  • estab_mun
  • pop_brasil
  • ipea
  • onu
  • iptu_sp
  • idh_udh
  • pop_distrito_sp
  • rais_geral
  • rais_ind
  • rais_serv
  • delegacias
  • ceps

Where the data goes

bquant fits no predictive models. Its output is parquet files on disk, and my other projects read them directly, with no API in between.

Urban Space reads the FipeZAP output and the geo tables to fit a hedonic pricing model for São Paulo residential property. It tests whether location, labor-market quality and HDI predict price per m² better than metro proximity alone. The police district data enters that model as a safety feature.

Solo Inteligente ingests the same parquet files into a RAG pipeline for land prospecting.