# Brookline Land Value Map & New-Growth Methodology

This documents the data sources, reconciliation logic, and classification
rules behind two related pieces of work in this folder:

1. **The land value map** (`site/index.html`) - a current-year (FY2026)
   parcel-level visualization of assessed land value, inspired by
   [Civic Mapper](https://www.civicmapper.org/).
2. **The new-growth / renovation classification** (`scripts/classify.py`) -
   a year-over-year analysis of Brookline's assessed value growth, splitting
   it into new construction, renovation, and ordinary market appreciation,
   for the town-finances whitepaper work in `../README.md`.

Both draw on the same underlying parcel data, so the data-source and
reconciliation sections below apply to both.

---

## 1. Data sources

| Source | File | What it provides | Vintage |
|---|---|---|---|
| Town of Brookline GIS (via MassGIS Data Hub) | `data/gis/brookline_parcels_2026-07-10.geojson` | Parcel polygon geometry, `LAND_VALUE`/`BLDG_VALUE`/`TOT_VALUE` split, owner, address | Data updated 2026-07-10; downloaded via the live FeatureServer (`services1.arcgis.com/Oknk0tvfHOElpgGU/.../Parcels/FeatureServer/0`), reprojected to EPSG:4326, 8,510 features |
| Town Assessor CAMA export | `data/2026-04 Brookline assessments consolidated.xlsx` | Full building-characteristic detail (units, living area, year built, grade/condition) for FY17, 18, 19, 21, 22, 23 | ~17,700-17,950 records/year |
| Town Assessor CAMA export | `data/FY2020%20PROPERTY%20ASSESSMENTS_...xlsx` | Same schema, fills the FY20 gap | 17,865 records |
| Town Assessor CAMA export | `data/2026-01 Brookline assessment data analysis.xlsx` (`FY24-ACTUAL` tab) | Same schema, FY24 | 18,007 records |
| Town Assessor CAMA export | `data/FY2025 2ROPERTY ASSESSMENTS.xlsx` (`FY25-ACTUAL` tab) | Same schema, FY25 (added 2026-08-30) | 18,041 records |
| Town Assessor, lighter extract | `2026-01 Brookline assessment data analysis.xlsx`, `FY26 actual` tab | Total value, unit count, land use code only - **no living area / year built / grade** | 18,081 records |
| Town Assessor, newer FY26 extract (added 2026-08-30, not yet used) | `data/FY2026 Property Assessments_data update...xlsx` | Total value, unit count, land use code, **year built, total finished area** - improvement over the lighter extract above, but still missing `DESCRIPTION`/`GRADE`/`CONDITION`/res-com value split | 18,082 records |
| Town Assessor + GIS reconciliation | `2026-01 Brookline assessment data analysis.xlsx`, `all years data` tab | Ben's hand-built crosswalk between CAMA parcel IDs and GIS parcel IDs, with 41 documented exceptions (parcel combines, renumbering) | 7,830 rows |
| MA Dept. of Revenue | `new growth data` tab in the consolidated workbook | Official certified New Growth Value by fiscal year, split into Residential and CIP (commercial/industrial/personal) | FY2003-FY2026 |

FY25 per-parcel data was obtained 2026-08-30 (see Section 3c) - the
classification pipeline now covers FY17-25. FY26 still only has extract-
level data (see the two FY26 rows above), so it can't be used for the
structural classification rules below - only for current-value totals (e.g.
the map) and unit/description counts.

---

## 2. Parcel ID reconciliation

Three different ID schemes appear across these sources:

- **CAMA `PARCEL-ID`**: e.g. `001-13-01` (block-lot-subunit). Condo units get
  separate rows sharing the same block-lot prefix (`001-13-01` through
  `001-13-09` for a 9-unit building at lot 13).
- **GIS `PARCELID`**: e.g. `001-13-00` - one polygon per physical lot. For
  condo buildings this "master" record carries `LAND_VALUE = BLDG_VALUE =
  TOT_VALUE = 0`; the real value lives only in the CAMA unit-level rows.
- **`block_lot`** (this project's canonical key): CAMA/GIS `PARCEL-ID` with
  the trailing `-SUB` segment stripped (e.g. `001-13`), used to join and
  aggregate both sources.

**Crosswalk**: Ben's `all years data` tab resolves 41 known exceptions where
the assessor and GIS disagree on parcel boundaries (e.g. GIS combines what
the assessor tracks as 3 separate lots; assessor lists 4 buildings A-D that
GIS treats as one parcel). All scripts apply this crosswalk (`raw_block_lot
-> canonical_block_lot`) before aggregating, so these known cases don't show
up as spurious "parcel appeared/disappeared" events in the year-over-year
comparison.

`scripts/build_history.py` and `scripts/build_fy26_dataset.py` both
implement this join; `output/parcel_history_fy17_25.csv` is the result for
FY17-25, aggregated to one row per `block_lot` per year (condo units summed).

**Validation**: joining FY26 CAMA data to the GIS layer via this crosswalk
matched 8,477 of 8,480 GIS parcels, and where GIS carries a nonzero value it
agrees with the CAMA total to within 0.13% (median). This is the basis for
trusting the crosswalk for both the map and the classification work.

---

## 3. Known data limitations

### 3a. No land/building split for ~983 parcels (~24% of assessed value)

The GIS layer's `LAND_VALUE`/`BLDG_VALUE` are genuinely zero (not missing -
actually zero) for most condo-classified parcels, because condo unit values
live only in the CAMA extract, which has **no land/building split at all**
(only combined `RESTOTLVAL`). Quantified: 983 parcels, ~$9.98B in CAMA-side
assessed value (~24% of the town total).

**Decision (2026-08-27, confirmed with Ben): exclude these from the land
value map** rather than estimate or substitute total value. They're flagged
`has_land_split: false` in `output/brookline_land_value_fy26.geojson` and
rendered flat/gray/toggleable in the map, with an on-page caveat, rather than
silently shown as $0. Not fixed - a real gap if a more complete condo-level
land value source is ever obtained (e.g. from the Assessor's internal
system, which may compute this even though it isn't exported to GIS).

### 3b. No square-footage data for commercial parcels

Checked directly: 100% of commercial-valued FY23 parcels have both
`LIVING-AREA` and `FINSH-AREA-1` null. The CAMA extract's detailed
building-characteristic columns appear to be residential-only. This means
**no structural commercial-expansion signal exists in this data** - see
Section 5 for how the classification methodology works around this.

### 3c. Fiscal year gaps

FY20 was missing initially, added 2026-08-27 (see Section 1). **FY25
resolved 2026-08-30**: Ben obtained the full Town Assessor CAMA export
(`data/FY2025 2ROPERTY ASSESSMENTS.xlsx`, sheet `FY25-ACTUAL`) - identical
175-column schema to FY17/18/19/21/22/23/24, confirmed by exact header
match (all required columns present, `build_history.py` printed no
"missing columns" warning on load). The full pipeline (`build_history.py`
-> `classify.py` -> `detect_growth_outliers.py` -> `build_timeline_
dataset.py` -> `build_growth_map_dataset.py` -> `bedrooms_by_new_unit.py`
-> `split_new_growth_by_type.py`) now covers **FY17-25** (8 consecutive
single-year comparisons). Output files renamed accordingly (`_fy17_24` ->
`_fy17_25`, `_fy18_24` -> `_fy18_25`).

**GIS matching confirmed clean**: FY2025's addition didn't change the
GIS-join match rate at all - `build_timeline_dataset.py` matches 8,477 of
8,480 GIS parcels for both FY2024 and FY2025 (identical to prior years'
match rate). The existing hand-built parcel-ID crosswalk (Section 2)
applies to FY2025 data without modification - no new remaps needed.

**FY26 still not attempted** - Ben supplied a newer FY2026 extract
(`data/FY2026 Property Assessments_data update...xlsx`) at the same time,
but it's a different, narrower 33-column schema: an improvement over the
old "lighter extract" (now has `Year Built` and `Total Finished Area`,
missing before), but still lacks `DESCRIPTION`, `GRADE`, `CONDITION`, and
the `RESTOTLVAL`/`COMTOTLVAL` split `build_history.py`'s `COLS` list
requires. Left out of the pipeline for now (Ben's call, 2026-08-30) rather
than building a partial-schema adapter.

**Res->commercial conversion edge case, root-caused and fixed same day**:
parcel `083-03` (FY2024->2025) tripped the `became_commercial` new-
construction rule (`com_value` 0 -> positive) while actually *losing* 3
residential units and *dropping* ~$1.3M in *net* value - a genuine
teardown-and-rebuild-as-different-use, but the rule's netted `tot_value`
delta made it look like negative-dollar "new construction." Ben's
diagnosis: this is really **two events on one parcel** - a residential
loss and a commercial gain - not one netted figure, exactly like a
demolished residential lot and a separately-built commercial lot would be
two events. Fixed in `classify.py` (`became_commercial_conversion` /
`res_units_decreased_conversion` reasons): when `became_commercial`
triggers **and** `res_units` simultaneously drops (a second,
corroborating physical signal), decompose into two rows using
`res_value`/`com_value` deltas separately (already tracked per-parcel)
instead of the netted total. **Deliberately scoped narrow** - checked
first: 71 parcel-year-pairs FY17-25 show opposite-signed res/com value
swings >$50k, but only 2 also have `res_units` actually dropping
(`083-03` and `171-48`, FY2022). The other ~69 are the assessor
reallocating value between res/com buckets on buildings that didn't
physically change (units and description both unchanged, or descriptions
just reordered - e.g. "COMM/RES" -> "RES/COMM") - decomposing those too
would manufacture phantom construction/demolition events, not fix a real
gap. Effect: FY2025 commercial new construction $-1,296,100 -> **+$2,038,700**
(right sign now); FY2022 $795,600 -> **$2,826,400** (the netting bug was
also *silently undercounting* real commercial construction whenever a
demolition happened to be smaller than the same-parcel commercial gain -
`171-48` had been quietly wrong since FY2022, just invisibly so since the
net stayed positive).

**Result: `RES_NEW_SHARE_OF_RES` and `COM_NEW_SHARE_OF_COM` both updated
in Phase 2** (`new_growth_scenario_forecast.py`, 2026-08-30). Residential:
0.209 -> **0.2867** (robust - each year's own detection rate supports it,
unaffected by the conversion fix). Commercial: first computed as an
untrustworthy 0.013 -> 0.0013 (before the conversion fix, driven entirely
by `083-03`'s negative artifact); after the fix, the real trailing-5yr
rate is **0.013 -> 0.0220** - *higher* than the original figure, not
lower, since the bug had been silently netting away real commercial
construction value. See `.claude/memorialized.md`, 2026-08-30, for the
full per-year numbers and reasoning.

---

## 4. Land value map methodology (`scripts/build_fy26_dataset.py`)

1. Load the GIS parcel layer and FY26 CAMA data (`FY26 actual` tab, which
   includes Ben's precomputed `parcel ID to match GIS` crosswalk column).
2. Aggregate CAMA `FY2026VALUE` by that crosswalk key (sums condo units to
   the parent block-lot).
3. Join to GIS parcels on `block_lot`. Compute `has_land_split`: false when
   GIS `LAND_VALUE` and `BLDG_VALUE` are both zero/missing *and* the CAMA
   side shows real value (i.e. a genuine data gap, not a legitimately
   worthless parcel).
4. `land_value_per_sqft = LAND_VALUE / TOT_LND_AREA`, only where
   `has_land_split` is true.
5. Export `output/brookline_land_value_fy26.geojson` with `null` (not `NaN`)
   for missing values - `json.dump(..., allow_nan=False)` enforces this,
   since Python's default `json` module happily writes non-standard `NaN`
   literals that JavaScript's `fetch().json()` silently rejects as invalid
   JSON with no useful error message. Learned this the hard way building the
   map - worth keeping the `allow_nan=False` guard in any future export.

**Map rendering** (`site/index.html`): MapLibre GL JS, plain OpenStreetMap
raster tiles (CARTO's free-tier vector *and* raster basemaps now require an
API key - confirmed by inspecting actual tile bytes, since they return
HTTP 200 with a watermarked "API key required" placeholder image rather than
an error), viridis quantile color ramp on `land_value_per_sqft`, 3D
`fill-extrusion` height scaled by the same metric, height/opacity sliders,
and a toggle to show/hide the `has_land_split: false` parcels (rendered
flat gray when shown).

**Not yet decided**: the OSM light basemap looks washed out against the dark
panel. A proper dark basemap needs a free-tier API key (MapTiler or Stadia)
that hasn't been set up yet.

---

## 5. New-growth / renovation classification (`scripts/classify.py`)

### 5a. Definitions (per Ben's whitepaper framework)

New Growth (MA DOR's statutory category) = **New Construction + Renovation**,
both distinct from ordinary market reassessment (which DOR's own methodology
already excludes from "new growth"). The dividing line between the two
differs by property type:

|  | Renovation ("invest in existing property") | New Construction ("create new property") |
|---|---|---|
| **Residential** | Increases value **without adding units** - deep energy retrofit, finished basement/attic, complete rebuild, 2-family → 2 condos conversion | Adds housing **units** - new construction, ADUs |
| **Commercial** | Increases value **without adding square footage** - renovations, new furniture/equipment (CIP "personal property") | Adds **square footage** - new construction |

Critically: a residential expansion that grows living area substantially
*without* adding a unit (e.g. a full rebuild that doubles square footage but
stays a single-family home) is **renovation**, not new construction, under
this framework - even though an earlier pass of this methodology got that
wrong (see Section 5d).

### 5b. Classification rules (in priority order, per year-over-year pair)

0. **Res->commercial conversion** (checked first, added 2026-08-30): if
   `COMTOTLVAL` goes from 0 to positive (a `became_commercial` trigger)
   **and** `res_units` simultaneously drops, don't treat it as one netted
   event - emit two: a commercial `new_construction` event
   (`became_commercial_conversion`, valued at the *commercial* value gain
   only) and a residential `unclassified` event
   (`res_units_decreased_conversion`, valued at the *residential* value
   loss only), using `res_value`/`com_value` separately rather than
   `tot_value`. Scoped to require both signals together (not just the
   value-split shifting) - see Section 3c for why the broader pattern
   (~70 other cases) is mostly reclassification noise, not real
   conversions.
1. **New construction - new/reclassified parcel**: block-lot exists in year 2
   only (`new_residential_parcel` / `new_commercial_parcel`), or an existing
   block-lot's `COMTOTLVAL` went from 0 to positive (`became_commercial`) -
   without a simultaneous unit drop (see rule 0 above).
2. **New construction - unit increase**: `res_units` increased
   (`res_units_increased`).
3. **Identified renovation - structural signal, no unit increase**:
   - record count changed (condo split/merge) - `record_count_changed_condo_split_or_merge`
   - living area grew - `living_area_grew_no_unit_increase`
   - description changed - `description_changed_no_unit_increase`
   - year built changed - `year_built_changed`
4. **Already-commercial parcels with no matching rule**: routed to the
   outlier check below rather than assumed market noise, since we can't
   structurally distinguish commercial renovation from ordinary
   reassessment without square-footage data.
5. **Outlier check** (everything not matched above): flag the top 1% of
   `|% value change|` per year-pair as `unclassified_large` (a review list,
   not a new-growth category); everything else is `market_adjustment`.

`res_units` **decreasing** is tracked separately as `unclassified` (rare,
worth a manual look - could be a data entry correction or an actual unit
removal).

### 5c. The renovation residual (Ben's approach, confirmed 2026-08-27)

Bottom-up renovation detection is fundamentally limited by what's in this
data (see 3b) - confirmed by reconciling against DOR's certified totals.
Rather than trying to detect every renovation event, the aggregate dollar
figure uses DOR's own certified total as ground truth:

```
Renovation ($) = DOR New Growth Total - Identified New Construction ($)
```

computed **separately for Residential and CIP**, since detection reliability
differs sharply between them (commercial detection is ~3% of DOR's CIP
total; residential detection runs ~15-50%, see results below). The
`identified_renovation` list from rule 3 above is a *named, real subset* of
this residual - useful for case studies and spot-checks - but should not be
mistaken for the full renovation total, and the residual should not be
mistaken for a precise renovation estimate: **it also silently absorbs any
real new construction the classifier failed to detect** (e.g. a commercial
expansion, since there's no sqft data to catch it). Treat the residual as an
upper bound on renovation's share, not a clean measurement.

### 5d. Calibration and validation performed

- **Living-area growth threshold**: before using *any* growth as a
  renovation signal, checked how common it is among structurally-unchanged
  parcels (`scripts/calibrate_living_area.py`). Result: a sharp cliff, not a
  continuum - 99% of stable-structure parcel-year-pairs show exactly 0%
  living-area growth; only 0.29% exceed 10%, and every top example is a
  large, real-looking jump (100+ sqft), not measurement noise. This
  validated using *any* positive growth (not just >10%) as a clean signal
  once it was moved to the renovation bucket (see below) rather than the
  new-construction bucket.
- **First-pass error, corrected**: an earlier version of this classifier
  counted living-area growth >10% *and* condo splits as **new
  construction**. This was wrong per Ben's own definitions (Section 5a) -
  both belong in the renovation bucket, since neither adds units. Fixed
  2026-08-27; the dollar impact was small (~$10.5M of ~$447M total identified
  new construction across all years) but conceptually this was a real bug.
- **Large-unclassified threshold, corrected**: the first version used a
  fixed $100k-and-15% cutoff, which flagged 3,086 parcel-year-pairs
  ($2.8B) - mostly ordinary reassessment swings on expensive single-family
  homes (median flagged delta: $382,900, a plausible one-year revaluation
  bump in this market, not a mystery). Replaced with a per-year-pair
  percentile outlier test (top 1% of unexplained `|% change|`), which
  reduced the flagged set to 543 pairs ($1.12B) - large enough to still be a
  meaningful QA list, small enough to be reviewable.
- **DOR reconciliation, corrected**: the FY19→FY21 comparison (before FY20
  was added) was being checked against only FY2021's single-year DOR figure
  instead of FY2020+FY2021 combined, producing an impossible >100% detection
  rate. Now moot - FY20 data fills the gap, so all 7 year-pairs are clean
  single-year comparisons.

### 5e. Results (FY17-24, all single-year comparisons)

Updated 2026-08-27 to include the demo-rebuild lineage correction (Section
5f) - a handful of full-rebuild-same-unit-count events that were previously
inflating the new-construction total moved to identified renovation instead:

| y2 (reporting year) | Identified new construction (res.) | DOR residential total | Renovation residual | Detection rate |
|---|---|---|---|---|
| 2018 | $41.4M | $166.6M | $125.2M | 25% |
| 2019 | $32.6M | $171.0M | $138.3M | 19% |
| 2020 | $20.6M | $193.4M | $172.8M | 11% |
| 2021 | $88.8M | $189.6M | $100.9M | 47% |
| 2022 | $42.9M | $191.5M | $148.6M | 22% |
| 2023 | $49.1M | $183.3M | $134.2M | 27% |
| 2024 | $57.8M | $267.5M | $209.6M | 22% |
| 2025 | $66.3M | $257.8M | $191.5M | 26% |

(FY2025 row added 2026-08-30, once the full FY25 assessor CAMA export arrived.)

The FY2021 spike (47% detection, +932 housing units that year - see
`output/unit_and_parcel_growth.csv`) is now traced. **Resolved 2026-08-28**:
it is not one large development, and is mostly *not* real construction.

Parcel-level investigation (`classification_fy17_25.csv` filtered to
`y2=2021, category=new_construction`, sorted by unit delta) found the unit
count is dominated by six parcels, cross-checked against news/permit/BHA
records:

| Parcel | Address | Owner | Units added | Value delta | Status |
|---|---|---|---|---|---|
| 078-08 | 370 Harvard St | Congregation Kehillath Israel | +62 | +$4.27M | **Confirmed real**: Brown Family House, 62-unit affordable senior housing built by 2Life Communities in partnership with the synagogue next door, groundbreaking Sept. 2019, completed/occupied 2020 - matches the FY2021 assessor jump exactly. |
| 137-03 | 55 Village Way | The Village at Brookline LP | +185 | -$61.8K | **Likely data artifact**: a 40+ year-old (c.1980) WinnCompanies mixed-income complex with no documented 2020-21 construction event. |
| 301-28 | 186 Chestnut St | Brookline Housing Authority | +153 | -$974.6K | **Likely data artifact**: part of BHA's High Street Veterans development (built 1950); BHA's own 2020 minutes show only minor capital work (doors/locks) here, not a redevelopment. BHA's real redevelopment of this portfolio (Sussman House, Walnut/High, 32 Marion) is bond-financed and dated 2023-2025, not 2020-21. |
| 020-01 | 338 St Paul St | Brookline Housing Authority | +83 | -$562.9K | **Likely data artifact**: part of BHA's Egmont Street Veterans development; 2020 minutes show only a bathroom-fan/window renovation (first 8 units occupied Jan 2020), not a unit-count-changing event. |
| 294-01 | 4 Walnut St | Brookline Housing Authority | +32 | -$97.7K | **Likely data artifact**: BHA's actual Walnut/High redevelopment (demolishing and rebuilding as ~96 passive-house units) is a 2024-2025 project per BHA's own environmental review and newsletters - not completed in FY2021. |
| 353-01 | 50 Goddard Ave | Hellenic College Inc | +100 | +$1.80M | **Unconfirmed**: no reliable source found for a 2019-2021 dorm/construction event; the school was in financial distress (NECHE probation, Jan 2020) around this time, making a self-funded ~100-unit addition less plausible on its face. Do not treat as confirmed construction. |

**Interpretation**: only the Harvard St/Brown Family House event (62 units,
+$4.27M) is a documented real construction event landing in FY2021. The
three Brookline Housing Authority parcels (+268 units combined, all
*negative* value delta since they're tax-exempt) show no matching
construction record in that fiscal year - their real redevelopments are
2023-2025 - so the FY2021 jump in their `res_units` field is most likely
the assessor's CAMA database populating a units count for these parcels for
the first time, not a real change in housing stock. Same likely explanation
for Village at Brookline (+185, long-established 1980 complex). The
Hellenic College entry is unconfirmed either way.

**Practical takeaway for the classifier**: `res_units_increased` is not a
reliable new-construction signal for tax-exempt/institutional ownership
classes (`AUTHORITIES`, `AFF-MULTI-UNITS`, `PSCHOOL`, `TOWN OWNED`) - it can
reflect a data-population event rather than new construction. This doesn't
change the Phase 1 revenue forecast (that uses DOR-certified levy-limit new
growth dollars directly, not this parcel classifier's unit counts), but it
is a real precision gap in this land-value-map classifier worth fixing
before leaning on it for Phase 2 new-growth-rate scenario work: consider
excluding or separately flagging these ownership classes in `classify.py`
rather than trusting `res_units_increased` at face value for them. Not yet
fixed in code - flagging as an open item.

Commercial detection is near-zero throughout (~3% of DOR's CIP total
identified, ~$11M of ~$360M across all years) - expected given 3b, not a
bug. Commercial parcel *count* fell every year (416 → 394, FY17-24) while
total commercial *value* grew ($1.96B → $2.57B, +31%) - consistent with
consolidation (fewer, larger commercial properties) rather than commercial
decline, worth stating with that nuance if this shows up in any public
writeup.

### 5f. Demolition → vacant → rebuilt lineage (built 2026-08-27)

A parcel that demolishes, sits vacant for 1+ years, then gets rebuilt
produces two misleading pairwise events under the rules above: a value drop
into vacancy (harmless - lands in `unclassified` via `res_units_decreased`,
already excluded from new construction) and a value jump back out that looks
exactly like new construction if units return to their original count
(**wrong** - no unit was added net of the full cycle, it's the same building
rebuilt, which per Section 5a is renovation, not new construction).

Fixed in `classify.py` by scanning each parcel's full FY17-24 timeline (not
just adjacent-year pairs) for a run of 1+ years with `description` in
`{VACANT LAND, RES-UNDEV, COMM-UNDEV, POTENTL DEV}`, bounded by non-vacant
years before and after. Where found, the pairwise events spanning the dip
are suppressed and replaced with **one consolidated event** classified on
net unit change from the last pre-demolition year to the first post-rebuild
year: units increased → new construction, same unit count → identified
renovation (`demo_rebuild_same_unit_count`), units decreased → left
unclassified for manual review. For the DOR reconciliation table, a
multi-year lineage event is attributed to its *ending* fiscal year (`y2`),
matching how DOR would recognize the completed rebuild in a single roll
year rather than smearing it across the span.

**Result**: found 33 demo-rebuild cycles; 22 had the same unit count
before/after (correctly moved from new-construction to identified-renovation
- a real precision improvement, not just a rounding change) and were
previously inflating the new-construction total. Detection rates by year
shifted modestly (e.g. FY2019 dropped from 26% to 19%) since some real
"new construction" events were actually rebuilds.

### 5g. Three different "events," three different timings (noted 2026-08-28)

The FY2021 investigation (Section 5e) surfaced a distinction worth stating
explicitly, since this classifier - and Phase 2's new-growth scenario work
built on top of it - can otherwise silently conflate three different
signals that don't happen in the same fiscal year:

1. **Parcel/category record change** - a new `block_lot` appears, or a
   `description`/class code changes (e.g. `VACANT LAND` → `SINGLE FAMILY`,
   or the assessor populates a previously-blank `res_units` field). This is
   what `classify.py` actually detects. As Section 5e found, this can fire
   with **no real construction behind it at all** (the three BHA parcels).
2. **Assessed value meaningfully changes** - the dollar signal DOR's
   "new growth" figure is built from. Under Massachusetts assessing
   practice, property under construction is valued at its percent-complete
   as of each January 1 - so a single project's value increase is often
   **spread across 2-3 fiscal years**, not recognized all at once.
3. **Tax/levy meaningfully changes** - DOR's "new growth applied to the
   levy limit," certified per fiscal year (this is the figure Phase 1's
   forecast and the statewide percentile analysis both use directly). By
   definition this tracks (2), not (1) - so Phase 1's numbers are
   unaffected by the (1)-only false positives found in Section 5e.

**Empirically measuring the (1)→(2) lag**: for all 273 `new_construction`
events (FY18-24, all years, not just FY2021), compared each parcel's total
assessed value in its detection year (`y2`) to the following year (`y2+1`).
**37.5% (93 of 248 events with usable data) show a further >10% value
increase the year after first detection** - i.e., the classifier caught
them mid-construction, and the assessed value kept ramping up afterward.
Median further change is a modest +4%, but the distribution has a long
tail: several events show the *majority* of their value still landing in
the following year(s) - e.g. 078-08 (370 Harvard St / Brown Family House,
Section 5e): $4.27M at first detection (FY2021) → $13.16M the next year
(+208%) → $14.25M the year after that (+8% more, now stabilizing). This is
expected/correct behavior per DOR's own rules for phased construction
assessment, not a classifier bug - but it means "when did this project's
new growth land" is itself a multi-year answer for a meaningful minority of
projects, not a single fiscal year.

**Not measurable from current data**: the *earlier* stages of the pipeline
- zoning approval → permit issuance → construction start → occupancy -
aren't in this project's data at all. `classify.py`'s output starts at "the
assessor recorded a change"; nothing here captures permit dates. Closing
this gap would need Brookline's building-permit records (Town permitting
portal or MassGIS) as a new data source. Flagged as an open item for Phase
2's zoning-capacity + lag-modeling work - worth pursuing if that sub-piece
gets prioritized, since without it any zoning-to-tax-roll lag estimate is
a guess rather than measured from Brookline's own permit history.

---

## 6. Time visualization (`scripts/build_timeline_dataset.py`, `site/timeline.html`)

Built 2026-08-27. The "lot boundaries change over time" problem flagged
above as a reason to defer this turned out to be avoidable: we only have
*one* geometry snapshot (today's GIS layer) in the first place, and
`parcel_history_fy17_25.csv` already aggregates every year's values to
*today's* canonical `block_lot` via the same crosswalk used everywhere else
in this project. So there's no need to reconstruct historical lot
boundaries at all - the approach is to attach every year's value/units/event
as separate properties on today's fixed geometry, and let the frontend swap
which year's property drives the color/height/highlight, entirely
client-side (no data reload on year change).

**Build** (`build_timeline_dataset.py`): pivots `parcel_history_fy17_25.csv`
to one row per `block_lot` with `value_2017`...`value_2024`,
`units_2017`...`units_2024`, and `desc_2017`...`desc_2024` columns, plus
`event_2018`...`event_2024` from `classification_fy17_25.csv` (the
classification category for the transition *ending* in that year - there's
no `event_2017` since that's the baseline). Joined to current GIS geometry
the same way as `build_fy26_dataset.py`. A parcel that didn't exist yet in
an earlier year simply has `null` for that year's properties (`allow_nan=False`
still applies - same NaN-to-null discipline as Section 4). Output:
`output/brookline_timeline.geojson` (~12MB, 8,480 features).

**Frontend** (`timeline.html`): same MapLibre GL / OSM-tiles setup as
`index.html`, plus a year slider (2017-2024, with prev/next buttons and an
auto-play toggle), and two outline layers highlighting parcels whose
`event_<year>` matches `new_construction` (red) or `identified_renovation`
(amber) for whichever year is selected - a visual index into the
classification work in Section 5, not just a value snapshot. Metric is
**total assessed value per sqft of land** (not land value specifically -
land/building splits don't exist for most years, only FY2026 has that per
Section 4).

**Bug found and fixed while building this**: the initial version filtered
which parcels render for a given year using `['has', 'value_2017']` in a
MapLibre expression - but since the build script always writes the
`value_<year>` key (as `null` when the parcel didn't exist yet), `has`
always returned true and the filter did nothing. Fixed by checking
`['!=', ['get', 'value_<year>'], null]` instead - `has` checks key
*existence*, not whether the value is `null`.

**Data-quality finding, worked around**: value-per-sqft is dominated by a
handful of tiny condo-unit land slivers (e.g. a 125 sqft nominal land
allocation under a large apartment building, carrying $17-22M in total
value → >$100,000/sqft) - the same underlying land-allocation gap as
Section 3a, just manifesting as a denominator problem here instead of a
`$0` numerator. Excluding parcels under 300 sqft from the color/height scale
computation (`MIN_LOT_SQFT` in `timeline.html`) brought the top of the scale
down from ~$13,000/sqft to ~$6,700/sqft; those parcels still render (clamped
to the top bucket), they just don't distort the scale for everyone else. The
remaining high end of the scale (dense apartment buildings on modest lots,
concentrated around Coolidge Corner) looks like genuine density signal, not
noise.

**First real finding from using it**: stepping through to FY2021 immediately
shows a visible cluster of red (new-construction) outlines in the
southwest/lower-density part of town - the source of the FY20→21
+932-housing-unit spike flagged in Section 5e. **Resolved 2026-08-28** (see
Section 5e for full detail): this cluster is three Brookline Housing
Authority parcels near Brookline Village (High St/Egmont St/Walnut St
corridor - consistent with "southwest/lower-density"), and is mostly a
data-population artifact rather than real construction - only one nearby
event (370 Harvard St, Brown Family House) is a confirmed real project.

### 6a. Not yet built

- A dark basemap (same open item as Section 4).

---

## 7. Growth-sources map (`scripts/build_growth_map_dataset.py`, `scripts/detect_growth_outliers.py`, `site/growth.html`)

Built 2026-08-27. A complementary view to `timeline.html`: instead of a
value-level snapshot per year, this shows **cumulative FY17→24 change** in
one view - "where did growth come from," not "what did values look like in
year X." Every parcel gets one of three treatments:

- **New construction** (flat highlight, red): ever classified
  `new_construction` in any year, FY18-24.
- **Likely renovated** (flat highlight, amber): ever classified
  `identified_renovation` or `unclassified_large`, **or** flagged by the new
  cumulative growth-outlier check below.
- **Everything else**: height/color = FY17→24 dollar growth per sqft of
  land (same viridis convention, same `MIN_LOT_SQFT`-excludes-slivers fix as
  `timeline.html`).

### 7a. Cumulative growth-outlier check (`detect_growth_outliers.py`)

Ben's idea: even without a structural signal, a parcel whose value grew
substantially faster than "normal" over the whole period is itself evidence
of value-adding work invisible to the per-year rules (e.g. a kitchen remodel
that only bumps value 3-4% in the specific year it's reflected - not enough
to trip the per-year 99th-percentile outlier test in Section 5b, but real
over 7 years).

**The baseline matters a lot, and the obvious choice is wrong.** The initial
proposal was to benchmark against MA Prop 2.5's ~2.5%/year figure - but that
caps the town's total tax *levy*, not individual assessed values, which
track the real estate market and aren't capped at all. Checked directly:
Brookline's total assessed value actually grew ~5.9%/year compounded (49.9%
over FY17→24), more than double the naive assumption. Used instead: the
**empirical median year-over-year % change among parcels already confirmed
to have no structural signal** (the `market_adjustment` bucket from
Section 5b) - this compounds to 47.6% over the same period, matching the
town-wide figure almost exactly, which is the validation that this is the
right baseline.

**Method**: for every parcel never classified `new_construction` (excluding
$0-starting parcels, a data artifact), compare actual FY17→24 growth against
the compounded baseline. Flag anything exceeding the baseline by >30
percentage points.

**Result**: 722 parcels exceed the threshold. Of those, 426 were already
caught by an existing structural rule or the per-year outlier test
(corroboration, not new information) - the useful yield is the other **296
parcels** newly identified, with a median growth of ~87% vs. the ~48%
baseline (a real but not extreme signal, unlike the already-flagged top
entries which run 400-1800% and are clearly different kinds of events, not
ordinary renovation). Full list: `output/growth_outlier_parcels.csv`. This
is a bottom-up *estimate*, not a certainty - "grew faster than typical" is
evidence of renovation, not proof, and the page's own caveat text says so.

### 7b. Known limitation: large parcels with a minor flagged event

Found while testing: parcel `441-01` (191 Clyde St, `COMM/RES`, 10.3M sqft /
~237 acres) renders as a solid highlighted block covering its entire
footprint because of a single FY2020→21 event (residential unit count
0→1, $3.55M of that year's growth) - even though ~90% of its FY17→24 growth
($34.8M total) came from ordinary appreciation in the other 6 years, same as
everywhere else. The flat highlight treatment doesn't distinguish "this
parcel's growth is entirely explained by one flagged event" from "this
parcel had one minor flagged event among years of ordinary appreciation."
**Decision (2026-08-27, confirmed with Ben): leave as-is, documented as a
known limitation** rather than engineering a fix (e.g. requiring the flagged
event to represent a meaningful share of total value, or scaling
opacity/height by that share) - this is the only parcel that large in the
flagged set (the rest of the top-10-by-area flagged parcels are more
reasonable 400K-2.6M sqft campus-style properties), so it's a single known
edge case, not a systemic issue.

---

## 8. Script inventory and run order

All scripts run from `land-value-map/` with the local venv active
(`source .venv/bin/activate`; `pip install pandas openpyxl` if rebuilding
the venv from scratch - `pandas`/`geopandas` needed a venv since this
machine's system Python (3.14) has no packages preinstalled).

| Script | Depends on | Produces |
|---|---|---|
| `scripts/build_fy26_dataset.py` | GIS geojson, FY26 CAMA tab | `output/brookline_land_value_fy26.geojson` (`site/index.html`'s data) |
| `scripts/build_history.py` | FY17-25 CAMA sheets, crosswalk tab | `output/parcel_history_fy17_25.csv` |
| `scripts/calibrate_living_area.py` | `parcel_history_fy17_25.csv` | prints the living-area growth distribution (no file output - calibration check only) |
| `scripts/classify.py` | `parcel_history_fy17_25.csv` | `output/classification_fy17_25.csv`, `output/town_totals_by_year.csv`, `output/unit_and_parcel_growth.csv` |
| `scripts/build_timeline_dataset.py` (superseded, see SS9) | `parcel_history_fy17_25.csv`, `classification_fy17_25.csv`, GIS geojson, FY26 CAMA tab | `output/brookline_timeline.geojson` |
| `scripts/detect_growth_outliers.py` | `parcel_history_fy17_25.csv`, `classification_fy17_25.csv` | `output/growth_outlier_parcels.csv` |
| `scripts/build_growth_map_dataset.py` | `parcel_history_fy17_25.csv`, `classification_fy17_25.csv`, `growth_outlier_parcels.csv`, GIS geojson, FY26 CAMA tab | `output/brookline_growth_sources.geojson` (`site/growth.html`'s data) |
| `scripts/build_full_history.py` | FY09-26 CAMA sheets (.xls/.xlsx), crosswalk tab | `output/parcel_full_history_fy09_26.csv` |
| `scripts/build_full_timeline_dataset.py` | `parcel_full_history_fy09_26.csv`, `classification_fy17_25.csv`, GIS geojson, FY26 CAMA tab | `output/brookline_timeline_fy09_26.geojson` (`site/timeline.html`'s data, current) |

Run order: `build_history.py` → `classify.py` → `detect_growth_outliers.py`
→ `build_growth_map_dataset.py` (each reads the previous step's output).
`build_fy26_dataset.py` is independent (map-only, doesn't touch the
multi-year history). `build_full_history.py` → `build_full_timeline_dataset.py`
is a separate, parallel chain feeding only `timeline.html` - it re-reads the
CAMA sheets itself (needs `TOTFYTAX`, which `build_history.py`/
`parcel_history_fy17_25.csv` never extracted) rather than reusing
`parcel_history_fy17_25.csv`, and extends back to FY2009. `classify.py`'s
FY17-25 classification output is still joined in for the new-construction/
renovation highlight layer - see SS9.

The map itself: `python3 -m http.server 8743` from `land-value-map/`, then
open `http://localhost:8743/site/index.html` (current-year land value),
`http://localhost:8743/site/timeline.html` (tax billed / assessed value over
time, FY2009-26), or `http://localhost:8743/site/growth.html` (where growth
came from, FY17→25 cumulative).

---

## 9. Timeline extended to FY2009-26, real tax data (2026-08-31)

Three changes to `timeline.html`, prompted directly by Ben's questions
while reviewing Part 1b of the full-story outline (`../README.md`):

**1. Real tax billed, not value x rate.** Ben: "check the assessed file
data for property tax data - I think it's in the files." It is:
`TOTFYTAX`, present in every CAMA extract FY2009-2025 (and `FY26 TAXES `
in the lighter FY26 extract) - the actual per-parcel tax bill, not an
approximation. This supersedes an earlier plan (same conversation) to
approximate revenue by multiplying assessed value by a historical tax
rate, which this project doesn't have a full FY09-26 series for anyway.
**Validated**: summing `TOTFYTAX` town-wide lands within ~2-3% of the
officially certified levy every year checked (FY09: $145.0M identified vs.
$147.3M certified; FY16: $191.5M vs. $195.0M; FY25: $305.8M vs. $312.6M) -
a small, consistent gap (likely abatements/CPA-surcharge timing, not a
data problem), close enough to trust as the real per-parcel figure.
`scripts/build_full_history.py` re-reads every year's CAMA source
directly to extract this (`parcel_history_fy17_25.csv` never captured
`TOTFYTAX`), producing `output/parcel_full_history_fy09_26.csv`. The map
now defaults to **tax billed** as the primary metric (toggle to assessed
value), and separately toggles between **$/sqft of land** and **total $**
- both toggles per Ben's direction ("the thing we care about most is
increase in property tax revenue per parcel, or per square foot").

**2. Extended back to FY2009**, using the older `.xls`-format CAMA
exports Ben pulled from the Town 2026-08-31 (FY2009-2015; FY2016 is
`.xlsx`). Per Ben's explicit direction, these years are **not** run
through `classify.py`'s new-construction/renovation rules - used as
historical record only (see README.md's Part 1b entry for the same
decision applied project-wide). Confirmed via the land-area check below
that this doesn't introduce a schema mismatch worth worrying about.

**3. "New parcel" highlight, answering Ben's question ("how will we show
growth for new parcels?").** A parcel that first appears mid-window (a
subdivision, lot combination that produced a new id, or a genuinely new
lot) has no earlier-year baseline to compute growth *from* - so rather
than fabricate a partial-period growth rate or silently render it as if
it always existed, `build_full_timeline_dataset.py` flags
`new_parcel_first_year` (the block_lot's first year with any value at
all, only set when that's after FY2009) and the frontend outlines it in
blue for that one year - a third highlight category alongside the
existing new-construction (red) and renovation (amber) outlines. 33
block-lots are flagged this way across FY2010-2026. From its first year
forward, a new parcel simply joins the normal color/height scale like
any other parcel - the discontinuity itself (no data before, real data
after) *is* the growth story for these, not a number to compute. A
genuinely disappeared parcel (existed in an early year, absorbed into
another lot by today) can't be shown as its own feature at all - there's
no current-day polygon for it to render on, the same one-geometry-
snapshot limit noted in Section 6. Its value is either correctly folded
into the surviving canonical block_lot (if the merge is one of the 41
documented crosswalk exceptions) or simply drops out of the visible
total for the years it was separate - a known, undocumented-until-now
edge case, not fixed here.

**Data-quality check performed** (Ben's request, before trusting
per-sqft as a metric): compared FY2025 CAMA `LANDAREA` against GIS
`TOT_LND_AREA` for 6,491 parcels. Median and 95th-percentile disagreement
is exactly 0.00% - same underlying record. The tail (99th percentile
~60% off) is the same tiny condo-unit land-sliver issue already
documented in Section 6 (a nominal 25-300 sqft land allocation under a
large building) - the existing `MIN_LOT_SQFT` exclusion already used for
value-per-sqft handles it the same way for tax-per-sqft.