fix: data integrity - dedup prices, batch writes, safe parsePrice, remove hardcoded locality #2

Merged

jperera merged 1 commit from fix/data-integrity into main

2026-04-02 20:03:22 +02:00

jperera commented

2026-04-02 19:57:37 +02:00

Owner

Summary

DB deduplication: Add unique index on prices(station_id, fuel_type, fecha_actualizacion) to prevent unbounded growth from duplicate price entries on every scheduler run. Uses INSERT OR IGNORE to skip gracefully.
Batch writes: New savePricesBatch() wraps all station+price saves in a single SQLite transaction instead of 40-60 individual transactions per collection cycle.
Safe parsePrice: parsePrice() returns null for invalid data instead of 0, which would have been stored as a real price and displayed to users as "0.000 EUR/L". Stations with unparseable prices are now filtered out.
Remove hardcoded locality: filterByPostalCodePrefix() no longer silently ORs in stations where localidad.startsWith('lorca'), which bypassed postal code filtering and was coupled to a specific city.

Files changed

src/lib/db.ts — Unique index, INSERT OR IGNORE, new savePricesBatch() with transaction
src/lib/api.ts — parsePrice() returns null, parseStationData() filters nulls, locality check removed
src/lib/scheduler.ts — Uses savePricesBatch() instead of loop with individual saves

Test plan

bun test — all 30 tests pass
Verify price collection still works with batch writes
Verify duplicate prices are not inserted on repeated runs

## Summary - **DB deduplication**: Add unique index on `prices(station_id, fuel_type, fecha_actualizacion)` to prevent unbounded growth from duplicate price entries on every scheduler run. Uses `INSERT OR IGNORE` to skip gracefully. - **Batch writes**: New `savePricesBatch()` wraps all station+price saves in a single SQLite transaction instead of 40-60 individual transactions per collection cycle. - **Safe parsePrice**: `parsePrice()` returns `null` for invalid data instead of `0`, which would have been stored as a real price and displayed to users as "0.000 EUR/L". Stations with unparseable prices are now filtered out. - **Remove hardcoded locality**: `filterByPostalCodePrefix()` no longer silently ORs in stations where `localidad.startsWith('lorca')`, which bypassed postal code filtering and was coupled to a specific city. ## Files changed - `src/lib/db.ts` — Unique index, `INSERT OR IGNORE`, new `savePricesBatch()` with transaction - `src/lib/api.ts` — `parsePrice()` returns `null`, `parseStationData()` filters nulls, locality check removed - `src/lib/scheduler.ts` — Uses `savePricesBatch()` instead of loop with individual saves ## Test plan - [x] `bun test` — all 30 tests pass - [ ] Verify price collection still works with batch writes - [ ] Verify duplicate prices are not inserted on repeated runs

jperera added 1 commit

2026-04-02 19:57:37 +02:00

fix: data integrity - dedup prices, batch writes, safe parsePrice, remove hardcoded locality 2c4ac06a13

- Add unique index on prices(station_id, fuel_type, fecha_actualizacion)
  to prevent duplicate price entries on every scheduler run
- Use INSERT OR IGNORE for price saves to skip duplicates gracefully
- Add savePricesBatch() with single transaction for all station+price
  writes instead of 40-60 individual transactions
- parsePrice() now returns null for invalid data instead of 0, which
  would have been stored as a real price and shown to users
- parseStationData() filters out stations with unparseable prices
- Remove hardcoded 'lorca' locality check from filterByPostalCodePrefix
  that silently bypassed postal code filtering

jperera force-pushed fix/data-integrity from 2c4ac06a13 to 5ecdd0f205

2026-04-02 20:03:06 +02:00

Compare