turf_saas

admin/turf_saas

Fork 0

Commit Graph

Author	SHA1	Message	Date
DevOps Engineer	86e85aa1c6	fix(HRT-72): fix Overpass OSM scraper — bounding box + Content-Type + User-Agent Bug 1: Replace area["name"="..."] query with direct bounding box (50.4,2.8,50.8,3.3) — area resolution fails silently on public Overpass API depending on server version. — Direct bbox is deterministic and reliable for MEL coverage. — Also simplify website filter to use [!"website"] tag negation syntax. Bug 2: Add explicit Content-Type: application/x-www-form-urlencoded header — Some network configs/proxies strip the implicit header set by requests.post(data={}). — Explicit header is best practice per Overpass API docs. Bug 3 (discovered during test): Add User-Agent header — overpass-api.de returns 406 Not Acceptable for User-Agent: python-requests/*. — Fix: send H3R7Tech-LeadHunter/1.0 as custom User-Agent. — Tested: 5 OSM leads returned from Lille center bounding box. Backup: leadhunter_scraper.py.backup_20260427_221429 Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-04-27 22:19:10 +02:00
DevOps Engineer	f9a45e6deb	feat(HRT-66): LeadHunter S1 — core scraping, scoring, CRM SQLite et API Flask - leadhunter_scraper.py : Google Places Nearby Search + Place Details avec compteur quota daily_quota.json (limite 900/jour), sleep(0.5) entre requêtes, fallback Overpass OSM boundary MEL, filtre website absent, déduplcation, rgpd_ok=True - leadhunter_scorer.py : moteur de scoring 0-8 pts critère n°1 = site web absent (+3), avis ≥50 (+2), note ≥4.0 (+2), téléphone (+1), note <3.0 (-1) - leadhunter_crm.py : CRM SQLite schéma validé CTO (id, source, name, address, phone, rating, reviews_count, website, score, rgpd_ok, scraped_at, status) CRUD : insert_lead, get_leads, update_lead_status, get_stats, export_csv - leadhunter_api.py : Flask service port 8769 GET /api/leads, POST /api/leads/scrape, GET /api/leads/stats, GET /api/leads/export, PATCH /api/leads/<id>/status, GET /health assert GOOGLE_PLACES_API_KEY au démarrage scraping asynchrone (thread) avec status endpoint - infra/turf-saas-leadhunter.service : service systemd EnvironmentFile=/home/h3r7/.env pour GOOGLE_PLACES_API_KEY Tests : py_compile OK, scorer testé, CRM SQLite testé Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-04-27 16:33:30 +02:00

Author

SHA1

Message

Date

DevOps Engineer

86e85aa1c6

fix(HRT-72): fix Overpass OSM scraper — bounding box + Content-Type + User-Agent

Bug 1: Replace area["name"="..."] query with direct bounding box (50.4,2.8,50.8,3.3)
  — area resolution fails silently on public Overpass API depending on server version.
  — Direct bbox is deterministic and reliable for MEL coverage.
  — Also simplify website filter to use [!"website"] tag negation syntax.

Bug 2: Add explicit Content-Type: application/x-www-form-urlencoded header
  — Some network configs/proxies strip the implicit header set by requests.post(data={}).
  — Explicit header is best practice per Overpass API docs.

Bug 3 (discovered during test): Add User-Agent header
  — overpass-api.de returns 406 Not Acceptable for User-Agent: python-requests/*.
  — Fix: send H3R7Tech-LeadHunter/1.0 as custom User-Agent.
  — Tested: 5 OSM leads returned from Lille center bounding box.

Backup: leadhunter_scraper.py.backup_20260427_221429

Co-Authored-By: Paperclip <noreply@paperclip.ing>

2026-04-27 22:19:10 +02:00

DevOps Engineer

f9a45e6deb

feat(HRT-66): LeadHunter S1 — core scraping, scoring, CRM SQLite et API Flask

- leadhunter_scraper.py : Google Places Nearby Search + Place Details
  avec compteur quota daily_quota.json (limite 900/jour),
  sleep(0.5) entre requêtes, fallback Overpass OSM boundary MEL,
  filtre website absent, déduplcation, rgpd_ok=True

- leadhunter_scorer.py : moteur de scoring 0-8 pts
  critère n°1 = site web absent (+3), avis ≥50 (+2),
  note ≥4.0 (+2), téléphone (+1), note <3.0 (-1)

- leadhunter_crm.py : CRM SQLite schéma validé CTO
  (id, source, name, address, phone, rating, reviews_count,
   website, score, rgpd_ok, scraped_at, status)
  CRUD : insert_lead, get_leads, update_lead_status, get_stats, export_csv

- leadhunter_api.py : Flask service port 8769
  GET /api/leads, POST /api/leads/scrape, GET /api/leads/stats,
  GET /api/leads/export, PATCH /api/leads/<id>/status, GET /health
  assert GOOGLE_PLACES_API_KEY au démarrage
  scraping asynchrone (thread) avec status endpoint

- infra/turf-saas-leadhunter.service : service systemd
  EnvironmentFile=/home/h3r7/.env pour GOOGLE_PLACES_API_KEY

Tests : py_compile OK, scorer testé, CRM SQLite testé

Co-Authored-By: Paperclip <noreply@paperclip.ing>

2026-04-27 16:33:30 +02:00

2 Commits