# Lead Generation System Architecture

## System Overview

```
SOURCES (pluggable)              SHARED PIPELINE              DOWNSTREAM
+-----------------+
| WF-1: ImmoScout |--+
|  (Gmail emails) |  |
+-----------------+  |
+-----------------+  |     +---------------------+     +---------------+
| WF-2: Google    |--+---->| WF-ENRICH (webhook) |---->| NocoDB        |
|  Maps API       |  |     |                     |     | Master Leads  |
+-----------------+  |     | 1. Dedup            |     +-------+-------+
+-----------------+  |     | 2. SearXNG website  |             |
| WF-X: Future    |--+     | 3. Crawl4AI Impr.   |     +-------v-------+
|  (IVD, LinkedIn)|        | 4. Hunter fallback  |     | Pipedrive CRM |
+-----------------+        | 5. Classify Dev/Brk |     +-------+-------+
                           | 6. Save NocoDB      |             |
                           +---------------------+     +-------v-------+
                                                       | WF-3: Outreach|
                                                       | WF-5: Replies |
                                                       +---------------+
```

## Workflows

| ID | Name | Trigger | Status |
|----|------|---------|--------|
| WF-1 | ImmoScout Lead Scraper | Schedule (4h) | BUILT (Branch B broken) |
| WF-2 | Google Maps Scraper | Schedule (daily) | NEW |
| WF-3 | Outreach Sequence | Schedule (weekdays) | BUILT |
| WF-5 | Reply Handling + Cal.com | IMAP + Webhook | BUILT |
| WF-ENRICH | Lead Enrichment Pipeline | Webhook | NEW |

## Raw Lead Contract

Every source sends this shape to WF-ENRICH via `POST /webhook/enrich-lead`:

```json
{
  "company": "Engel & Volkers Leipzig",
  "contactPerson": "Max Muller",
  "phone": "+49 341 2156780",
  "website": "https://www.engelvoelkers.com/leipzig",
  "email": null,
  "address": "Grimmaische Str. 15, 04109 Leipzig",
  "source": "google_maps",
  "sourceId": "ChIJN1t_tDeuEmsRUsoyG83frY4",
  "region": "Leipzig",
  "exposeUrl": null
}
```

Required fields: `company`, `source`. Everything else is optional - WF-ENRICH fills gaps.

## Dedup Strategy

1. **NocoDB search** by `company_name` (exact match)
2. Future: add `source_id` column for Google Place ID / Expose ID exact dedup
3. WF-ENRICH returns `{ status: "duplicate" }` or `{ status: "enriched" }` so callers know

## Enrichment Pipeline (WF-ENRICH)

```
Input --> Validate --> Dedup --> Has Website?
  |                              |YES        |NO
  |                              v            v
  |                          Crawl4AI     SearXNG Search
  |                          /impressum   Find Website
  |                              |            |
  |                              v            v
  |                          LLM Extract  Website Found?
  |                          Contact Info  |YES    |NO
  |                              |         |       |
  |                              v         v       |
  |                          Has Email? ------+    |
  |                          |YES   |NO       |    |
  |                          |      v         |    |
  |                          |   Hunter.io    |    |
  |                          |      |         |    |
  |                          v      v         v    v
  |                          Classify Dev/Broker <-+
  |                              |
  |                              v
  |                          Save NocoDB
  |                          + Pipedrive (optional)
  |                              |
  |                              v
  +-- Duplicate? --> Respond "duplicate"
                             Respond "enriched"
```

## Google Maps (WF-2) Coverage

- **Cities**: 50 major German cities (Berlin through Trier)
- **Search terms**: Immobilienmakler, Projektentwickler, Bautraeger, Immobilienentwickler, Hausverwaltung
- **Total queries**: 250 (50 x 5)
- **Daily batch**: 10 queries per run
- **Full rotation**: 25 days to cover all Germany, then repeats
- **Dedup**: WF-ENRICH handles it - same company in multiple cities won't duplicate
- **Cost**: ~$0.035/query x 10/day = ~$10.50/month

## Adding a New Source

To add a new lead source (e.g., IVD directory, LinkedIn):

1. Create a new workflow (WF-X)
2. Extract leads with at minimum: `company` + `source`
3. For each lead, POST to `/webhook/enrich-lead` with the raw lead contract
4. WF-ENRICH handles: dedup, website resolution, Impressum crawl, email finding, classification, saving

No need to duplicate enrichment logic. Just produce raw leads and call the webhook.

## Credentials

| Service | n8n Credential ID | Used By |
|---------|-------------------|---------|
| NocoDB (SAITS) | Wrl6izLljL19To7P | WF-1, WF-3, WF-5, WF-ENRICH |
| Pipedrive (search) | aaEMVtuoOEERYnpr | WF-1 |
| Pipedrive (create) | qKWEtHCuBUnzXz2S | WF-1, WF-ENRICH (disabled) |
| Pipedrive (Hafiz) | YHw7lY4WD1y7ADiy | WF-3, WF-5 |
| Hunter.io | hbfOGc7SwKzUWW0W | WF-1, WF-ENRICH |
| Google Gemini | aCfN9MB8QiwWx6Db | WF-1, WF-3, WF-ENRICH |
| OpenRouter | Z98cVoKXpY86hooz | WF-1, WF-3, WF-5, WF-ENRICH |
| Gmail OAuth | 10aXSIcKBrPBH1rt | WF-1 |
| IMAP | mRjeIxogp2299B0r | WF-5 |

## Setup: Google Maps API (WF-2)

1. Go to Google Cloud Console > APIs & Services
2. Enable "Places API (New)"
3. Create API key, restrict to Places API only
4. In n8n: Settings > Variables > add `GOOGLE_MAPS_API_KEY`
5. Import `wf2-google-maps-scraper.json`
6. Ensure WF-ENRICH is active first (webhook must be listening)
7. Test with a manual trigger before enabling schedule

## Setup: WF-ENRICH

1. Import `wf-enrich-lead.json` into n8n
2. Activate the workflow (webhook starts listening)
3. Test: `curl -X POST http://localhost:5678/webhook/enrich-lead -H "Content-Type: application/json" -d '{"company": "Test GmbH", "source": "manual", "region": "Leipzig"}'`
4. Verify NocoDB entry created
5. Optional: enable Pipedrive nodes when ready

## NocoDB Master Table Fields

Existing fields (from WF-1):
`company_name`, `contact_person`, `type`, `source`, `region`, `campaign`, `website`, `domain`, `email`, `phone`, `email_source`, `expose_url`, `extracted_at`, `enrichment_status`, `workflow_source`, `outreach_stage`, `last_email_date`, `reply_status`, `reply_date`, `meeting_booked`, `meeting_date`, `notes`, `impressum_content`, `linkedin_content`

Recommended additions for Google Maps:
- `source_id` (text) - Google Place ID, Expose ID, etc. for exact dedup
- `address` (text) - Full address from Google Maps
- `rating` (number) - Google Maps rating
- `review_count` (number) - Number of Google reviews
- `google_maps_url` (URL) - Link to Google Maps listing

## Future Refactoring: WF-1

WF-1 currently has enrichment logic inline. To unify:
1. Keep WF-1's Gmail parsing + company extraction as-is
2. Replace the enrichment pipeline (SearXNG -> Crawl4AI -> Hunter -> Classify -> Save) with a single HTTP call to WF-ENRICH webhook
3. This makes WF-1 ~15 nodes instead of ~58
4. Not urgent - WF-1 works. Refactor when convenient.