# Airbyte Network Architecture

## Overview

Airbyte deployment uses the **existing** Nginx Proxy Manager from `01-infra`. No additional nginx is needed in `04-ingestion`.

## Network Flow

```
Internet (HTTPS)
    ↓
Nginx Proxy Manager (01-infra)
    - Container: nginx-proxy-manager
    - Ports: 80, 443, 8021 (admin)
    - Network: shared_data_network
    ↓
airbyte-proxy (deployed by abctl)
    - Container: airbyte-proxy
    - Internal Port: 8000
    - External Port: 8030 (mapped)
    - Network: shared_data_network
    ↓
Airbyte Services
    - airbyte-server
    - airbyte-worker
    - airbyte-webapp
    - airbyte-temporal
    - etc.
```

## Access Methods

### 1. Production (via Domain)
```
https://ai.sriphat.com/airbyte
    ↓
Nginx Proxy Manager (01-infra)
    ↓
airbyte-proxy:8000 (internal)
    ↓
Airbyte Services
```

### 2. Local/Development
```
http://localhost:8030
    ↓
airbyte-proxy:8030 (port mapping)
    ↓
Airbyte Services
```

### 3. Direct IP Access
```
http://[SERVER_IP]:8030
    ↓
airbyte-proxy:8030 (port mapping)
    ↓
Airbyte Services
```

## Components

### 01-infra (Shared Infrastructure)
- **Nginx Proxy Manager**: External reverse proxy
  - Handles SSL/TLS termination
  - Routes traffic to backend services
  - Manages authentication (OAuth2/Basic Auth)
  - Domain: ai.sriphat.com

- **PostgreSQL**: Shared database
  - Databases: `airbyte`, `temporal`, `temporal_visibility`
  - Used by Airbyte for metadata storage

- **Keycloak**: Identity provider (optional)
  - Can be integrated via OAuth2 Proxy
  - Provides SSO for all services

### 04-ingestion (Airbyte)
- **airbyte-proxy**: Internal nginx (deployed by abctl)
  - Routes between Airbyte microservices
  - NOT for external access
  - Listens on port 8000 (internal), 8030 (external)

- **Airbyte Services**: Deployed by abctl
  - All services connect to `shared_data_network`
  - Communicate with PostgreSQL and each other

## Network Configuration

### shared_data_network
All services connect to this Docker network:
- nginx-proxy-manager (01-infra)
- postgres (01-infra)
- keycloak (01-infra)
- airbyte-proxy (04-ingestion)
- airbyte-server (04-ingestion)
- airbyte-worker (04-ingestion)
- airbyte-webapp (04-ingestion)
- airbyte-temporal (04-ingestion)
- etc.

### Port Mappings

**External Ports:**
- 80, 443: Nginx Proxy Manager (HTTPS)
- 8021: Nginx Proxy Manager Admin UI
- 8030: Airbyte (direct access, optional)
- 5435: PostgreSQL (external access)

**Internal Ports:**
- 8000: airbyte-proxy (accessed by Nginx Proxy Manager)
- 5432: postgres (internal network only)
- 8080: keycloak (internal network only)

## Why No Additional Nginx?

1. **abctl deploys airbyte-proxy**: This is Airbyte's internal nginx for routing between microservices
2. **Nginx Proxy Manager exists**: Already running in `01-infra` for external access
3. **Shared network**: Both can communicate via `shared_data_network`
4. **Single point of entry**: Nginx Proxy Manager handles all external traffic

## Configuration Steps

1. **Deploy Infrastructure** (01-infra)
   ```bash
   cd 01-infra
   docker compose --env-file ../.env.global up -d
   ```

2. **Deploy Airbyte** (04-ingestion)
   ```bash
   cd 04-ingestion
   bash setup-airbyte.sh
   ```
   - This deploys airbyte-proxy automatically
   - Connects to shared_data_network
   - Uses shared PostgreSQL

3. **Configure Nginx Proxy Manager**
   - Add proxy host for `ai.sriphat.com`
   - Forward to `airbyte-proxy:8000`
   - Enable SSL
   - Add authentication (optional)

## Security Layers

1. **SSL/TLS**: Nginx Proxy Manager (Let's Encrypt)
2. **Authentication**: OAuth2 Proxy + Keycloak OR Basic Auth
3. **Network Isolation**: Docker network (shared_data_network)
4. **Firewall**: Only expose necessary ports

## Troubleshooting

### Cannot access via domain
- Check Nginx Proxy Manager is running
- Verify proxy host configuration
- Check DNS points to server
- Verify SSL certificate

### Cannot access locally
- Check airbyte-proxy is running: `docker ps | grep airbyte-proxy`
- Verify port 8030 is mapped
- Check firewall allows port 8030

### Services cannot communicate
- Verify all containers on `shared_data_network`
- Check network: `docker network inspect shared_data_network`
- Verify container names resolve (postgres, airbyte-proxy, etc.)