add superset airbyte setup and merge md file

04-ingestion/ARCHITECTURE.md

@@ -0,0 +1,169 @@
+# Airbyte Network Architecture
+
+## Overview
+
+Airbyte deployment uses the **existing** Nginx Proxy Manager from `01-infra`. No additional nginx is needed in `04-ingestion`.
+
+## Network Flow
+
+```
+Internet (HTTPS)
+    ↓
+Nginx Proxy Manager (01-infra)
+    - Container: nginx-proxy-manager
+    - Ports: 80, 443, 8021 (admin)
+    - Network: shared_data_network
+    ↓
+airbyte-proxy (deployed by abctl)
+    - Container: airbyte-proxy
+    - Internal Port: 8000
+    - External Port: 8030 (mapped)
+    - Network: shared_data_network
+    ↓
+Airbyte Services
+    - airbyte-server
+    - airbyte-worker
+    - airbyte-webapp
+    - airbyte-temporal
+    - etc.
+```
+
+## Access Methods
+
+### 1. Production (via Domain)
+```
+https://ai.sriphat.com/airbyte
+    ↓
+Nginx Proxy Manager (01-infra)
+    ↓
+airbyte-proxy:8000 (internal)
+    ↓
+Airbyte Services
+```
+
+### 2. Local/Development
+```
+http://localhost:8030
+    ↓
+airbyte-proxy:8030 (port mapping)
+    ↓
+Airbyte Services
+```
+
+### 3. Direct IP Access
+```
+http://[SERVER_IP]:8030
+    ↓
+airbyte-proxy:8030 (port mapping)
+    ↓
+Airbyte Services
+```
+
+## Components
+
+### 01-infra (Shared Infrastructure)
+- **Nginx Proxy Manager**: External reverse proxy
+  - Handles SSL/TLS termination
+  - Routes traffic to backend services
+  - Manages authentication (OAuth2/Basic Auth)
+  - Domain: ai.sriphat.com
+
+- **PostgreSQL**: Shared database
+  - Databases: `airbyte`, `temporal`, `temporal_visibility`
+  - Used by Airbyte for metadata storage
+
+- **Keycloak**: Identity provider (optional)
+  - Can be integrated via OAuth2 Proxy
+  - Provides SSO for all services
+
+### 04-ingestion (Airbyte)
+- **airbyte-proxy**: Internal nginx (deployed by abctl)
+  - Routes between Airbyte microservices
+  - NOT for external access
+  - Listens on port 8000 (internal), 8030 (external)
+
+- **Airbyte Services**: Deployed by abctl
+  - All services connect to `shared_data_network`
+  - Communicate with PostgreSQL and each other
+
+## Network Configuration
+
+### shared_data_network
+All services connect to this Docker network:
+- nginx-proxy-manager (01-infra)
+- postgres (01-infra)
+- keycloak (01-infra)
+- airbyte-proxy (04-ingestion)
+- airbyte-server (04-ingestion)
+- airbyte-worker (04-ingestion)
+- airbyte-webapp (04-ingestion)
+- airbyte-temporal (04-ingestion)
+- etc.
+
+### Port Mappings
+
+**External Ports:**
+- 80, 443: Nginx Proxy Manager (HTTPS)
+- 8021: Nginx Proxy Manager Admin UI
+- 8030: Airbyte (direct access, optional)
+- 5435: PostgreSQL (external access)
+
+**Internal Ports:**
+- 8000: airbyte-proxy (accessed by Nginx Proxy Manager)
+- 5432: postgres (internal network only)
+- 8080: keycloak (internal network only)
+
+## Why No Additional Nginx?
+
+1. **abctl deploys airbyte-proxy**: This is Airbyte's internal nginx for routing between microservices
+2. **Nginx Proxy Manager exists**: Already running in `01-infra` for external access
+3. **Shared network**: Both can communicate via `shared_data_network`
+4. **Single point of entry**: Nginx Proxy Manager handles all external traffic
+
+## Configuration Steps
+
+1. **Deploy Infrastructure** (01-infra)
+   ```bash
+   cd 01-infra
+   docker compose --env-file ../.env.global up -d
+   ```
+
+2. **Deploy Airbyte** (04-ingestion)
+   ```bash
+   cd 04-ingestion
+   bash setup-airbyte.sh
+   ```
+   - This deploys airbyte-proxy automatically
+   - Connects to shared_data_network
+   - Uses shared PostgreSQL
+
+3. **Configure Nginx Proxy Manager**
+   - Add proxy host for `ai.sriphat.com`
+   - Forward to `airbyte-proxy:8000`
+   - Enable SSL
+   - Add authentication (optional)
+
+## Security Layers
+
+1. **SSL/TLS**: Nginx Proxy Manager (Let's Encrypt)
+2. **Authentication**: OAuth2 Proxy + Keycloak OR Basic Auth
+3. **Network Isolation**: Docker network (shared_data_network)
+4. **Firewall**: Only expose necessary ports
+
+## Troubleshooting
+
+### Cannot access via domain
+- Check Nginx Proxy Manager is running
+- Verify proxy host configuration
+- Check DNS points to server
+- Verify SSL certificate
+
+### Cannot access locally
+- Check airbyte-proxy is running: `docker ps | grep airbyte-proxy`
+- Verify port 8030 is mapped
+- Check firewall allows port 8030
+
+### Services cannot communicate
+- Verify all containers on `shared_data_network`
+- Check network: `docker network inspect shared_data_network`
+- Verify container names resolve (postgres, airbyte-proxy, etc.)