add superset airbyte setup and merge md file

2026-03-02 21:58:51 +07:00
parent 550d926139
commit 6f6009d63e
15 changed files with 1220 additions and 19 deletions
--- a/04-ingestion/NGINX-SETUP.md
+++ b/04-ingestion/NGINX-SETUP.md
@@ -0,0 +1,183 @@
+# Nginx Proxy Manager Configuration for Airbyte
+
+## Overview
+
+This guide explains how to configure Nginx Proxy Manager to expose Airbyte at `https://ai.sriphat.com/airbyte` with optional Keycloak authentication.
+
+## Prerequisites
+
+- Airbyte installed and running (port 8030)
+- Nginx Proxy Manager running (port 8021 for admin)
+- Domain `ai.sriphat.com` pointing to your server
+- SSL certificate (Let's Encrypt recommended)
+
+## Step 1: Access Nginx Proxy Manager
+
+1. Open browser: `http://localhost:8021`
+2. Login with admin credentials (from `.env.global`)
+
+## Step 2: Add Proxy Host
+
+### Basic Configuration
+
+1. Click **"Proxy Hosts"** → **"Add Proxy Host"**
+
+2. **Details Tab:**
+   - Domain Names: `ai.sriphat.com`
+   - Scheme: `http`
+   - Forward Hostname/IP: `airbyte-proxy`
+   - Forward Port: `8000`
+   - Cache Assets: ✓ (enabled)
+   - Block Common Exploits: ✓ (enabled)
+   - Websockets Support: ✓ (enabled)
+
+3. **Custom Locations Tab:**
+   - Click **"Add Location"**
+   - Location: `/airbyte`
+   - Scheme: `http`
+   - Forward Hostname/IP: `airbyte-proxy`
+   - Forward Port: `8000`
+   - Custom Config:
+     ```nginx
+     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+     proxy_set_header X-Forwarded-Proto $scheme;
+     proxy_set_header X-Real-IP $remote_addr;
+     proxy_set_header Host $host;
+     
+     # Remove /airbyte prefix when forwarding
+     rewrite ^/airbyte/(.*) /$1 break;
+     ```
+
+4. **SSL Tab:**
+   - SSL Certificate: Select existing or create new Let's Encrypt
+   - Force SSL: ✓ (enabled)
+   - HTTP/2 Support: ✓ (enabled)
+   - HSTS Enabled: ✓ (enabled)
+
+5. Click **"Save"**
+
+## Step 3: Configure Keycloak Authentication (Optional)
+
+Since Airbyte doesn't natively support Keycloak, we'll use nginx authentication.
+
+### Option A: OAuth2 Proxy with Keycloak
+
+1. Deploy OAuth2 Proxy container:
+   ```bash
+   docker run -d \
+     --name oauth2-proxy \
+     --network shared_data_network \
+     -p 4180:4180 \
+     quay.io/oauth2-proxy/oauth2-proxy:latest \
+     --provider=keycloak-oidc \
+     --client-id=airbyte \
+     --client-secret=YOUR_CLIENT_SECRET \
+     --redirect-url=https://ai.sriphat.com/oauth2/callback \
+     --oidc-issuer-url=https://ai.sriphat.com/keycloak/realms/master \
+     --cookie-secret=RANDOM_SECRET_32_CHARS \
+     --email-domain=* \
+     --upstream=http://airbyte-proxy:8000
+   ```
+
+2. Update Nginx Proxy Host Custom Config:
+   ```nginx
+   # OAuth2 authentication
+   auth_request /oauth2/auth;
+   error_page 401 = /oauth2/sign_in;
+   
+   # Pass auth headers
+   auth_request_set $user $upstream_http_x_auth_request_user;
+   auth_request_set $email $upstream_http_x_auth_request_email;
+   proxy_set_header X-User $user;
+   proxy_set_header X-Email $email;
+   
+   # OAuth2 proxy location
+   location /oauth2/ {
+       proxy_pass http://oauth2-proxy:4180;
+       proxy_set_header Host $host;
+       proxy_set_header X-Real-IP $remote_addr;
+       proxy_set_header X-Scheme $scheme;
+   }
+   ```
+
+### Option B: Basic Authentication (Simpler)
+
+1. In Nginx Proxy Manager, go to **Access Lists**
+2. Create new Access List:
+   - Name: `Airbyte Access`
+   - Satisfy Any: ✓
+   - Add users with username/password
+3. Apply Access List to Airbyte Proxy Host
+
+### Option C: IP Whitelist
+
+1. In Nginx Proxy Manager Access List
+2. Add allowed IP addresses:
+   - Internal network: `192.168.0.0/16`
+   - VPN range: `10.0.0.0/8`
+   - Specific IPs as needed
+
+## Step 4: Keycloak Client Setup (for OAuth2 Proxy)
+
+1. Login to Keycloak: `http://localhost:8080`
+2. Select realm (or create new)
+3. Go to **Clients** → **Create**
+4. Client Configuration:
+   - Client ID: `airbyte`
+   - Client Protocol: `openid-connect`
+   - Access Type: `confidential`
+   - Valid Redirect URIs: `https://ai.sriphat.com/oauth2/callback`
+   - Web Origins: `https://ai.sriphat.com`
+5. Save and copy **Client Secret** from Credentials tab
+
+## Step 5: Test Configuration
+
+1. Access Airbyte:
+   - External: `https://ai.sriphat.com/airbyte`
+   - Local: `http://localhost:8030`
+
+2. Verify:
+   - SSL certificate is valid
+   - Authentication works (if enabled)
+   - Websockets work (for real-time updates)
+   - No CORS errors in browser console
+
+## Troubleshooting
+
+### 502 Bad Gateway
+- Check if `airbyte-proxy` container is running
+- Verify network connectivity: `docker network inspect shared_data_network`
+- Check logs: `docker logs airbyte-proxy`
+
+### Authentication Loop
+- Clear browser cookies
+- Verify OAuth2 Proxy configuration
+- Check Keycloak client settings
+
+### WebSocket Errors
+- Ensure "Websockets Support" is enabled in nginx
+- Check browser console for connection errors
+- Verify proxy headers are set correctly
+
+### SSL Certificate Issues
+- Use Let's Encrypt for automatic renewal
+- Ensure domain DNS points to server
+- Check firewall allows ports 80 and 443
+
+## Security Recommendations
+
+1. **Always use HTTPS** in production
+2. **Enable authentication** (OAuth2 or Basic Auth)
+3. **Whitelist IPs** if possible
+4. **Enable rate limiting** in nginx
+5. **Regular security updates** for all components
+6. **Monitor access logs** for suspicious activity
+
+## Alternative: Direct Access
+
+For development or internal use, access directly:
+```
+http://[SERVER_IP]:8030
+```
+
+No authentication required, but only accessible from local network.