sriphat-dataplatform/04-ingestion/NGINX-SETUP.md

Nginx Proxy Manager Configuration for Airbyte

Overview

This guide explains how to configure Nginx Proxy Manager to expose Airbyte at https://ai.sriphat.com/airbyte with optional Keycloak authentication.

Prerequisites

• Airbyte installed and running (port 8030)
• Nginx Proxy Manager running (port 8021 for admin)
• Domain ai.sriphat.com pointing to your server
• SSL certificate (Let's Encrypt recommended)

Step 1: Access Nginx Proxy Manager

1. Open browser: http://localhost:8021
2. Login with admin credentials (from .env.global)

Step 2: Add Proxy Host

Basic Configuration

1. Click "Proxy Hosts" → "Add Proxy Host"

2. Details Tab:

   • Domain Names: ai.sriphat.com
   • Scheme: http
   • Forward Hostname/IP: airbyte-proxy
   • Forward Port: 8000
   • Cache Assets: ✓ (enabled)
   • Block Common Exploits: ✓ (enabled)
   • Websockets Support: ✓ (enabled)

3. Custom Locations Tab:

   • Click "Add Location"
   • Location: /airbyte
   • Scheme: http
   • Forward Hostname/IP: airbyte-proxy
   • Forward Port: 8000
   • Custom Config:

     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     proxy_set_header X-Forwarded-Proto $scheme;
     proxy_set_header X-Real-IP $remote_addr;
     proxy_set_header Host $host;
     
     # Remove /airbyte prefix when forwarding
     rewrite ^/airbyte/(.*) /$1 break;
     

4. SSL Tab:

   • SSL Certificate: Select existing or create new Let's Encrypt
   • Force SSL: ✓ (enabled)
   • HTTP/2 Support: ✓ (enabled)
   • HSTS Enabled: ✓ (enabled)

5. Click "Save"

Step 3: Configure Keycloak Authentication (Optional)

Since Airbyte doesn't natively support Keycloak, we'll use nginx authentication.

Option A: OAuth2 Proxy with Keycloak

1. Deploy OAuth2 Proxy container:

   docker run -d \
     --name oauth2-proxy \
     --network shared_data_network \
     -p 4180:4180 \
     quay.io/oauth2-proxy/oauth2-proxy:latest \
     --provider=keycloak-oidc \
     --client-id=airbyte \
     --client-secret=YOUR_CLIENT_SECRET \
     --redirect-url=https://ai.sriphat.com/oauth2/callback \
     --oidc-issuer-url=https://ai.sriphat.com/keycloak/realms/master \
     --cookie-secret=RANDOM_SECRET_32_CHARS \
     --email-domain=* \
     --upstream=http://airbyte-proxy:8000
   

2. Update Nginx Proxy Host Custom Config:

   # OAuth2 authentication
   auth_request /oauth2/auth;
   error_page 401 = /oauth2/sign_in;
   
   # Pass auth headers
   auth_request_set $user $upstream_http_x_auth_request_user;
   auth_request_set $email $upstream_http_x_auth_request_email;
   proxy_set_header X-User $user;
   proxy_set_header X-Email $email;
   
   # OAuth2 proxy location
   location /oauth2/ {
       proxy_pass http://oauth2-proxy:4180;
       proxy_set_header Host $host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Scheme $scheme;
   }
   

Option B: Basic Authentication (Simpler)

1. In Nginx Proxy Manager, go to Access Lists
2. Create new Access List:
   • Name: Airbyte Access
   • Satisfy Any: ✓
   • Add users with username/password
3. Apply Access List to Airbyte Proxy Host

Option C: IP Whitelist

1. In Nginx Proxy Manager Access List
2. Add allowed IP addresses:
   • Internal network: 192.168.0.0/16
   • VPN range: 10.0.0.0/8
   • Specific IPs as needed

Step 4: Keycloak Client Setup (for OAuth2 Proxy)

1. Login to Keycloak: http://localhost:8080
2. Select realm (or create new)
3. Go to Clients → Create
4. Client Configuration:
   • Client ID: airbyte
   • Client Protocol: openid-connect
   • Access Type: confidential
   • Valid Redirect URIs: https://ai.sriphat.com/oauth2/callback
   • Web Origins: https://ai.sriphat.com
5. Save and copy Client Secret from Credentials tab

Step 5: Test Configuration

1. Access Airbyte:

   • External: https://ai.sriphat.com/airbyte
   • Local: http://localhost:8030

2. Verify:

   • SSL certificate is valid
   • Authentication works (if enabled)
   • Websockets work (for real-time updates)
   • No CORS errors in browser console

Troubleshooting

502 Bad Gateway

• Check if airbyte-proxy container is running
• Verify network connectivity: docker network inspect shared_data_network
• Check logs: docker logs airbyte-proxy

Authentication Loop

• Clear browser cookies
• Verify OAuth2 Proxy configuration
• Check Keycloak client settings

WebSocket Errors

• Ensure "Websockets Support" is enabled in nginx
• Check browser console for connection errors
• Verify proxy headers are set correctly

SSL Certificate Issues

• Use Let's Encrypt for automatic renewal
• Ensure domain DNS points to server
• Check firewall allows ports 80 and 443

Security Recommendations

1. Always use HTTPS in production
2. Enable authentication (OAuth2 or Basic Auth)
3. Whitelist IPs if possible
4. Enable rate limiting in nginx
5. Regular security updates for all components
6. Monitor access logs for suspicious activity

Alternative: Direct Access

For development or internal use, access directly:

http://[SERVER_IP]:8030

No authentication required, but only accessible from local network.