Files

jigoong 6f6009d63e add superset airbyte setup and merge md file

2026-03-02 21:58:51 +07:00

4.2 KiB

Raw Blame History

Airbyte Network Architecture

Overview

Airbyte deployment uses the existing Nginx Proxy Manager from 01-infra. No additional nginx is needed in 04-ingestion.

Network Flow

Internet (HTTPS)
    ↓
Nginx Proxy Manager (01-infra)
    - Container: nginx-proxy-manager
    - Ports: 80, 443, 8021 (admin)
    - Network: shared_data_network
    ↓
airbyte-proxy (deployed by abctl)
    - Container: airbyte-proxy
    - Internal Port: 8000
    - External Port: 8030 (mapped)
    - Network: shared_data_network
    ↓
Airbyte Services
    - airbyte-server
    - airbyte-worker
    - airbyte-webapp
    - airbyte-temporal
    - etc.

Access Methods

1. Production (via Domain)

https://ai.sriphat.com/airbyte
    ↓
Nginx Proxy Manager (01-infra)
    ↓
airbyte-proxy:8000 (internal)
    ↓
Airbyte Services

2. Local/Development

http://localhost:8030
    ↓
airbyte-proxy:8030 (port mapping)
    ↓
Airbyte Services

3. Direct IP Access

http://[SERVER_IP]:8030
    ↓
airbyte-proxy:8030 (port mapping)
    ↓
Airbyte Services

Components

01-infra (Shared Infrastructure)

Nginx Proxy Manager: External reverse proxy
- Handles SSL/TLS termination
- Routes traffic to backend services
- Manages authentication (OAuth2/Basic Auth)
- Domain: ai.sriphat.com
PostgreSQL: Shared database
- Databases: airbyte, temporal, temporal_visibility
- Used by Airbyte for metadata storage
Keycloak: Identity provider (optional)
- Can be integrated via OAuth2 Proxy
- Provides SSO for all services

04-ingestion (Airbyte)

airbyte-proxy: Internal nginx (deployed by abctl)
- Routes between Airbyte microservices
- NOT for external access
- Listens on port 8000 (internal), 8030 (external)
Airbyte Services: Deployed by abctl
- All services connect to shared_data_network
- Communicate with PostgreSQL and each other

Network Configuration

shared_data_network

All services connect to this Docker network:

nginx-proxy-manager (01-infra)
postgres (01-infra)
keycloak (01-infra)
airbyte-proxy (04-ingestion)
airbyte-server (04-ingestion)
airbyte-worker (04-ingestion)
airbyte-webapp (04-ingestion)
airbyte-temporal (04-ingestion)
etc.

Port Mappings

External Ports:

80, 443: Nginx Proxy Manager (HTTPS)
8021: Nginx Proxy Manager Admin UI
8030: Airbyte (direct access, optional)
5435: PostgreSQL (external access)

Internal Ports:

8000: airbyte-proxy (accessed by Nginx Proxy Manager)
5432: postgres (internal network only)
8080: keycloak (internal network only)

Why No Additional Nginx?

abctl deploys airbyte-proxy: This is Airbyte's internal nginx for routing between microservices
Nginx Proxy Manager exists: Already running in 01-infra for external access
Shared network: Both can communicate via shared_data_network
Single point of entry: Nginx Proxy Manager handles all external traffic

Configuration Steps

Deploy Infrastructure (01-infra)

cd 01-infra
docker compose --env-file ../.env.global up -d

Deploy Airbyte (04-ingestion)
```
cd 04-ingestion
bash setup-airbyte.sh
```
- This deploys airbyte-proxy automatically
- Connects to shared_data_network
- Uses shared PostgreSQL
Configure Nginx Proxy Manager
- Add proxy host for ai.sriphat.com
- Forward to airbyte-proxy:8000
- Enable SSL
- Add authentication (optional)

Security Layers

SSL/TLS: Nginx Proxy Manager (Let's Encrypt)
Authentication: OAuth2 Proxy + Keycloak OR Basic Auth
Network Isolation: Docker network (shared_data_network)
Firewall: Only expose necessary ports

Troubleshooting

Cannot access via domain

Check Nginx Proxy Manager is running
Verify proxy host configuration
Check DNS points to server
Verify SSL certificate

Cannot access locally

Check airbyte-proxy is running: docker ps | grep airbyte-proxy
Verify port 8030 is mapped
Check firewall allows port 8030

Services cannot communicate

Verify all containers on shared_data_network
Check network: docker network inspect shared_data_network
Verify container names resolve (postgres, airbyte-proxy, etc.)

4.2 KiB Raw Blame History