add files infra docker service for data platform
This commit is contained in:
jigoong
298
DEPLOYMENT.md
Normal file
298
DEPLOYMENT.md
Normal file
@@ -0,0 +1,298 @@
|
||||
# Sriphat Data Platform - Deployment Guide
|
||||
|
||||
## 📋 Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Nginx Proxy Manager │
|
||||
│ (Gateway + SSL + Domain Routing) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────┼─────────────────────┐
|
||||
│ │ │
|
||||
┌───────▼────────┐ ┌────────▼────────┐ ┌───────▼────────┐
|
||||
│ Keycloak │ │ API Service │ │ Superset │
|
||||
│ (SSO) │ │ (FastAPI) │ │ (BI) │
|
||||
└────────────────┘ └─────────────────┘ └────────────────┘
|
||||
│ │ │
|
||||
└─────────────────────┼─────────────────────┘
|
||||
│
|
||||
┌─────────▼─────────┐
|
||||
│ PostgreSQL │
|
||||
│ (Data Warehouse) │
|
||||
└───────────────────┘
|
||||
│
|
||||
┌─────────▼─────────┐
|
||||
│ Airbyte │
|
||||
│ (Data Ingestion) │
|
||||
└───────────────────┘
|
||||
```
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### Prerequisites
|
||||
- Docker & Docker Compose installed
|
||||
- Minimum 8GB RAM
|
||||
- 50GB disk space
|
||||
|
||||
### Step 1: Clone & Configure
|
||||
```bash
|
||||
cd e:\git3\sriphat-dataplatform
|
||||
|
||||
# Review and update credentials in .env.global
|
||||
notepad .env.global
|
||||
```
|
||||
|
||||
### Step 2: Start All Services
|
||||
```bash
|
||||
# On Linux/Mac
|
||||
bash start-all.sh
|
||||
|
||||
# On Windows (PowerShell)
|
||||
bash start-all.sh
|
||||
# OR manually:
|
||||
# 1. cd 00-network && bash create-network.sh
|
||||
# 2. cd ../01-infra && docker compose --env-file ../.env.global up -d
|
||||
# 3. Wait 30 seconds for PostgreSQL
|
||||
# 4. cd ../03-apiservice && docker compose --env-file ../.env.global up --build -d
|
||||
# 5. cd ../04-ingestion && docker compose --env-file ../.env.global up -d
|
||||
# 6. cd ../06-analytics && docker compose --env-file ../.env.global up -d
|
||||
```
|
||||
|
||||
### Step 3: Verify Services
|
||||
```bash
|
||||
docker ps
|
||||
```
|
||||
|
||||
You should see:
|
||||
- nginx-proxy-manager
|
||||
- keycloak
|
||||
- postgres
|
||||
- apiservice
|
||||
- airbyte-webapp, airbyte-server, airbyte-worker, airbyte-temporal
|
||||
- superset
|
||||
|
||||
## 🔑 Access Points
|
||||
|
||||
| Service | URL | Default Credentials |
|
||||
|---------|-----|---------------------|
|
||||
| **Nginx Proxy Manager** | http://localhost:81 | admin@example.com / changeme |
|
||||
| **Keycloak Admin** | http://localhost:8080 | See KEYCLOAK_ADMIN in .env.global |
|
||||
| **API Service** | http://localhost/apiservice | See ADMIN_USERNAME in .env.global |
|
||||
| **Airbyte** | http://localhost/airbyte | Configure via Nginx first |
|
||||
| **Superset** | http://localhost/superset | See SUPERSET_ADMIN_USERNAME in .env.global |
|
||||
|
||||
## 📝 Post-Installation Setup
|
||||
|
||||
### 1. Configure Nginx Proxy Manager
|
||||
|
||||
1. Access http://localhost:81
|
||||
2. Login with default credentials (change on first login)
|
||||
3. Add Proxy Hosts:
|
||||
|
||||
**API Service:**
|
||||
- Domain: `api.sriphat.local` (or your domain)
|
||||
- Forward Hostname: `apiservice`
|
||||
- Forward Port: `8000`
|
||||
- Custom locations:
|
||||
- Location: `/apiservice`
|
||||
- Forward Hostname: `apiservice`
|
||||
- Forward Port: `8000`
|
||||
|
||||
**Keycloak:**
|
||||
- Domain: `auth.sriphat.local`
|
||||
- Forward Hostname: `keycloak`
|
||||
- Forward Port: `8080`
|
||||
|
||||
**Superset:**
|
||||
- Domain: `bi.sriphat.local`
|
||||
- Forward Hostname: `superset`
|
||||
- Forward Port: `8088`
|
||||
|
||||
**Airbyte:**
|
||||
- Domain: `etl.sriphat.local`
|
||||
- Forward Hostname: `airbyte-webapp`
|
||||
- Forward Port: `8000`
|
||||
|
||||
### 2. Setup Keycloak SSO
|
||||
|
||||
1. Access Keycloak admin console
|
||||
2. Create new Realm: `sriphat`
|
||||
3. Create Clients:
|
||||
- **superset-client** (for Superset OAuth)
|
||||
- **apiservice-client** (for API Service)
|
||||
4. Configure OIDC settings
|
||||
5. Create Users and assign roles
|
||||
|
||||
### 3. Initialize API Service
|
||||
|
||||
```bash
|
||||
# Access admin UI
|
||||
# http://api.sriphat.local/apiservice/admin/
|
||||
|
||||
# Create API Client
|
||||
# 1. Go to ApiClient menu
|
||||
# 2. Create new client (e.g., "mobile-app")
|
||||
|
||||
# Generate API Key
|
||||
curl -X POST "http://api.sriphat.local/apiservice/admin/api-keys/generate?client_id=1&permissions=feed.checkpoint:write&name=production-key" \
|
||||
-H "Cookie: session=<your-admin-session>"
|
||||
|
||||
# Test API
|
||||
curl -X POST "http://api.sriphat.local/apiservice/api/v1/feed/checkpoint" \
|
||||
-H "Authorization: Bearer <api-key>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '[{"id":1,"hn":123,"vn":456,"location":"OPD","type":"Scan","timestamp_in":"2026-02-16T10:00:00","timestamp_out":null,"waiting_time":null,"bu":"SRIPHAT"}]'
|
||||
```
|
||||
|
||||
### 4. Configure Airbyte Sources
|
||||
|
||||
1. Access Airbyte UI
|
||||
2. Setup Sources:
|
||||
- SQL Server (HIS Database)
|
||||
- Oracle (Lab System)
|
||||
- REST API endpoints
|
||||
3. Setup Destination:
|
||||
- PostgreSQL (host: `postgres`, database: `postgres`, schemas: `raw_data`)
|
||||
4. Create Connections and schedule syncs
|
||||
|
||||
### 5. Setup Superset Dashboards
|
||||
|
||||
1. Access Superset
|
||||
2. Add Database Connection:
|
||||
- PostgreSQL: `postgresql://postgres:password@postgres:5432/postgres`
|
||||
3. Create Datasets from `analytics` schema
|
||||
4. Build Dashboards
|
||||
|
||||
## 🔒 Security Checklist
|
||||
|
||||
- [ ] Change all default passwords in `.env.global`
|
||||
- [ ] Enable SSL in Nginx Proxy Manager (Let's Encrypt)
|
||||
- [ ] Configure Keycloak with hospital LDAP/AD
|
||||
- [ ] Enable Row-Level Security (RLS) in PostgreSQL
|
||||
- [ ] Restrict network access (firewall rules)
|
||||
- [ ] Setup backup strategy for PostgreSQL data
|
||||
- [ ] Enable audit logging in all services
|
||||
- [ ] Configure session timeouts
|
||||
|
||||
## 🛠️ Maintenance
|
||||
|
||||
### View Logs
|
||||
```bash
|
||||
# All services
|
||||
docker compose -f 01-infra/docker-compose.yml logs -f
|
||||
|
||||
# Specific service
|
||||
docker logs -f apiservice
|
||||
docker logs -f keycloak
|
||||
docker logs -f superset
|
||||
```
|
||||
|
||||
### Backup Database
|
||||
```bash
|
||||
docker exec postgres pg_dump -U postgres postgres > backup_$(date +%Y%m%d).sql
|
||||
```
|
||||
|
||||
### Restore Database
|
||||
```bash
|
||||
docker exec -i postgres psql -U postgres postgres < backup_20260216.sql
|
||||
```
|
||||
|
||||
### Update Services
|
||||
```bash
|
||||
# Stop all
|
||||
bash stop-all.sh
|
||||
|
||||
# Pull latest images
|
||||
docker compose -f 01-infra/docker-compose.yml pull
|
||||
docker compose -f 04-ingestion/docker-compose.yml pull
|
||||
docker compose -f 06-analytics/docker-compose.yml pull
|
||||
|
||||
# Rebuild API service
|
||||
cd 03-apiservice
|
||||
docker compose --env-file ../.env.global build
|
||||
|
||||
# Start all
|
||||
cd ..
|
||||
bash start-all.sh
|
||||
```
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
|
||||
### PostgreSQL connection issues
|
||||
```bash
|
||||
# Check if PostgreSQL is ready
|
||||
docker exec postgres pg_isready -U postgres
|
||||
|
||||
# Check schemas
|
||||
docker exec postgres psql -U postgres -c "\dn"
|
||||
```
|
||||
|
||||
### Keycloak not starting
|
||||
```bash
|
||||
# Check logs
|
||||
docker logs keycloak
|
||||
|
||||
# Ensure PostgreSQL is ready first
|
||||
docker restart keycloak
|
||||
```
|
||||
|
||||
### API Service can't connect to DB
|
||||
```bash
|
||||
# Verify network
|
||||
docker network inspect shared_data_network
|
||||
|
||||
# Check environment variables
|
||||
docker exec apiservice env | grep DB_
|
||||
```
|
||||
|
||||
### Airbyte worker issues
|
||||
```bash
|
||||
# Ensure Docker socket is mounted
|
||||
docker exec airbyte-worker ls -la /var/run/docker.sock
|
||||
|
||||
# Check Temporal
|
||||
docker logs airbyte-temporal
|
||||
```
|
||||
|
||||
## 📊 Monitoring
|
||||
|
||||
### Resource Usage
|
||||
```bash
|
||||
docker stats
|
||||
```
|
||||
|
||||
### Health Checks
|
||||
```bash
|
||||
# PostgreSQL
|
||||
curl http://localhost:5432 || echo "PostgreSQL internal only - OK"
|
||||
|
||||
# Nginx Proxy Manager
|
||||
curl -I http://localhost:81
|
||||
|
||||
# Keycloak
|
||||
curl -I http://localhost:8080
|
||||
|
||||
# API Service (via network)
|
||||
docker exec nginx-proxy-manager curl -I http://apiservice:8000/apiservice/docs
|
||||
```
|
||||
|
||||
## 🔄 Scaling
|
||||
|
||||
### Increase API Service Workers
|
||||
Edit `03-apiservice/Dockerfile`:
|
||||
```dockerfile
|
||||
CMD ["gunicorn","-k","uvicorn.workers.UvicornWorker","app.main:app","--bind","0.0.0.0:8000","--workers","4"]
|
||||
```
|
||||
|
||||
### Add Read Replicas (PostgreSQL)
|
||||
- Configure streaming replication
|
||||
- Update connection strings for read-only queries
|
||||
|
||||
## 📞 Support
|
||||
|
||||
For issues:
|
||||
1. Check logs: `docker logs <container-name>`
|
||||
2. Verify network: `docker network inspect shared_data_network`
|
||||
3. Review configuration: `.env.global`
|
||||
4. Restart specific service: `docker restart <container-name>`
|
||||
Reference in New Issue
Block a user