add files infra docker service for data platform

README-UBUNTU.md

@@ -0,0 +1,384 @@
+# Sriphat Data Platform - Ubuntu Server Installation Guide
+
+## 📋 System Requirements
+
+- **OS**: Ubuntu Server 20.04 LTS or 22.04 LTS
+- **RAM**: Minimum 8GB (16GB recommended)
+- **Disk**: 50GB free space
+- **CPU**: 4 cores (8 cores recommended)
+- **Network**: Static IP recommended
+
+## 🚀 Quick Install (Recommended)
+
+### Option 1: Automated Installation
+
+```bash
+# Clone repository
+git clone <repository-url> /opt/sriphat-dataplatform
+cd /opt/sriphat-dataplatform
+
+# Run install script
+bash install.sh
+```
+
+Script จะทำให้อัตโนมัติ:
+- ติดตั้ง Docker และ Docker Compose
+- สร้าง .env.global พร้อม random passwords
+- สร้าง backup directory
+- รัน services ทั้งหมด
+
+### Option 2: Manual Installation
+
+#### Step 1: Setup Ubuntu Server
+
+```bash
+# Update system
+sudo apt-get update
+sudo apt-get upgrade -y
+
+# Run setup script
+sudo bash setup-ubuntu.sh
+```
+
+#### Step 2: Logout and Login
+
+```bash
+# Logout to apply docker group permissions
+exit
+
+# Login again via SSH
+ssh user@server
+```
+
+#### Step 3: Configure Environment
+
+```bash
+cd /opt/sriphat-dataplatform
+
+# Copy and edit .env.global
+cp .env.global.example .env.global
+nano .env.global
+
+# Update these values:
+# - DB_PASSWORD (strong password)
+# - KEYCLOAK_ADMIN_PASSWORD
+# - SUPERSET_SECRET_KEY
+# - ADMIN_SECRET_KEY
+# - ADMIN_PASSWORD
+```
+
+#### Step 4: Start Services
+
+```bash
+# Make scripts executable
+chmod +x *.sh
+chmod +x 00-network/*.sh
+
+# Start all services
+bash start-all.sh
+```
+
+## 🔧 Post-Installation
+
+### 1. Check Services Status
+
+```bash
+# View running containers
+docker ps
+
+# Check logs
+docker logs nginx-proxy-manager
+docker logs keycloak
+docker logs postgres
+docker logs apiservice
+```
+
+### 2. Access Services
+
+```bash
+# Get server IP
+hostname -I
+
+# Access points:
+# - Nginx Proxy Manager: http://<server-ip>:81
+# - Keycloak: http://<server-ip>:8080
+```
+
+### 3. Configure Firewall (if needed)
+
+```bash
+# Check firewall status
+sudo ufw status
+
+# Allow additional ports if needed
+sudo ufw allow 8088/tcp  # Superset (if direct access needed)
+```
+
+### 4. Setup Domain Names
+
+In Nginx Proxy Manager (port 81):
+1. Add Proxy Hosts for each service
+2. Configure SSL with Let's Encrypt
+3. Point your domain DNS to server IP
+
+## 📦 Directory Structure
+
+```bash
+/opt/sriphat-dataplatform/          # Main directory
+├── 01-infra/
+│   └── data/postgres/              # PostgreSQL data
+├── 04-ingestion/
+│   └── data/                       # Airbyte data
+├── 06-analytics/
+│   └── data/                       # Superset data
+└── /backups/sriphat-data/          # Backup location
+```
+
+## 🔄 Backup Setup
+
+### Automatic Daily Backup
+
+```bash
+# Edit crontab
+crontab -e
+
+# Add this line (backup at 2 AM daily)
+0 2 * * * /opt/sriphat-dataplatform/backup-daily.sh
+
+# Verify cron job
+crontab -l
+```
+
+### Manual Backup
+
+```bash
+# Run backup script
+bash backup-daily.sh
+
+# Or backup manually
+bash stop-all.sh
+sudo tar -czf /backups/sriphat-backup-$(date +%Y%m%d).tar.gz /opt/sriphat-dataplatform
+bash start-all.sh
+```
+
+## 🛠️ Maintenance Commands
+
+### Start/Stop Services
+
+```bash
+# Start all
+bash start-all.sh
+
+# Stop all
+bash stop-all.sh
+
+# Restart specific service
+cd 03-apiservice
+docker compose --env-file ../.env.global restart
+```
+
+### View Logs
+
+```bash
+# All services
+docker compose -f 01-infra/docker-compose.yml logs -f
+
+# Specific service
+docker logs -f apiservice
+docker logs -f postgres
+```
+
+### Update Services
+
+```bash
+# Stop services
+bash stop-all.sh
+
+# Pull latest images
+docker compose -f 01-infra/docker-compose.yml pull
+docker compose -f 04-ingestion/docker-compose.yml pull
+docker compose -f 06-analytics/docker-compose.yml pull
+
+# Rebuild API service
+cd 03-apiservice
+docker compose --env-file ../.env.global build --no-cache
+
+# Start services
+cd ..
+bash start-all.sh
+```
+
+### Clean Up
+
+```bash
+# Remove unused images
+docker image prune -a
+
+# Remove unused volumes (careful!)
+docker volume prune
+
+# Clean build cache
+docker builder prune
+```
+
+## 🐛 Troubleshooting
+
+### Docker Permission Denied
+
+```bash
+# Add user to docker group
+sudo usermod -aG docker $USER
+
+# Logout and login again
+exit
+```
+
+### Port Already in Use
+
+```bash
+# Check what's using the port
+sudo netstat -tulpn | grep :80
+sudo netstat -tulpn | grep :8080
+
+# Kill process or change port in docker-compose.yml
+```
+
+### PostgreSQL Won't Start
+
+```bash
+# Check logs
+docker logs postgres
+
+# Check permissions
+sudo chown -R 999:999 01-infra/data/postgres
+
+# Restart
+docker restart postgres
+```
+
+### Services Can't Connect to PostgreSQL
+
+```bash
+# Check network
+docker network inspect shared_data_network
+
+# Verify PostgreSQL is ready
+docker exec postgres pg_isready -U postgres
+
+# Restart dependent services
+cd 03-apiservice
+docker compose --env-file ../.env.global restart
+```
+
+### Disk Space Issues
+
+```bash
+# Check disk usage
+df -h
+
+# Check Docker disk usage
+docker system df
+
+# Clean up
+docker system prune -a --volumes
+```
+
+## 🔒 Security Hardening
+
+### 1. Change Default Passwords
+
+```bash
+# Edit .env.global
+nano .env.global
+
+# Update all passwords
+# Restart services
+bash stop-all.sh
+bash start-all.sh
+```
+
+### 2. Setup SSL
+
+In Nginx Proxy Manager:
+1. Add domain
+2. Request SSL certificate (Let's Encrypt)
+3. Force SSL redirect
+
+### 3. Restrict Firewall
+
+```bash
+# Close unnecessary ports after Nginx setup
+sudo ufw delete allow 8080/tcp  # Keycloak (access via Nginx only)
+
+# Allow only from specific IPs
+sudo ufw allow from 192.168.1.0/24 to any port 81
+```
+
+### 4. Enable Fail2ban
+
+```bash
+# Install fail2ban
+sudo apt-get install fail2ban
+
+# Configure for SSH
+sudo systemctl enable fail2ban
+sudo systemctl start fail2ban
+```
+
+## 📊 Monitoring
+
+### System Resources
+
+```bash
+# Real-time monitoring
+htop
+
+# Docker stats
+docker stats
+
+# Disk usage
+df -h
+du -sh /opt/sriphat-dataplatform/*
+```
+
+### Service Health
+
+```bash
+# Check all containers
+docker ps -a
+
+# Check specific service health
+docker inspect --format='{{.State.Health.Status}}' postgres
+```
+
+## 🔄 Migration from Windows
+
+If migrating from Windows development:
+
+```bash
+# 1. Backup data on Windows
+# (use backup-daily.ps1)
+
+# 2. Copy backup to Ubuntu
+scp backup-*.zip user@ubuntu-server:/tmp/
+
+# 3. Extract on Ubuntu
+cd /opt/sriphat-dataplatform
+unzip /tmp/backup-*.zip
+
+# 4. Fix permissions
+sudo chown -R $USER:$USER .
+sudo chown -R 999:999 01-infra/data/postgres
+
+# 5. Start services
+bash start-all.sh
+```
+
+## 📞 Support
+
+For issues:
+1. Check logs: `docker logs <container-name>`
+2. Verify network: `docker network inspect shared_data_network`
+3. Check disk space: `df -h`
+4. Review firewall: `sudo ufw status`
+5. Consult DEPLOYMENT.md for detailed troubleshooting