add files infra docker service for data platform
This commit is contained in:
jigoong
277
BACKUP.md
Normal file
277
BACKUP.md
Normal file
@@ -0,0 +1,277 @@
|
||||
# Backup & Restore Guide
|
||||
|
||||
## 📂 โครงสร้าง Data Folders
|
||||
|
||||
ทุก service ใช้ bind mount (./data) เพื่อให้ backup ง่าย:
|
||||
|
||||
```
|
||||
sriphat-dataplatform/
|
||||
├── 01-infra/
|
||||
│ ├── data/
|
||||
│ │ └── postgres/ # PostgreSQL database files
|
||||
│ ├── data/ # Nginx Proxy Manager config
|
||||
│ └── letsencrypt/ # SSL certificates
|
||||
├── 04-ingestion/
|
||||
│ └── data/
|
||||
│ ├── workspace/ # Airbyte workspace
|
||||
│ └── airbyte/ # Airbyte metadata
|
||||
└── 06-analytics/
|
||||
└── data/
|
||||
└── superset_home/ # Superset config & metadata
|
||||
```
|
||||
|
||||
## 🔄 Backup Strategy
|
||||
|
||||
### Option 1: Full Backup (Recommended)
|
||||
|
||||
สำรองทั้งโปรเจกต์ (รวม config + data):
|
||||
|
||||
```bash
|
||||
# หยุด services ก่อน (เพื่อความสมบูรณ์ของข้อมูล)
|
||||
bash stop-all.sh
|
||||
|
||||
# Backup ทั้งโฟลเดอร์
|
||||
cd e:\git3
|
||||
tar -czf sriphat-dataplatform-backup-$(date +%Y%m%d-%H%M%S).tar.gz sriphat-dataplatform/
|
||||
|
||||
# หรือใช้ robocopy บน Windows
|
||||
robocopy sriphat-dataplatform E:\backups\sriphat-dataplatform-$(Get-Date -Format 'yyyyMMdd-HHmmss') /MIR /R:3 /W:5
|
||||
|
||||
# รัน services ต่อ
|
||||
bash start-all.sh
|
||||
```
|
||||
|
||||
### Option 2: Backup เฉพาะ Data Folders
|
||||
|
||||
```bash
|
||||
# สร้างโฟลเดอร์ backup
|
||||
mkdir -p E:\backups\data-$(date +%Y%m%d)
|
||||
|
||||
# Backup PostgreSQL
|
||||
docker exec postgres pg_dumpall -U postgres > E:\backups\data-$(date +%Y%m%d)\postgres-dump.sql
|
||||
|
||||
# Backup data folders
|
||||
robocopy 01-infra\data E:\backups\data-$(date +%Y%m%d)\01-infra-data /MIR
|
||||
robocopy 01-infra\letsencrypt E:\backups\data-$(date +%Y%m%d)\01-infra-letsencrypt /MIR
|
||||
robocopy 04-ingestion\data E:\backups\data-$(date +%Y%m%d)\04-ingestion-data /MIR
|
||||
robocopy 06-analytics\data E:\backups\data-$(date +%Y%m%d)\06-analytics-data /MIR
|
||||
```
|
||||
|
||||
### Option 3: Hot Backup (ไม่ต้องหยุด service)
|
||||
|
||||
```bash
|
||||
# Backup PostgreSQL (แบบ online)
|
||||
docker exec postgres pg_dumpall -U postgres | gzip > postgres-backup-$(date +%Y%m%d).sql.gz
|
||||
|
||||
# Backup Nginx config
|
||||
tar -czf nginx-backup-$(date +%Y%m%d).tar.gz 01-infra/data 01-infra/letsencrypt
|
||||
|
||||
# Backup Airbyte
|
||||
tar -czf airbyte-backup-$(date +%Y%m%d).tar.gz 04-ingestion/data
|
||||
|
||||
# Backup Superset
|
||||
tar -czf superset-backup-$(date +%Y%m%d).tar.gz 06-analytics/data
|
||||
```
|
||||
|
||||
## 📥 Restore
|
||||
|
||||
### Full Restore
|
||||
|
||||
```bash
|
||||
# หยุด services ทั้งหมด
|
||||
bash stop-all.sh
|
||||
|
||||
# ลบข้อมูลเก่า (ระวัง!)
|
||||
rm -rf 01-infra/data 04-ingestion/data 06-analytics/data
|
||||
|
||||
# แตกไฟล์ backup
|
||||
tar -xzf sriphat-dataplatform-backup-YYYYMMDD-HHMMSS.tar.gz
|
||||
|
||||
# รัน services
|
||||
bash start-all.sh
|
||||
```
|
||||
|
||||
### Restore PostgreSQL Only
|
||||
|
||||
```bash
|
||||
# หยุด services ที่ใช้ database
|
||||
cd 03-apiservice && docker compose down
|
||||
cd ../04-ingestion && docker compose down
|
||||
cd ../06-analytics && docker compose down
|
||||
|
||||
# Restore database
|
||||
docker exec -i postgres psql -U postgres < postgres-backup-20260216.sql
|
||||
|
||||
# หรือ restore จาก dump file
|
||||
gunzip < postgres-backup-20260216.sql.gz | docker exec -i postgres psql -U postgres
|
||||
|
||||
# รัน services ต่อ
|
||||
cd ../03-apiservice && docker compose --env-file ../.env.global up -d
|
||||
cd ../04-ingestion && docker compose --env-file ../.env.global up -d
|
||||
cd ../06-analytics && docker compose --env-file ../.env.global up -d
|
||||
```
|
||||
|
||||
### Restore Specific Service
|
||||
|
||||
```bash
|
||||
# หยุด service
|
||||
cd 06-analytics
|
||||
docker compose down
|
||||
|
||||
# Restore data
|
||||
rm -rf data/superset_home
|
||||
tar -xzf ../backups/superset-backup-20260216.tar.gz
|
||||
|
||||
# รัน service
|
||||
docker compose --env-file ../.env.global up -d
|
||||
```
|
||||
|
||||
## ⏰ Automated Backup (Scheduled)
|
||||
|
||||
### Windows Task Scheduler
|
||||
|
||||
สร้าง script `backup-daily.ps1`:
|
||||
|
||||
```powershell
|
||||
# backup-daily.ps1
|
||||
$BackupPath = "E:\backups\sriphat-data"
|
||||
$Date = Get-Date -Format "yyyyMMdd-HHmmss"
|
||||
$BackupFolder = "$BackupPath\$Date"
|
||||
|
||||
# สร้างโฟลเดอร์
|
||||
New-Item -ItemType Directory -Path $BackupFolder -Force
|
||||
|
||||
# Backup PostgreSQL
|
||||
docker exec postgres pg_dumpall -U postgres | Out-File "$BackupFolder\postgres.sql"
|
||||
|
||||
# Backup data folders
|
||||
robocopy "E:\git3\sriphat-dataplatform\01-infra\data" "$BackupFolder\01-infra-data" /MIR /R:3 /W:5
|
||||
robocopy "E:\git3\sriphat-dataplatform\01-infra\letsencrypt" "$BackupFolder\01-infra-letsencrypt" /MIR /R:3 /W:5
|
||||
robocopy "E:\git3\sriphat-dataplatform\04-ingestion\data" "$BackupFolder\04-ingestion-data" /MIR /R:3 /W:5
|
||||
robocopy "E:\git3\sriphat-dataplatform\06-analytics\data" "$BackupFolder\06-analytics-data" /MIR /R:3 /W:5
|
||||
|
||||
# Compress
|
||||
Compress-Archive -Path $BackupFolder -DestinationPath "$BackupPath\backup-$Date.zip"
|
||||
|
||||
# ลบ backup เก่า (เก็บไว้ 30 วัน)
|
||||
Get-ChildItem $BackupPath -Filter "backup-*.zip" | Where-Object {$_.LastWriteTime -lt (Get-Date).AddDays(-30)} | Remove-Item
|
||||
```
|
||||
|
||||
ตั้งเวลารัน Task Scheduler:
|
||||
1. เปิด Task Scheduler
|
||||
2. Create Basic Task
|
||||
3. Trigger: Daily เวลา 02:00
|
||||
4. Action: Start a program
|
||||
- Program: `powershell.exe`
|
||||
- Arguments: `-ExecutionPolicy Bypass -File "E:\git3\sriphat-dataplatform\backup-daily.ps1"`
|
||||
|
||||
### Linux Cron Job
|
||||
|
||||
```bash
|
||||
# เพิ่มใน crontab
|
||||
crontab -e
|
||||
|
||||
# Backup ทุกวันเวลา 02:00
|
||||
0 2 * * * /path/to/sriphat-dataplatform/backup-daily.sh
|
||||
```
|
||||
|
||||
สร้าง `backup-daily.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
BACKUP_DIR="/backups/sriphat-data"
|
||||
DATE=$(date +%Y%m%d-%H%M%S)
|
||||
|
||||
mkdir -p $BACKUP_DIR/$DATE
|
||||
|
||||
# Backup PostgreSQL
|
||||
docker exec postgres pg_dumpall -U postgres | gzip > $BACKUP_DIR/$DATE/postgres.sql.gz
|
||||
|
||||
# Backup data folders
|
||||
tar -czf $BACKUP_DIR/$DATE/01-infra-data.tar.gz 01-infra/data 01-infra/letsencrypt
|
||||
tar -czf $BACKUP_DIR/$DATE/04-ingestion-data.tar.gz 04-ingestion/data
|
||||
tar -czf $BACKUP_DIR/$DATE/06-analytics-data.tar.gz 06-analytics/data
|
||||
|
||||
# ลบ backup เก่า (เก็บไว้ 30 วัน)
|
||||
find $BACKUP_DIR -name "*.tar.gz" -mtime +30 -delete
|
||||
```
|
||||
|
||||
## 🔐 Backup Security
|
||||
|
||||
### Encrypt Backup
|
||||
|
||||
```bash
|
||||
# Backup และ encrypt ด้วย GPG
|
||||
tar -czf - 01-infra/data | gpg --symmetric --cipher-algo AES256 -o backup-encrypted-$(date +%Y%m%d).tar.gz.gpg
|
||||
|
||||
# Decrypt และ restore
|
||||
gpg --decrypt backup-encrypted-20260216.tar.gz.gpg | tar -xzf -
|
||||
```
|
||||
|
||||
### Remote Backup
|
||||
|
||||
```bash
|
||||
# Sync ไปยัง remote server (rsync)
|
||||
rsync -avz --delete E:\git3\sriphat-dataplatform\01-infra\data\ user@backup-server:/backups/sriphat/01-infra-data/
|
||||
|
||||
# หรือใช้ rclone (Google Drive, OneDrive, S3)
|
||||
rclone sync E:\git3\sriphat-dataplatform\01-infra\data remote:sriphat-backup/01-infra-data
|
||||
```
|
||||
|
||||
## 📊 Backup Checklist
|
||||
|
||||
- [ ] PostgreSQL database (pg_dumpall)
|
||||
- [ ] Nginx Proxy Manager config (01-infra/data)
|
||||
- [ ] SSL certificates (01-infra/letsencrypt)
|
||||
- [ ] Airbyte connections (04-ingestion/data)
|
||||
- [ ] Superset dashboards (06-analytics/data)
|
||||
- [ ] Environment files (.env.global)
|
||||
- [ ] Custom configs (superset_config.py, etc.)
|
||||
|
||||
## 🚨 Disaster Recovery
|
||||
|
||||
### Scenario 1: PostgreSQL Corruption
|
||||
|
||||
```bash
|
||||
# หยุด services
|
||||
bash stop-all.sh
|
||||
|
||||
# ลบ data folder
|
||||
rm -rf 01-infra/data/postgres
|
||||
|
||||
# Restore จาก backup
|
||||
docker compose -f 01-infra/docker-compose.yml --env-file .env.global up -d postgres
|
||||
sleep 10
|
||||
gunzip < postgres-backup-latest.sql.gz | docker exec -i postgres psql -U postgres
|
||||
|
||||
# รัน services ทั้งหมด
|
||||
bash start-all.sh
|
||||
```
|
||||
|
||||
### Scenario 2: Complete System Failure
|
||||
|
||||
```bash
|
||||
# ติดตั้ง Docker ใหม่
|
||||
# Clone repository
|
||||
git clone <repo-url> sriphat-dataplatform
|
||||
cd sriphat-dataplatform
|
||||
|
||||
# Restore backup
|
||||
tar -xzf /path/to/backup.tar.gz
|
||||
|
||||
# Start
|
||||
bash start-all.sh
|
||||
```
|
||||
|
||||
## 📝 Best Practices
|
||||
|
||||
1. **Backup ทุกวัน** - ตั้ง automated backup
|
||||
2. **Test restore** - ทดสอบ restore อย่างน้อยเดือนละครั้ง
|
||||
3. **3-2-1 Rule**:
|
||||
- 3 copies ของข้อมูล
|
||||
- 2 media types ที่แตกต่างกัน
|
||||
- 1 offsite backup
|
||||
4. **Monitor backup** - ตรวจสอบว่า backup สำเร็จทุกวัน
|
||||
5. **Document** - บันทึกขั้นตอน restore ไว้ชัดเจน
|
||||
6. **Encrypt** - เข้ารหัส backup ที่มีข้อมูลสำคัญ
|
||||
7. **Version control** - เก็บ backup หลายเวอร์ชัน (อย่างน้อย 30 วัน)
|
||||
Reference in New Issue
Block a user