feat: MinIO integration — bucket finance, API service upload, Nginx routing
- 01-infra/nginx-configs: add MinIO /minio/ and /minio-console/ location blocks (port 9000 S3 API, port 9001 Console UI, path stripping via rewrite) - 03-apiservice: integrate MinIO minio-python SDK for file upload - requirements.txt: add minio==7.2.11 - app/core/config.py: add MINIO_ENDPOINT, ACCESS_KEY, SECRET_KEY, BUCKET_FINANCE, USE_SSL - app/services/minio_client.py: new — upload_file(), get_presigned_url(), delete_file() - app/routes/pages.py: replace local /data/uploads/ write with MinIO upload to finance bucket - docker-compose.yml: pass MinIO env vars to container - .env.example: document MinIO vars - 07-minio/.env.example: add MINIO_SVC_ACCESS_KEY/SECRET_KEY section - 07-minio/README.md: add Python minio SDK and Airflow DAG usage guide - CLAUDE.md: project context (servers, SSH, paths, service distribution) - document-obsidiant/: initial Obsidian docs for all services
This commit is contained in:
jigoong
@@ -0,0 +1,264 @@
|
||||
---
|
||||
tags:
|
||||
- project/sriphat
|
||||
- minio
|
||||
- storage
|
||||
- s3
|
||||
created: 2026-05-07
|
||||
status: active
|
||||
folder: 07-minio
|
||||
---
|
||||
|
||||
# MinIO Object Storage (07-minio)
|
||||
|
||||
> **Docker Compose:** `07-minio/docker-compose.yml`
|
||||
> **Env File:** `07-minio/.env`
|
||||
|
||||
## Overview
|
||||
|
||||
MinIO เป็น S3-compatible object storage สำหรับ:
|
||||
- เก็บ raw data files (CSV, JSON, Parquet)
|
||||
- เก็บ ML/AI models และ training data
|
||||
- เก็บ backups และ reports
|
||||
- Keycloak SSO integration
|
||||
|
||||
---
|
||||
|
||||
## Container
|
||||
|
||||
| รายการ | ค่า |
|
||||
|--------|-----|
|
||||
| **Container** | `minio` |
|
||||
| **Image** | `minio/minio:latest` |
|
||||
| **API Port** | `9000:9000` |
|
||||
| **Console Port** | `9001:9001` |
|
||||
| **Console URL** | `https://ai.sriphat.com/minio-console` |
|
||||
| **API URL** | `https://ai.sriphat.com/minio` |
|
||||
| **Direct (Dev)** | `http://192.168.100.9:9001` (console) |
|
||||
| **Region** | `ap-southeast-1` |
|
||||
|
||||
---
|
||||
|
||||
## Use Cases
|
||||
|
||||
| Use Case | ตัวอย่าง |
|
||||
|----------|---------|
|
||||
| **Data Lake** | Raw CSV, JSON, Parquet จาก Airbyte |
|
||||
| **ML/AI Workflows** | Model files, training datasets, experiment artifacts |
|
||||
| **Backup Storage** | Database dumps, application backups |
|
||||
| **Report Files** | Excel, PDF reports จาก Finance |
|
||||
| **Media Storage** | Images, documents จากระบบ HIS |
|
||||
| **Application Storage** | File uploads จาก API Service |
|
||||
|
||||
---
|
||||
|
||||
## Authentication
|
||||
|
||||
### 1. Root Credentials (Default)
|
||||
|
||||
```bash
|
||||
MINIO_ROOT_USER=minioadmin
|
||||
MINIO_ROOT_PASSWORD=<strong-password>
|
||||
```
|
||||
|
||||
### 2. Keycloak SSO (แนะนำ)
|
||||
|
||||
เชื่อมต่อผ่าน OpenID Connect:
|
||||
|
||||
```bash
|
||||
MINIO_IDENTITY_OPENID_CONFIG_URL=https://ai.sriphat.com/keycloak/realms/sriphat/.well-known/openid-configuration
|
||||
MINIO_IDENTITY_OPENID_CLIENT_ID=minio-client
|
||||
MINIO_IDENTITY_OPENID_CLIENT_SECRET=<secret>
|
||||
MINIO_IDENTITY_OPENID_CLAIM_NAME=policy
|
||||
MINIO_IDENTITY_OPENID_SCOPES=openid,profile,email
|
||||
MINIO_IDENTITY_OPENID_REDIRECT_URI=https://ai.sriphat.com/minio-console/oauth_callback
|
||||
```
|
||||
|
||||
**Policy Mapping:** User ใน Keycloak ต้องมี attribute `policy` ที่ map กับ MinIO policy
|
||||
|
||||
---
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
# Credentials
|
||||
MINIO_ROOT_USER=minioadmin
|
||||
MINIO_ROOT_PASSWORD=<secret>
|
||||
|
||||
# URLs
|
||||
MINIO_SERVER_URL=https://ai.sriphat.com/minio
|
||||
MINIO_BROWSER_REDIRECT_URL=https://ai.sriphat.com/minio-console
|
||||
|
||||
# Region
|
||||
MINIO_REGION=ap-southeast-1
|
||||
|
||||
# Keycloak SSO
|
||||
MINIO_IDENTITY_OPENID_CONFIG_URL=<keycloak-oidc-url>
|
||||
MINIO_IDENTITY_OPENID_CLIENT_ID=<client-id>
|
||||
MINIO_IDENTITY_OPENID_CLIENT_SECRET=<secret>
|
||||
MINIO_IDENTITY_OPENID_CLAIM_NAME=policy
|
||||
MINIO_IDENTITY_OPENID_SCOPES=openid,profile,email
|
||||
MINIO_IDENTITY_OPENID_REDIRECT_URI=<redirect-uri>
|
||||
|
||||
TZ=Asia/Bangkok
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Volume Mounts
|
||||
|
||||
```
|
||||
07-minio/
|
||||
├── data/ → /data (object storage data)
|
||||
└── certs/ → /root/.minio/certs:ro (SSL certificates)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## การใช้งาน MinIO Client (mc)
|
||||
|
||||
```bash
|
||||
# Install
|
||||
wget https://dl.min.io/client/mc/release/linux-amd64/mc
|
||||
chmod +x mc && sudo mv mc /usr/local/bin/
|
||||
|
||||
# Config alias
|
||||
mc alias set sriphat https://ai.sriphat.com/minio minioadmin <password>
|
||||
|
||||
# List buckets
|
||||
mc ls sriphat
|
||||
|
||||
# Create bucket
|
||||
mc mb sriphat/raw-data
|
||||
mc mb sriphat/ml-models
|
||||
mc mb sriphat/backups
|
||||
mc mb sriphat/reports
|
||||
|
||||
# Upload
|
||||
mc cp data.csv sriphat/raw-data/
|
||||
mc cp -r ./models/ sriphat/ml-models/
|
||||
|
||||
# Set bucket policy
|
||||
mc anonymous set none sriphat/raw-data # private
|
||||
mc anonymous set download sriphat/public # public read
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Python SDK (boto3)
|
||||
|
||||
```python
|
||||
import boto3
|
||||
from botocore.client import Config
|
||||
|
||||
s3 = boto3.client(
|
||||
's3',
|
||||
endpoint_url='https://ai.sriphat.com/minio',
|
||||
aws_access_key_id='minioadmin',
|
||||
aws_secret_access_key='<password>',
|
||||
config=Config(signature_version='s3v4'),
|
||||
region_name='ap-southeast-1'
|
||||
)
|
||||
|
||||
# Upload file
|
||||
s3.upload_file('data.csv', 'raw-data', 'data.csv')
|
||||
|
||||
# Download file
|
||||
s3.download_file('raw-data', 'data.csv', 'local-data.csv')
|
||||
|
||||
# List objects
|
||||
for obj in s3.list_objects_v2(Bucket='raw-data').get('Contents', []):
|
||||
print(obj['Key'])
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Recommended Bucket Structure
|
||||
|
||||
```
|
||||
sriphat/
|
||||
├── raw-data/ # ข้อมูลดิบจาก Airbyte / HIS
|
||||
│ ├── his/
|
||||
│ ├── oracle-lab/
|
||||
│ └── finance-excel/
|
||||
├── processed-data/ # ข้อมูลที่ transform แล้ว
|
||||
├── ml-models/ # ML/AI model files
|
||||
│ ├── waiting-time/
|
||||
│ └── patient-flow/
|
||||
├── reports/ # Excel, PDF reports
|
||||
├── backups/ # Database backups
|
||||
│ └── postgres/
|
||||
└── uploads/ # User uploads จาก API Service
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security
|
||||
|
||||
```bash
|
||||
# สร้าง read-only policy
|
||||
cat > readonly-policy.json << 'EOF'
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["s3:GetObject", "s3:ListBucket"],
|
||||
"Resource": ["arn:aws:s3:::*"]
|
||||
}
|
||||
]
|
||||
}
|
||||
EOF
|
||||
|
||||
mc admin policy create sriphat readonly-policy readonly-policy.json
|
||||
|
||||
# Assign policy ให้ user
|
||||
mc admin policy attach sriphat readonly-policy --user=analyst-user
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Health Check
|
||||
|
||||
```bash
|
||||
# ตรวจสอบสถานะ
|
||||
curl -f http://localhost:9000/minio/health/live
|
||||
docker exec minio curl -f http://localhost:9000/minio/health/live
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Backup Strategy
|
||||
|
||||
```bash
|
||||
# Backup data directory
|
||||
tar -czf minio-backup-$(date +%Y%m%d).tar.gz 07-minio/data/
|
||||
|
||||
# Sync to remote
|
||||
rsync -avz 07-minio/data/ backup-server:/backups/minio/
|
||||
|
||||
# Restore
|
||||
docker compose down
|
||||
tar -xzf minio-backup-20260501.tar.gz
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Keycloak Setup (สำหรับ SSO)
|
||||
|
||||
ดูรายละเอียดที่ `07-minio/KEYCLOAK_INTEGRATION.md`
|
||||
|
||||
1. สร้าง Client `minio-client` ใน Keycloak Realm `sriphat`
|
||||
2. ตั้งค่า Valid Redirect URIs: `https://ai.sriphat.com/minio-console/oauth_callback`
|
||||
3. สร้าง Client Scope `minio-policy`
|
||||
4. เพิ่ม User Attribute Mapper `policy`
|
||||
5. กำหนด `policy` attribute ให้กับ users ตาม MinIO policies
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- [[00-Project-Overview]]
|
||||
- [[01-Infrastructure]] (Keycloak SSO)
|
||||
- [[07-Security-Strategy]]
|
||||
Reference in New Issue
Block a user