境部署DolphinScheduler與SeaTunnel實戰(zhàn))
1. 項目背景與核心挑戰(zhàn)在傳統(tǒng)企業(yè)IT架構(gòu)向云原生轉(zhuǎn)型的過程中Kubernetesk8s已成為事實上的容器編排標準。然而在實際生產(chǎn)環(huán)境中由于安全合規(guī)要求大量企業(yè)需要在內(nèi)網(wǎng)隔離環(huán)境下部署云原生應(yīng)用套件。本次要解決的正是這樣一個典型場景在完全離線的k8s集群中部署DolphinScheduler工作流調(diào)度系統(tǒng)和SeaTunnel數(shù)據(jù)集成工具。這個組合方案特別適合需要處理以下需求的企業(yè)存在嚴格網(wǎng)絡(luò)安全隔離要求的金融、政務(wù)等行業(yè)需要自動化調(diào)度異構(gòu)數(shù)據(jù)源ETL流程的場景希望利用聲明式配置管理整個數(shù)據(jù)流水線的團隊關(guān)鍵難點提示離線環(huán)境意味著所有依賴鏡像、charts包、系統(tǒng)組件都需要預(yù)先下載并完成內(nèi)部倉庫的搭建任何遺漏都可能導致部署過程中斷。2. 離線環(huán)境準備工作2.1 基礎(chǔ)設(shè)施規(guī)劃建議建議采用以下節(jié)點規(guī)格作為基準可根據(jù)實際負載調(diào)整節(jié)點類型CPU內(nèi)存磁盤數(shù)量Master4核16G100G3Worker8核32G200G至少2網(wǎng)絡(luò)方面需要確保所有節(jié)點間網(wǎng)絡(luò)延遲2ms節(jié)點間帶寬≥1Gbps提前規(guī)劃好Pod CIDR和Service CIDR2.2 離線資源打包清單需要預(yù)先準備的離線資源包括基礎(chǔ)組件包k8s各節(jié)點所需系統(tǒng)依賴conntrack、socat等容器運行時推薦containerd 1.6Helm 3.10二進制文件鏡像倉庫方案# 搭建Harbor私有倉庫示例 helm repo add harbor https://helm.goharbor.io helm fetch harbor/harbor --version 1.10.0應(yīng)用鏡像清單DolphinScheduler 3.2.1全套鏡像含web、api、worker等SeaTunnel 2.3.2引擎鏡像相關(guān)中間件鏡像ZooKeeper、MySQL等Helm charts包helm repo add dolphinscheduler https://dolphinscheduler.apache.org helm pull dolphinscheduler/dolphinscheduler --version 3.2.13. 核心組件部署實戰(zhàn)3.1 DolphinScheduler部署要點3.1.1 數(shù)據(jù)庫初始化建議使用獨立的MySQL實例非k8s內(nèi)部署CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8mb4; CREATE USER ds_user% IDENTIFIED BY StrongPass123; GRANT ALL PRIVILEGES ON dolphinscheduler.* TO ds_user%;3.1.2 Helm定制化配置關(guān)鍵values.yaml配置項postgresql: enabled: false # 禁用內(nèi)置PG externalDatabase: type: mysql host: mysql.internal port: 3306 username: ds_user password: StrongPass123 registry: type: harbor url: harbor.internal repository: library/dolphinscheduler部署命令helm install dolphinscheduler ./dolphinscheduler-3.2.1.tgz \ -n ds --create-namespace \ -f values.yaml3.2 SeaTunnel集成方案3.2.1 引擎部署模式選擇推薦使用Spark on k8s模式spark: master: k8s://https://kubernetes.default.svc deployMode: cluster image: harbor.internal/library/seatunnel-spark:2.3.23.2.2 任務(wù)配置示例典型JDBC源到HDFS的作業(yè)配置env { execution.parallelism 3 } source { JdbcSource { url jdbc:mysql://mysql.internal:3306/source_db username etl_user password EtlPass456 query SELECT * FROM orders WHERE update_time ${last_update} } } transform { Sql { query SELECT user_id, SUM(amount) AS total FROM orders GROUP BY user_id } } sink { Hdfs { path hdfs://namenode:8020/data/orders_agg file_format parquet } }4. 生產(chǎn)環(huán)境調(diào)優(yōu)指南4.1 性能關(guān)鍵參數(shù)DolphinScheduler工作節(jié)點配置建議worker: resources: limits: cpu: 2 memory: 4Gi requests: cpu: 1 memory: 2Gi env: - name: TASK_EXECUTE_THREADS value: 10 # 根據(jù)CPU核數(shù)調(diào)整4.2 高可用保障措施為關(guān)鍵組件配置Pod反親和性affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app.kubernetes.io/component operator: In values: [master-server] topologyKey: kubernetes.io/hostname配置合理的健康檢查livenessProbe: httpGet: path: /actuator/health port: 12345 initialDelaySeconds: 30 periodSeconds: 105. 典型問題排查手冊5.1 鏡像拉取失敗現(xiàn)象Pod狀態(tài)為ImagePullBackOff 解決方案確認harbor證書已添加到各個節(jié)點mkdir -p /etc/docker/certs.d/harbor.internal cp harbor-ca.crt /etc/docker/certs.d/harbor.internal/ca.crt檢查secret配置是否正確kubectl create secret docker-registry harbor-secret \ --docker-serverharbor.internal \ --docker-usernameadmin \ --docker-passwordHarbor12345 \ -n ds5.2 權(quán)限相關(guān)問題現(xiàn)象SeaTunnel任務(wù)報Permission denied 處理方法為Spark Driver配置合適的安全上下文securityContext: runAsUser: 1000 fsGroup: 1000在HDFS端配置ACLhdfs dfs -setfacl -R -m user:spark:rwx /data6. 監(jiān)控與運維建議6.1 指標采集方案推薦使用Prometheus Operator采集指標DolphinScheduler暴露的指標端點metrics: enabled: true port: 12345對應(yīng)的ServiceMonitor配置apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: release: prometheus name: dolphinscheduler-monitor spec: endpoints: - port: metrics selector: matchLabels: app.kubernetes.io/instance: dolphinscheduler6.2 日志收集策略建議采用Filebeat ELK方案filebeatConfig: filebeat.yml: | filebeat.inputs: - type: container paths: - /var/log/containers/*.log output.elasticsearch: hosts: [elasticsearch.internal:9200] 離線K8s環(huán)境部署DolphinScheduler與SeaTunnel實戰(zhàn)
1. 項目背景與核心挑戰(zhàn)在傳統(tǒng)企業(yè)IT架構(gòu)向云原生轉(zhuǎn)型的過程中Kubernetesk8s已成為事實上的容器編排標準。然而在實際生產(chǎn)環(huán)境中由于安全合規(guī)要求大量企業(yè)需要在內(nèi)網(wǎng)隔離環(huán)境下部署云原生應(yīng)用套件。本次要解決的正是這樣一個典型場景在完全離線的k8s集群中部署DolphinScheduler工作流調(diào)度系統(tǒng)和SeaTunnel數(shù)據(jù)集成工具。這個組合方案特別適合需要處理以下需求的企業(yè)存在嚴格網(wǎng)絡(luò)安全隔離要求的金融、政務(wù)等行業(yè)需要自動化調(diào)度異構(gòu)數(shù)據(jù)源ETL流程的場景希望利用聲明式配置管理整個數(shù)據(jù)流水線的團隊關(guān)鍵難點提示離線環(huán)境意味著所有依賴鏡像、charts包、系統(tǒng)組件都需要預(yù)先下載并完成內(nèi)部倉庫的搭建任何遺漏都可能導致部署過程中斷。2. 離線環(huán)境準備工作2.1 基礎(chǔ)設(shè)施規(guī)劃建議建議采用以下節(jié)點規(guī)格作為基準可根據(jù)實際負載調(diào)整節(jié)點類型CPU內(nèi)存磁盤數(shù)量Master4核16G100G3Worker8核32G200G至少2網(wǎng)絡(luò)方面需要確保所有節(jié)點間網(wǎng)絡(luò)延遲2ms節(jié)點間帶寬≥1Gbps提前規(guī)劃好Pod CIDR和Service CIDR2.2 離線資源打包清單需要預(yù)先準備的離線資源包括基礎(chǔ)組件包k8s各節(jié)點所需系統(tǒng)依賴conntrack、socat等容器運行時推薦containerd 1.6Helm 3.10二進制文件鏡像倉庫方案# 搭建Harbor私有倉庫示例 helm repo add harbor https://helm.goharbor.io helm fetch harbor/harbor --version 1.10.0應(yīng)用鏡像清單DolphinScheduler 3.2.1全套鏡像含web、api、worker等SeaTunnel 2.3.2引擎鏡像相關(guān)中間件鏡像ZooKeeper、MySQL等Helm charts包helm repo add dolphinscheduler https://dolphinscheduler.apache.org helm pull dolphinscheduler/dolphinscheduler --version 3.2.13. 核心組件部署實戰(zhàn)3.1 DolphinScheduler部署要點3.1.1 數(shù)據(jù)庫初始化建議使用獨立的MySQL實例非k8s內(nèi)部署CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8mb4; CREATE USER ds_user% IDENTIFIED BY StrongPass123; GRANT ALL PRIVILEGES ON dolphinscheduler.* TO ds_user%;3.1.2 Helm定制化配置關(guān)鍵values.yaml配置項postgresql: enabled: false # 禁用內(nèi)置PG externalDatabase: type: mysql host: mysql.internal port: 3306 username: ds_user password: StrongPass123 registry: type: harbor url: harbor.internal repository: library/dolphinscheduler部署命令helm install dolphinscheduler ./dolphinscheduler-3.2.1.tgz \ -n ds --create-namespace \ -f values.yaml3.2 SeaTunnel集成方案3.2.1 引擎部署模式選擇推薦使用Spark on k8s模式spark: master: k8s://https://kubernetes.default.svc deployMode: cluster image: harbor.internal/library/seatunnel-spark:2.3.23.2.2 任務(wù)配置示例典型JDBC源到HDFS的作業(yè)配置env { execution.parallelism 3 } source { JdbcSource { url jdbc:mysql://mysql.internal:3306/source_db username etl_user password EtlPass456 query SELECT * FROM orders WHERE update_time ${last_update} } } transform { Sql { query SELECT user_id, SUM(amount) AS total FROM orders GROUP BY user_id } } sink { Hdfs { path hdfs://namenode:8020/data/orders_agg file_format parquet } }4. 生產(chǎn)環(huán)境調(diào)優(yōu)指南4.1 性能關(guān)鍵參數(shù)DolphinScheduler工作節(jié)點配置建議worker: resources: limits: cpu: 2 memory: 4Gi requests: cpu: 1 memory: 2Gi env: - name: TASK_EXECUTE_THREADS value: 10 # 根據(jù)CPU核數(shù)調(diào)整4.2 高可用保障措施為關(guān)鍵組件配置Pod反親和性affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app.kubernetes.io/component operator: In values: [master-server] topologyKey: kubernetes.io/hostname配置合理的健康檢查livenessProbe: httpGet: path: /actuator/health port: 12345 initialDelaySeconds: 30 periodSeconds: 105. 典型問題排查手冊5.1 鏡像拉取失敗現(xiàn)象Pod狀態(tài)為ImagePullBackOff 解決方案確認harbor證書已添加到各個節(jié)點mkdir -p /etc/docker/certs.d/harbor.internal cp harbor-ca.crt /etc/docker/certs.d/harbor.internal/ca.crt檢查secret配置是否正確kubectl create secret docker-registry harbor-secret \ --docker-serverharbor.internal \ --docker-usernameadmin \ --docker-passwordHarbor12345 \ -n ds5.2 權(quán)限相關(guān)問題現(xiàn)象SeaTunnel任務(wù)報Permission denied 處理方法為Spark Driver配置合適的安全上下文securityContext: runAsUser: 1000 fsGroup: 1000在HDFS端配置ACLhdfs dfs -setfacl -R -m user:spark:rwx /data6. 監(jiān)控與運維建議6.1 指標采集方案推薦使用Prometheus Operator采集指標DolphinScheduler暴露的指標端點metrics: enabled: true port: 12345對應(yīng)的ServiceMonitor配置apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: release: prometheus name: dolphinscheduler-monitor spec: endpoints: - port: metrics selector: matchLabels: app.kubernetes.io/instance: dolphinscheduler6.2 日志收集策略建議采用Filebeat ELK方案filebeatConfig: filebeat.yml: | filebeat.inputs: - type: container paths: - /var/log/containers/*.log output.elasticsearch: hosts: [elasticsearch.internal:9200]