• 日期: 2026-08-04
  • 标签: 程序员

  • 升级手册

    # ArgoCD 升级操作手册

> 适用范围：EKS 生产集群 ArgoCD `v3.0.6 → v3.1.x → v3.2.x` 两跳升级  
> 生产命名空间：`argocd` | 演练命名空间：`argocd-test`

---

## 目录

1. [目录结构](#目录结构)
2. [前置条件](#前置条件)
3. [升级 YAML 准备](#升级-yaml-准备)
4. [生产升级流程](#生产升级流程)
5. [回滚流程](#回滚流程)
6. [从备份完整恢复](#从备份完整恢复)
7. [升级后巡检](#升级后巡检)
8. [跨集群 RBAC 修复](#跨集群-rbac-修复)
9. [演练流程（argocd-test）](#演练流程argocd-test)
10. [关键配置说明](#关键配置说明)
11. [常见问题](#常见问题)

---

## 目录结构

```
argocd-jenkins/
├── README.md                          # 本手册
├── argocd-check-v32.sh                # 升级后只读巡检（生产/演练通用）
├── argocd-upgrade-post-v31-prod.sh    # 生产第一跳升级后修复（NS=argocd）
├── argocd-upgrade-post-v32-prod.sh    # 生产第二跳升级后修复（NS=argocd）
├── argocd-backup-prod.sh              # 生产备份（NS=argocd，升级前必须执行）
├── argocd-restore-prod.sh             # 生产从备份完整恢复（NS=argocd）
├── argocd-fix-manager-rbac-prod.sh    # 修复目标集群 argocd-manager RBAC（在目标集群执行）
└── argocd-test/
    ├── argocd-backup-test.sh               # 演练备份（NS=argocd-test）
    ├── argocd-restore-test.sh              # 演练从备份完整恢复（NS=argocd-test）
    ├── argocd-delete-test.sh               # 演练清理命名空间（NS=argocd-test）
    ├── argocd-fix-manager-rbac-test.sh     # 修复目标集群 argocd-manager RBAC
    ├── argocd-patch-manifest-test.py       # 将 install.yaml 的命名空间替换为 argocd-test（演练用）
    ├── argocd-setup-test.sh                # 演练环境初始化（仅 argocd-test NS）
    ├── argocd-upgrade-post-v31-test.sh     # 演练第一跳升级后修复（NS=argocd-test）
    └── argocd-upgrade-post-v32-test.sh     # 演练第二跳升级后修复（NS=argocd-test）
```

---

## 前置条件

### 工具

```bash
# kubectl 已配置并指向目标集群
kubectl version --client
kubectl config current-context

# python3 + PyYAML（演练 patch 脚本依赖）
python3 --version
pip3 install pyyaml
```

### 权限确认

```bash
# 确认对 argocd 命名空间有 apply 权限
kubectl auth can-i create deployment -n argocd
kubectl auth can-i patch configmap -n argocd

# 确认 ClusterRole 操作权限（install.yaml 含集群级 RBAC）
kubectl auth can-i create clusterrole
```

### 升级前状态快照

```bash
# 确认当前版本
kubectl -n argocd get deployment argocd-server \
  -o jsonpath='{.spec.template.spec.containers[0].image}'; echo

# 确认所有 Pod 健康
kubectl -n argocd get pods

# 确认 Redis HA 正常
kubectl -n argocd get pods -l app=argocd-redis-ha
kubectl -n argocd get service argocd-redis-ha-haproxy

# 确认 replicas（升级后需恢复为 2）
kubectl -n argocd get deployment argocd-server argocd-repo-server \
  -o=custom-columns='NAME:.metadata.name,REPLICAS:.spec.replicas'

# 确认 argocd-cmd-params-cm 当前值（升级前记录，apply 后会被重置）
kubectl -n argocd get configmap argocd-cmd-params-cm \
  -o jsonpath='{.data}' | python3 -m json.tool
```

---

## 升级 YAML 准备

> **说明**：install.yaml 必须使用 `ha/install.yaml`（含 Redis HA StatefulSet），
> 不能用默认的 `install.yaml`。生产直接使用原始 HA YAML，无需 patch 命名空间。

### 回滚备用 YAML（v3.0.6，升级前提前准备）

```bash
# 升级前提前下载，回滚第一跳时需要
curl -fsSL "https://raw.githubusercontent.com/argoproj/argo-cd/v3.0.6/manifests/ha/install.yaml" \
  -o argocd-install-v3.0.6-ha-prod.yaml
```

### 第一跳 YAML（v3.1.x）

```bash
# 确认最新 v3.1.x patch 版本：https://github.com/argoproj/argo-cd/releases
# 示例使用 v3.1.16，实际替换为当前最新版本
V31="v3.1.16"

curl -fsSL "https://raw.githubusercontent.com/argoproj/argo-cd/${V31}/manifests/ha/install.yaml" \
  -o argocd-install-v31-ha-prod.yaml

echo "下载完成: argocd-install-v31-ha-prod.yaml"
```

### 第二跳 YAML（v3.2.x）

```bash
# 确认最新 v3.2.x patch 版本：https://github.com/argoproj/argo-cd/releases
V32="v3.2.12"   # 替换为实际版本，如 v3.2.3

curl -fsSL "https://raw.githubusercontent.com/argoproj/argo-cd/${V32}/manifests/ha/install.yaml" \
  -o argocd-install-v32-ha-prod.yaml

echo "下载完成: argocd-install-v32-ha-prod.yaml"
```

### dry-run 预检（apply 前必做）

```bash
# 第一跳预检
kubectl apply --dry-run=server -f argocd-install-v31-ha-prod.yaml -n argocd 2>&1 \
  | grep -E "error|Error|Warning" || echo "dry-run 通过"

# 第二跳预检
kubectl apply --dry-run=server -f argocd-install-v32-ha-prod.yaml -n argocd 2>&1 \
  | grep -E "error|Error|Warning" || echo "dry-run 通过"
```

---

## 生产升级流程

### Step 0：升级前备份

```bash
# 在 argocd-jenkins/ 根目录执行
./argocd-backup-prod.sh
# 输出：./argocd-backup-YYYYMMDD-HHMMSS/
# 确认末尾出现 "备份完整性验证通过" 再继续
```

备份内容：Application / AppProject / ApplicationSet CR、ConfigMap、Secret、
ClusterRole/CRB、CRD、Deployment/StatefulSet/Service，并生成 `manifest.txt` 记录资源数量。

---

### Step 1：第一跳升级（v3.0.6 → v3.1.x）

```bash
cd argocd-jenkins/   # 回到根目录

# 1. 应用新版本 YAML
kubectl apply -f argocd-install-v31-ha-prod.yaml -n argocd

# 2. 等待 rollout 完成（按顺序等，确保依赖链就绪）
kubectl -n argocd rollout status statefulset/argocd-redis-ha-server        --timeout=300s
kubectl -n argocd rollout status deployment/argocd-redis-ha-haproxy        --timeout=300s
kubectl -n argocd rollout status deployment/argocd-server                  --timeout=300s
kubectl -n argocd rollout status deployment/argocd-repo-server             --timeout=300s
kubectl -n argocd rollout status deployment/argocd-applicationset-controller --timeout=300s
kubectl -n argocd rollout status deployment/argocd-notifications-controller  --timeout=300s
kubectl -n argocd rollout status statefulset/argocd-application-controller   --timeout=300s

# 3. 立即执行修复脚本（apply 会通过三路合并重置 argocd-cmd-params-cm，必须立即恢复）
./argocd-upgrade-post-v31-prod.sh
```

修复脚本自动处理：
- 恢复 `redis.server=argocd-redis-ha-haproxy:6379`
- 恢复 `server.insecure=true`
- 强制恢复 `replicas=2`（argocd-server、argocd-repo-server）
- 校验 `argocd-manager-role` 保护状态
- Redis HA 连通性验证

---

### Step 2：第二跳升级（v3.1.x → v3.2.x）

```bash
# 1. 应用新版本 YAML
kubectl apply -f argocd-install-v32-ha-prod.yaml -n argocd

# 2. 等待 rollout 完成
kubectl -n argocd rollout status statefulset/argocd-redis-ha-server        --timeout=300s
kubectl -n argocd rollout status deployment/argocd-redis-ha-haproxy        --timeout=300s
kubectl -n argocd rollout status deployment/argocd-server                  --timeout=300s
kubectl -n argocd rollout status deployment/argocd-repo-server             --timeout=300s
kubectl -n argocd rollout status deployment/argocd-applicationset-controller --timeout=300s
kubectl -n argocd rollout status deployment/argocd-notifications-controller  --timeout=300s
kubectl -n argocd rollout status statefulset/argocd-application-controller   --timeout=300s

# 3. 立即执行修复脚本
./argocd-upgrade-post-v32-prod.sh
```

v3.2.x 额外检查（修复脚本自动执行）：
- `applicationset-controller` RBAC 新增 `coordination.k8s.io/leases` 权限
- 手动验证：`kubectl get clusterrole argocd-applicationset-controller -o yaml | grep coordination`

---

### Step 3：升级后巡检

```bash
./argocd-check-v32.sh
# 输出目录：/tmp/argocd-check-YYYYMMDD-HHMMSS/
# 关注末尾 PASS / WARN / FAIL 统计，FAIL=0 视为通过
```

---

## 回滚流程

### 触发条件

- `CrashLoopBackOff` 持续 > 5 分钟
- Degraded Application 比例 > 30%
- Redis HA 连通失败且无法自愈

### 第二跳回滚（v3.2.x → v3.1.x）

```bash
# 1. 回滚镜像
kubectl apply -f argocd-install-v31-ha-prod.yaml -n argocd

# 2. 立即修复 CM（apply 会重置，必须立即 patch，不能等待）
kubectl -n argocd patch configmap argocd-cmd-params-cm --type merge \
  -p '{"data":{"redis.server":"argocd-redis-ha-haproxy:6379","server.insecure":"true"}}'

# 3. 恢复 replicas=2
kubectl -n argocd scale deployment argocd-server argocd-repo-server --replicas=2

# 4. 重启使 CM 生效
kubectl -n argocd rollout restart deployment/argocd-server deployment/argocd-repo-server
kubectl -n argocd rollout restart statefulset/argocd-application-controller
kubectl -n argocd rollout status deployment/argocd-server --timeout=180s
```

### 第一跳回滚（v3.1.x → v3.0.6）

```bash
kubectl apply -f argocd-install-v3.0.6-ha-prod.yaml -n argocd

kubectl -n argocd patch configmap argocd-cmd-params-cm --type merge \
  -p '{"data":{"redis.server":"argocd-redis-ha-haproxy:6379","server.insecure":"true"}}'

kubectl -n argocd scale deployment argocd-server argocd-repo-server --replicas=2

kubectl -n argocd rollout restart deployment/argocd-server deployment/argocd-repo-server
kubectl -n argocd rollout restart statefulset/argocd-application-controller
kubectl -n argocd rollout status deployment/argocd-server --timeout=180s
```

---

## 从备份完整恢复

适用场景：回滚后仍异常、CR/ConfigMap/Secret 数据损坏。

```bash
# 在 argocd-jenkins/ 根目录执行
# 脚本启动时会显示备份 manifest.txt 内容，并要求输入 yes 确认 kubectl context
./argocd-restore-prod.sh argocd-backup-/
```

恢复顺序（任一步骤失败立即中止）：

| 步骤 | 内容 |
|---|---|
| 1 | CRD apply + 等待 `Established` |
| 2 | 命名空间 |
| 3 | ClusterRole / ClusterRoleBinding |
| 4 | ServiceAccount / Role / RoleBinding / ConfigMap / Secret |
| 4a | ConfigMap 恢复后强制校验 redis.server + server.insecure |
| 5 | Service / Deployment / StatefulSet + 等待 rollout 就绪 |
| 6 | Application / AppProject / ApplicationSet CR |
| 验证 | 与 manifest.txt 核对资源数量，不匹配则 exit 1 |

---

## 升级后巡检

```bash
./argocd-check-v32.sh
```

检查项：
- argocd-server 镜像版本符合 v3.2.x
- 所有 Pod Running，无 CrashLoop
- Pod 重启次数（>0 需关注）
- argocd-cmd-params-cm 中 redis.server / server.insecure 值
- replicas=2（argocd-server、argocd-repo-server）
- Redis HA HAProxy ping 连通性
- applicationset-controller RBAC（coordination.k8s.io/leases）

---

## 跨集群 RBAC 修复

**症状**：ArgoCD UI 中目标集群 Application 显示 `SyncFailed`  
**日志**：`User "system:serviceaccount:kube-system:argocd-manager" cannot get resource`

```bash
# 1. 切换到目标集群（不是 ArgoCD 所在集群）
kubectl config use-context 

# 2. 执行修复
./argocd-fix-manager-rbac-prod.sh

# 3. 切回 ArgoCD 集群，触发重新同步
kubectl config use-context 
argocd app sync 
# 或在 ArgoCD UI 点击 Sync / Refresh
```

---

## 演练流程（argocd-test）

演练在独立测试集群的 `argocd-test` 命名空间执行，与生产完全隔离。

### 环境初始化

```bash
cd argocd-test/

# 1. 下载 v3.0.6 HA YAML（如已存在跳过）
curl -fsSL https://raw.githubusercontent.com/argoproj/argo-cd/v3.0.6/manifests/ha/install.yaml \
  -o argocd-install-v3.0.6-ha.yaml

# 2. Patch 命名空间（argocd → argocd-test）
python3 argocd-patch-manifest-test.py argocd-install-v3.0.6-ha.yaml argocd-install-v3.0.6-ha-test.yaml

# 3. 部署基线（或使用 setup 脚本自动完成以上步骤）
./argocd-setup-test.sh
```

### 第一跳演练（v3.0.6 → v3.1.x）

```bash
# 1. 下载并 patch v3.1.x YAML
V31="v3.1.16"   # 替换为当前最新版本
curl -fsSL "https://raw.githubusercontent.com/argoproj/argo-cd/${V31}/manifests/ha/install.yaml" \
  -o argocd-install-${V31}-ha.yaml
python3 argocd-patch-manifest-test.py argocd-install-${V31}-ha.yaml argocd-install-v31-ha-test.yaml

# 2. dry-run 预检
kubectl apply --dry-run=server -f argocd-install-v31-ha-test.yaml -n argocd-test 2>&1 \
  | grep -E "error|Error|Warning" || echo "dry-run 通过"

# 3. 应用
kubectl apply -f argocd-install-v31-ha-test.yaml -n argocd-test

# 4. 等待 rollout
kubectl -n argocd-test rollout status statefulset/argocd-redis-ha-server          --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-redis-ha-haproxy          --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-server                    --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-repo-server               --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-applicationset-controller --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-notifications-controller  --timeout=300s
kubectl -n argocd-test rollout status statefulset/argocd-application-controller   --timeout=300s

# 5. 执行修复脚本
./argocd-upgrade-post-v31-test.sh
```

### 第二跳演练（v3.1.x → v3.2.x）

```bash
V32="v3.2.x"   # 替换为实际版本
curl -fsSL "https://raw.githubusercontent.com/argoproj/argo-cd/${V32}/manifests/ha/install.yaml" \
  -o argocd-install-${V32}-ha.yaml
python3 argocd-patch-manifest-test.py argocd-install-${V32}-ha.yaml argocd-install-v32-ha-test.yaml

kubectl apply --dry-run=server -f argocd-install-v32-ha-test.yaml -n argocd-test 2>&1 \
  | grep -E "error|Error|Warning" || echo "dry-run 通过"

kubectl apply -f argocd-install-v32-ha-test.yaml -n argocd-test

kubectl -n argocd-test rollout status statefulset/argocd-redis-ha-server          --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-redis-ha-haproxy          --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-server                    --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-repo-server               --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-applicationset-controller --timeout=300s
kubectl -n argocd-test rollout status deployment/argocd-notifications-controller  --timeout=300s
kubectl -n argocd-test rollout status statefulset/argocd-application-controller   --timeout=300s

./argocd-upgrade-post-v32-test.sh
```

### 演练验证

```bash
# Pod 状态
kubectl -n argocd-test get pods -o wide

# CM 关键字段
kubectl -n argocd-test get configmap argocd-cmd-params-cm \
  -o jsonpath='{.data.redis\.server}{"\n"}{.data.server\.insecure}{"\n"}'
# 期望：argocd-redis-ha-haproxy:6379 / true

# 镜像版本
kubectl -n argocd-test get deployment argocd-server \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="argocd-server")].image}'; echo

# 详细巡检参考 argocd-test/README.md 阶段五
```

### 清理演练环境

```bash
cd argocd-test/
# 传入备份目录名作为确认凭证，脚本会二次确认后删除
./argocd-delete-test.sh ./argocd-backup-/
```

---

## 关键配置说明

| 配置项 | 值 | 原因 |
|---|---|---|
| `redis.server` | `argocd-redis-ha-haproxy:6379` | `kubectl apply -f install.yaml` 通过三路合并重置此字段，**每次 apply 后必须立即 patch** |
| `server.insecure` | `true` | TLS 在 Ingress 层终止，同上，apply 后会被重置 |
| `replicas` | 生产=2 / 演练=1 | install.yaml 可能覆盖为 1，升级后升级脚本自动强制恢复 |
| `ha/install.yaml` | 必须用 HA 版本 | 包含 Redis HA StatefulSet，普通 install.yaml 无 Redis HA |

### argocd-patch-manifest-test.py 说明

仅演练环境需要。作用：将 `install.yaml` 中所有 `namespace: argocd` 深度替换为 `argocd-test`，
覆盖 metadata、subjects、webhooks 等所有位置，并保留 CRD。

```bash
# 用法
python3 argocd-test/argocd-patch-manifest-test.py  
```

生产环境直接使用原始 `ha/install.yaml`，不需要 patch。

---

## 常见问题

**Q: apply 后 argocd-server CrashLoop，日志显示无法连接 Redis**  
A: `install.yaml` 三路合并重置了 `argocd-cmd-params-cm`。立即执行：
```bash
kubectl -n argocd patch configmap argocd-cmd-params-cm --type merge \
  -p '{"data":{"redis.server":"argocd-redis-ha-haproxy:6379","server.insecure":"true"}}'
kubectl -n argocd rollout restart deployment/argocd-server
```

**Q: applicationset-controller 报 Forbidden（v3.2.x）**  
A: v3.2.x 新增了 `coordination.k8s.io/leases` 权限。检查 ClusterRole 是否包含：
```bash
kubectl get clusterrole argocd-applicationset-controller -o yaml | grep -A3 coordination
```
若缺失，重新 apply `argocd-install-v32-ha-prod.yaml` 并执行 `./argocd-upgrade-post-v32-prod.sh`。

**Q: 恢复后 Application 数量少于备份记录**  
A: `argocd-restore-prod.sh` 末尾会对比 `manifest.txt` 并 exit 1。先检查 CRD 是否 Established：
```bash
kubectl get crd | grep argoproj.io
kubectl wait crd/applications.argoproj.io --for=condition=Established --timeout=60s
```
CRD 就绪后重新执行 `./argocd-restore-prod.sh dir>`。

**Q: dry-run 报 unknown field 或 CRD 不存在**  
A: 目标集群 CRD 版本与 install.yaml 不匹配。先 apply CRD 部分：
```bash
kubectl apply -f argocd-install-v31-ha-prod.yaml -n argocd \
  --selector="app.kubernetes.io/part-of=argocd" --dry-run=server
```

**Q: argocd-manager-role 在目标集群丢失**  
A: 切换到目标集群执行修复，再回 ArgoCD 集群触发同步：
```bash
kubectl config use-context 
./argocd-fix-manager-rbac-prod.sh
kubectl config use-context 
argocd app sync 
```

**Q: Redis HA HAProxy ping 失败（需要密码）**  
A: 生产 Redis 配置了 AUTH，redis-cli 需要 `-a ` 参数，或通过 Secret 获取：
```bash
REDIS_PASS=$(kubectl -n argocd get secret argocd-redis-ha -o jsonpath='{.data.auth}' | base64 -d)
kubectl -n argocd exec  -c redis -- \
  redis-cli -h argocd-redis-ha-haproxy -p 6379 -a "${REDIS_PASS}" ping
```
    readme.md

     

  • 备份 恢复 升级 巡检 补丁等配置文件

    #!/usr/bin/env bash
# argocd-backup-prod.sh
# 备份 argocd 命名空间所有资源 + 集群级 RBAC + CRD
#
# 使用方法：
#   ./argocd-backup-prod.sh
#   备份目录：./argocd-backup-YYYYMMDD-HHMMSS/
set -euo pipefail

NS="argocd"
BACKUP_DIR="./argocd-backup-$(date +%Y%m%d-%H%M%S)"
mkdir -p "${BACKUP_DIR}"

echo "====================================================================="
echo "  ArgoCD 资源备份"
echo "  命名空间: ${NS}"
echo "  备份目录: ${BACKUP_DIR}"
echo "  时间: $(date '+%Y-%m-%d %H:%M:%S')"
echo "====================================================================="
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 1: ArgoCD CR（最重要：Application / AppProject / ApplicationSet）
# ─────────────────────────────────────────────────────────────────────────────
echo "[1/5] 备份 ArgoCD CR（Application / AppProject / ApplicationSet）..."
for cr in applications appprojects applicationsets; do
    count=$(kubectl -n "${NS}" get "${cr}.argoproj.io" --no-headers 2>/dev/null | wc -l | tr -d ' ' || echo "0")
    if [[ "${count}" -gt 0 ]]; then
        kubectl -n "${NS}" get "${cr}.argoproj.io" -o yaml > "${BACKUP_DIR}/${cr}.yaml"
        echo "  ✅ ${cr}: ${count} 个 → ${cr}.yaml"
    else
        echo "  ℹ️  ${cr}: 无资源，跳过"
    fi
done
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 2: ConfigMap（argocd-cm / argocd-cmd-params-cm / argocd-rbac-cm 等）
# ─────────────────────────────────────────────────────────────────────────────
echo "[2/5] 备份 ConfigMap..."
kubectl -n "${NS}" get configmap -o yaml > "${BACKUP_DIR}/configmaps.yaml"
count=$(kubectl -n "${NS}" get configmap --no-headers | wc -l | tr -d ' ')
echo "  ✅ ConfigMap: ${count} 个 → configmaps.yaml"
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 3: Secret（repo 凭证、argocd-secret、TLS 等）
# ─────────────────────────────────────────────────────────────────────────────
echo "[3/5] 备份 Secret..."
kubectl -n "${NS}" get secret -o yaml > "${BACKUP_DIR}/secrets.yaml"
count=$(kubectl -n "${NS}" get secret --no-headers | wc -l | tr -d ' ')
echo "  ✅ Secret: ${count} 个 → secrets.yaml（含敏感数据，请妥善保管）"
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 4: 命名空间内其余资源（Deployment / Service / SA / Role 等）
# ─────────────────────────────────────────────────────────────────────────────
echo "[4/5] 备份命名空间内工作负载资源..."
for resource in deployment statefulset service serviceaccount role rolebinding; do
    kubectl -n "${NS}" get "${resource}" -o yaml > "${BACKUP_DIR}/${resource}s.yaml" 2>/dev/null
done
echo "  ✅ deployment / statefulset / service / serviceaccount / role / rolebinding → 各独立文件"
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 5: 集群级资源（ClusterRole / CRB / CRD）
# ─────────────────────────────────────────────────────────────────────────────
echo "[5/5] 备份集群级 RBAC 和 CRD..."

# ClusterRole
kubectl get clusterrole -o yaml 2>/dev/null | python3 - <<'PYEOF' > "${BACKUP_DIR}/clusterroles.yaml" || true
import sys, yaml
docs = [d for d in yaml.safe_load_all(sys.stdin)
        if d and 'argocd' in d.get('metadata', {}).get('name', '')]
print(yaml.dump_all(docs, default_flow_style=False, allow_unicode=True))
PYEOF
cr_count=$(grep -c "^kind: ClusterRole$" "${BACKUP_DIR}/clusterroles.yaml" 2>/dev/null || true)
if [[ "${cr_count:-0}" -gt 0 ]]; then
    echo "  ✅ ClusterRole: ${cr_count} 个 → clusterroles.yaml"
else
    echo "  ⚠️  ClusterRole: 0 个（权限不足或无 argocd ClusterRole，升级时由 install.yaml 重建）"
fi

# ClusterRoleBinding
kubectl get clusterrolebinding -o yaml 2>/dev/null | python3 - <<'PYEOF' > "${BACKUP_DIR}/clusterrolebindings.yaml" || true
import sys, yaml
docs = [d for d in yaml.safe_load_all(sys.stdin)
        if d and 'argocd' in d.get('metadata', {}).get('name', '')]
print(yaml.dump_all(docs, default_flow_style=False, allow_unicode=True))
PYEOF
crb_count=$(grep -c "^kind: ClusterRoleBinding$" "${BACKUP_DIR}/clusterrolebindings.yaml" 2>/dev/null || true)
if [[ "${crb_count:-0}" -gt 0 ]]; then
    echo "  ✅ ClusterRoleBinding: ${crb_count} 个 → clusterrolebindings.yaml"
else
    echo "  ⚠️  ClusterRoleBinding: 0 个（权限不足或无 argocd CRB，升级时由 install.yaml 重建）"
fi

# CRD
kubectl get crd -o yaml 2>/dev/null | python3 - <<'PYEOF' > "${BACKUP_DIR}/crds.yaml" || true
import sys, yaml
docs = [d for d in yaml.safe_load_all(sys.stdin)
        if d and 'argoproj.io' in d.get('metadata', {}).get('name', '')]
print(yaml.dump_all(docs, default_flow_style=False, allow_unicode=True))
PYEOF
crd_count=$(grep -c "^kind: CustomResourceDefinition$" "${BACKUP_DIR}/crds.yaml" 2>/dev/null || true)
if [[ "${crd_count:-0}" -gt 0 ]]; then
    echo "  ✅ CRD: ${crd_count} 个 → crds.yaml"
else
    echo "  ⚠️  CRD: 0 个（权限不足或无 argoproj.io CRD，升级时由 install.yaml 重建）"
fi
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# 备份完整性验证 + manifest.txt 生成（供 restore 恢复核对用）
# ─────────────────────────────────────────────────────────────────────────────
verify_backup() {
    local fail=0

    echo "====================================================================="
    echo "  备份完整性验证"
    echo "====================================================================="
    echo ""

    echo "--- 文件检查 ---"
    for f in configmaps.yaml secrets.yaml crds.yaml clusterroles.yaml clusterrolebindings.yaml \
              deployments.yaml statefulsets.yaml services.yaml serviceaccounts.yaml \
              roles.yaml rolebindings.yaml; do
        local path="${BACKUP_DIR}/${f}"
        if [[ -f "${path}" ]] && [[ -s "${path}" ]]; then
            local lines
            lines=$(wc -l < "${path}" | tr -d ' ')
            echo "  ✅ ${f} (${lines} 行)"
        else
            echo "  ℹ️  ${f} 不存在或为空（如无对应资源则正常）"
        fi
    done
    echo ""

    echo "--- CRD 数量验证（ArgoCD 预期 >= 4 个）---"
    local crd_count=0
    if [[ -f "${BACKUP_DIR}/crds.yaml" ]]; then
        crd_count=$(grep -c "^kind: CustomResourceDefinition$" "${BACKUP_DIR}/crds.yaml" 2>/dev/null || true)
    fi
    if [[ "${crd_count}" -lt 4 ]]; then
        echo "  ⚠️  CRD: ${crd_count} 个 < 4（权限不足无法备份，升级时由 install.yaml 重建，不阻断备份）"
    else
        echo "  ✅ CRD: ${crd_count} 个（>= 4）"
    fi
    echo ""

    echo "--- 关键 ConfigMap 验证 ---"
    if grep -q "argocd-cmd-params-cm" "${BACKUP_DIR}/configmaps.yaml" 2>/dev/null; then
        echo "  ✅ argocd-cmd-params-cm 存在于备份"
    else
        echo "  ⛔ argocd-cmd-params-cm 未在 configmaps.yaml 中找到"
        fail=1
    fi
    echo ""

    echo "--- 关键 Secret 验证 ---"
    if grep -q "argocd-secret" "${BACKUP_DIR}/secrets.yaml" 2>/dev/null; then
        echo "  ✅ argocd-secret 存在于备份"
    else
        echo "  ⚠️  argocd-secret 未在 secrets.yaml 中找到（如已外部管理则正常）"
    fi
    echo ""

    echo "--- 生成 manifest.txt（restore 恢复时核对数量用）---"
    {
        echo "backup_time=$(date '+%Y-%m-%d %H:%M:%S')"
        echo "namespace=${NS}"
        echo "crd_count=${crd_count}"
        for cr in applications appprojects applicationsets; do
            local cnt
            cnt=$(kubectl -n "${NS}" get "${cr}.argoproj.io" --no-headers 2>/dev/null \
                | wc -l | tr -d ' ' || echo "0")
            echo "${cr}_count=${cnt}"
        done
        echo "configmap_count=$(kubectl -n "${NS}" get configmap --no-headers 2>/dev/null \
            | wc -l | tr -d ' ' || echo "0")"
        echo "secret_count=$(kubectl -n "${NS}" get secret --no-headers 2>/dev/null \
            | wc -l | tr -d ' ' || echo "0")"
        echo "deployment_count=$(kubectl -n "${NS}" get deployment --no-headers 2>/dev/null \
            | wc -l | tr -d ' ' || echo "0")"
    } > "${BACKUP_DIR}/manifest.txt"
    echo "  ✅ manifest.txt 已生成:"
    cat "${BACKUP_DIR}/manifest.txt" | sed 's/^/    /'
    echo ""

    if [[ "${fail}" -eq 1 ]]; then
        echo "  ⛔ 备份验证失败！请确认集群状态后重新备份"
        echo "     不建议使用不完整的备份作为恢复依据"
        exit 1
    fi
    echo "  ✅ 备份完整性验证通过"
    echo ""
}

verify_backup

# ─────────────────────────────────────────────────────────────────────────────
# 备份清单
# ─────────────────────────────────────────────────────────────────────────────
echo "====================================================================="
echo "  备份完成 ✅"
echo "  目录: ${BACKUP_DIR}"
echo ""
ls -lh "${BACKUP_DIR}"
echo ""
echo "  下一步:"
echo "    升级失败需回滚时: ./argocd-restore-prod.sh ${BACKUP_DIR}"
echo "====================================================================="
    argocd-backup-prod.sh
    #!/usr/bin/env bash
# argocd-restore-prod.sh
# 从备份恢复 argocd 命名空间资源 + 集群级 RBAC + CRD
#
# 使用方法：
#   ./argocd-restore-prod.sh dir>
#
# 恢复顺序（每步均有错误检查，任何失败立即中止）：
#   1. 前置校验（context 确认 + 备份目录完整性）
#   2. CRD（apply + 等待 Established，再进行下一步）
#   3. 命名空间
#   4. ClusterRole / ClusterRoleBinding
#   5. ServiceAccount / Role / RoleBinding / ConfigMap / Secret（+ CM 强制校验）
#   6. Service / Deployment / StatefulSet（+ 等待 rollout 就绪）
#   7. Application / AppProject / ApplicationSet CR
#   8. 恢复后计数核对（与 manifest.txt 对比）
set -euo pipefail

NS="argocd"
REDIS_HA_ADDR="argocd-redis-ha-haproxy:6379"

if [[ $# -lt 1 ]]; then
    echo "Usage: $0 "
    echo "  示例: $0 ./argocd-backup-20260803-120000"
    exit 1
fi

BACKUP_DIR="$1"
if [[ ! -d "${BACKUP_DIR}" ]]; then
    echo "⛔ 备份目录不存在: ${BACKUP_DIR}"
    exit 1
fi

echo "====================================================================="
echo "  ArgoCD 资源恢复"
echo "  命名空间: ${NS}"
echo "  备份目录: ${BACKUP_DIR}"
echo "  时间: $(date '+%Y-%m-%d %H:%M:%S')"
echo "====================================================================="
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# 前置校验 1: kubectl context 确认（防止误操作到错误集群）
# ─────────────────────────────────────────────────────────────────────────────
CURRENT_CTX=$(kubectl config current-context 2>/dev/null || echo "unknown")
echo "  ⚠️  当前 kubectl context: ${CURRENT_CTX}"
echo "  恢复目标命名空间: ${NS}"
if [[ -f "${BACKUP_DIR}/manifest.txt" ]]; then
    echo ""
    echo "  备份时记录 (manifest.txt):"
    cat "${BACKUP_DIR}/manifest.txt" | sed 's/^/    /'
fi
echo ""
read -rp "  确认恢复到此集群？输入 yes 继续: " CTX_CONFIRM
if [[ "${CTX_CONFIRM}" != "yes" ]]; then
    echo "  已取消。请先切换到正确的 kubectl context 后重试"
    exit 0
fi
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# 前置校验 2: 关键备份文件存在性检查
# ─────────────────────────────────────────────────────────────────────────────
echo "[PRE] 备份目录完整性检查..."
PRE_FAIL=0
for f in configmaps.yaml secrets.yaml crds.yaml; do
    if [[ ! -f "${BACKUP_DIR}/${f}" ]] || [[ ! -s "${BACKUP_DIR}/${f}" ]]; then
        echo "  ⛔ 关键备份文件缺失或为空: ${f}"
        PRE_FAIL=1
    else
        echo "  ✅ ${f}"
    fi
done
if [[ "${PRE_FAIL}" -eq 1 ]]; then
    echo "  ⛔ 备份目录不完整，中止恢复（请重新执行 ./argocd-backup-prod.sh）"
    exit 1
fi
echo "  ✅ 备份目录检查通过"
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# 工具：清理 kubectl export YAML 中的运行时字段，apply 前必须去除
# ─────────────────────────────────────────────────────────────────────────────
clean_yaml() {
    python3 - "$1" <<'PYEOF'
import sys, yaml

def clean(obj):
    if not isinstance(obj, dict):
        return obj
    # 移除运行时字段
    for key in ('resourceVersion', 'uid', 'generation',
                'creationTimestamp', 'selfLink', 'managedFields'):
        obj.pop(key, None)
    metadata = obj.get('metadata', {})
    for key in ('resourceVersion', 'uid', 'generation',
                'creationTimestamp', 'selfLink', 'managedFields'):
        metadata.pop(key, None)
    # 移除 last-applied-configuration（可能超大，引发 apply 失败或三路合并异常覆盖 redis.server）
    annotations = metadata.get('annotations', {})
    if annotations:
        annotations.pop('kubectl.kubernetes.io/last-applied-configuration', None)
        if not annotations:
            metadata.pop('annotations', None)
    # 移除 status（避免 apply 冲突）
    obj.pop('status', None)
    # 递归清理子字段
    for v in obj.values():
        if isinstance(v, dict):
            clean(v)
        elif isinstance(v, list):
            for item in v:
                if isinstance(item, dict):
                    clean(item)
    return obj

input_file = sys.argv[1]
with open(input_file) as f:
    docs = list(yaml.safe_load_all(f))

cleaned = [clean(d) for d in docs if d]
print(yaml.dump_all(cleaned, default_flow_style=False, allow_unicode=True))
PYEOF
}

# ─────────────────────────────────────────────────────────────────────────────
# 工具：apply 并严格检查错误（任何失败立即 exit 1，不静默忽略）
# ─────────────────────────────────────────────────────────────────────────────
apply_if_exists() {
    local file="$1"
    local label="$2"
    if [[ -f "${file}" ]] && [[ -s "${file}" ]]; then
        local tmp
        tmp=$(mktemp /tmp/argocd-restore-XXXXXX.yaml)
        clean_yaml "${file}" > "${tmp}"
        local out
        if ! out=$(kubectl apply -f "${tmp}" 2>&1); then
            echo "  ⛔ ${label} apply 失败（中止恢复）:"
            echo "${out}" | sed 's/^/    /'
            rm -f "${tmp}"
            exit 1
        fi
        echo "${out}" | grep -v "^$" | sed 's/^/    /'
        rm -f "${tmp}"
        echo "  ✅ ${label}"
    else
        echo "  ℹ️  ${label}: 备份文件不存在或为空，跳过"
    fi
}

# ─────────────────────────────────────────────────────────────────────────────
# Step 1: CRD（apply + 等待每个 CRD Established，再继续）
# ─────────────────────────────────────────────────────────────────────────────
echo "[1/6] 恢复 CRD..."
apply_if_exists "${BACKUP_DIR}/crds.yaml" "CRD (argoproj.io)"
echo "  等待 CRD Established（最多 60s/个，未就绪则后续 CR apply 必定失败）..."
for crd in applications.argoproj.io appprojects.argoproj.io applicationsets.argoproj.io; do
    if kubectl get crd "${crd}" &>/dev/null 2>&1; then
        if kubectl wait crd/"${crd}" --for=condition=Established --timeout=60s 2>/dev/null; then
            echo "  ✅ ${crd} Established"
        else
            echo "  ⛔ ${crd} 等待 Established 超时，中止恢复"
            exit 1
        fi
    fi
done
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 2: 命名空间
# ─────────────────────────────────────────────────────────────────────────────
echo "[2/6] 恢复命名空间 ${NS}..."
kubectl create namespace "${NS}" --dry-run=client -o yaml | kubectl apply -f -
echo "  ✅ namespace/${NS} 就绪"
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 3: 集群级 RBAC
# ─────────────────────────────────────────────────────────────────────────────
echo "[3/6] 恢复 ClusterRole / ClusterRoleBinding..."
apply_if_exists "${BACKUP_DIR}/clusterroles.yaml"        "ClusterRole"
apply_if_exists "${BACKUP_DIR}/clusterrolebindings.yaml" "ClusterRoleBinding"
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 4: 基础资源（ServiceAccount / Role / RoleBinding / ConfigMap / Secret）
# ─────────────────────────────────────────────────────────────────────────────
echo "[4/6] 恢复基础资源..."
apply_if_exists "${BACKUP_DIR}/serviceaccounts.yaml" "ServiceAccount"
apply_if_exists "${BACKUP_DIR}/roles.yaml"           "Role"
apply_if_exists "${BACKUP_DIR}/rolebindings.yaml"    "RoleBinding"
apply_if_exists "${BACKUP_DIR}/configmaps.yaml"      "ConfigMap"
apply_if_exists "${BACKUP_DIR}/secrets.yaml"         "Secret"

# ConfigMap 恢复后强制校验并修复 redis.server（防止备份遗漏或三路合并异常覆盖）
echo "  [CM 强制校验] 确认 redis.server + server.insecure..."
CURRENT_REDIS=$(kubectl -n "${NS}" get configmap argocd-cmd-params-cm \
    -o jsonpath='{.data.redis\.server}' 2>/dev/null || echo "")
CURRENT_INSECURE=$(kubectl -n "${NS}" get configmap argocd-cmd-params-cm \
    -o jsonpath='{.data.server\.insecure}' 2>/dev/null || echo "")
if [[ "${CURRENT_REDIS}" != "${REDIS_HA_ADDR}" || "${CURRENT_INSECURE}" != "true" ]]; then
    echo "  ⚠️  CM 值不正确（redis.server=[${CURRENT_REDIS}] insecure=[${CURRENT_INSECURE}]），强制修复..."
    kubectl -n "${NS}" patch configmap argocd-cmd-params-cm \
        --type merge \
        -p "{\"data\":{\"redis.server\":\"${REDIS_HA_ADDR}\",\"server.insecure\":\"true\"}}"
    echo "  ✅ redis.server + server.insecure 已强制修复"
else
    echo "  ✅ redis.server=${CURRENT_REDIS}  server.insecure=${CURRENT_INSECURE}"
fi
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 5: 工作负载（Service / Deployment / StatefulSet + 等待 rollout 就绪）
# ─────────────────────────────────────────────────────────────────────────────
echo "[5/6] 恢复工作负载..."
apply_if_exists "${BACKUP_DIR}/services.yaml"     "Service"
apply_if_exists "${BACKUP_DIR}/deployments.yaml"  "Deployment"
apply_if_exists "${BACKUP_DIR}/statefulsets.yaml" "StatefulSet"

echo "  等待工作负载 rollout 就绪（最多 5 分钟）..."
ROLLOUT_FAIL=0
for deploy in argocd-server argocd-repo-server argocd-applicationset-controller \
              argocd-notifications-controller argocd-redis-ha-haproxy; do
    if kubectl -n "${NS}" get deployment "${deploy}" &>/dev/null 2>&1; then
        replicas=$(kubectl -n "${NS}" get deployment "${deploy}" \
            -o jsonpath='{.spec.replicas}' 2>/dev/null || echo "0")
        if [[ "${replicas:-0}" -gt 0 ]]; then
            if ! kubectl -n "${NS}" rollout status deployment/"${deploy}" --timeout=300s; then
                echo "  ❌ ${deploy} rollout 超时/失败"
                ROLLOUT_FAIL=$((ROLLOUT_FAIL + 1))
            fi
        fi
    fi
done
for sts in argocd-application-controller argocd-redis-ha-server; do
    if kubectl -n "${NS}" get statefulset "${sts}" &>/dev/null 2>&1; then
        replicas=$(kubectl -n "${NS}" get statefulset "${sts}" \
            -o jsonpath='{.spec.replicas}' 2>/dev/null || echo "0")
        if [[ "${replicas:-0}" -gt 0 ]]; then
            if ! kubectl -n "${NS}" rollout status statefulset/"${sts}" --timeout=300s; then
                echo "  ❌ ${sts} rollout 超时/失败"
                ROLLOUT_FAIL=$((ROLLOUT_FAIL + 1))
            fi
        fi
    fi
done
if [[ "${ROLLOUT_FAIL}" -gt 0 ]]; then
    echo "  ⛔ ${ROLLOUT_FAIL} 个工作负载 rollout 失败，请检查上方详情后手动处理"
    exit 1
fi
echo "  ✅ 所有工作负载就绪"
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 6: ArgoCD CR（Application / AppProject / ApplicationSet）
# ─────────────────────────────────────────────────────────────────────────────
echo "[6/6] 恢复 ArgoCD CR..."
apply_if_exists "${BACKUP_DIR}/appprojects.yaml"     "AppProject"
apply_if_exists "${BACKUP_DIR}/applications.yaml"    "Application"
apply_if_exists "${BACKUP_DIR}/applicationsets.yaml" "ApplicationSet"
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# 恢复后验证（与 manifest.txt 核对，最终确认无丢失）
# ─────────────────────────────────────────────────────────────────────────────
echo "====================================================================="
echo "  恢复后验证"
echo "====================================================================="
echo ""
VERIFY_FAIL=0

echo "--- ArgoCD CR 数量核对 ---"
for cr in applications appprojects applicationsets; do
    actual=$(kubectl -n "${NS}" get "${cr}.argoproj.io" --no-headers 2>/dev/null \
        | wc -l | tr -d ' ' || echo "0")
    if [[ -f "${BACKUP_DIR}/manifest.txt" ]]; then
        expected=$(grep "^${cr}_count=" "${BACKUP_DIR}/manifest.txt" 2>/dev/null \
            | cut -d= -f2 || echo "")
        if [[ -n "${expected}" ]] && [[ "${actual}" -ne "${expected}" ]]; then
            echo "  ⚠️  ${cr}: 恢复 ${actual} 个，备份记录 ${expected} 个（数量不匹配）"
            VERIFY_FAIL=$((VERIFY_FAIL + 1))
        else
            echo "  ✅ ${cr}: ${actual} 个"
        fi
    else
        echo "  ✅ ${cr}: ${actual} 个（无 manifest.txt，跳过数量核对）"
    fi
done
echo ""

echo "--- argocd-cmd-params-cm 关键字段 ---"
FINAL_REDIS=$(kubectl -n "${NS}" get configmap argocd-cmd-params-cm \
    -o jsonpath='{.data.redis\.server}' 2>/dev/null || echo "")
FINAL_INSECURE=$(kubectl -n "${NS}" get configmap argocd-cmd-params-cm \
    -o jsonpath='{.data.server\.insecure}' 2>/dev/null || echo "")
if [[ "${FINAL_REDIS}" == "${REDIS_HA_ADDR}" ]]; then
    echo "  ✅ redis.server: ${FINAL_REDIS}"
else
    echo "  ⛔ redis.server 最终值错误: [${FINAL_REDIS}]（预期 ${REDIS_HA_ADDR}）"
    VERIFY_FAIL=$((VERIFY_FAIL + 1))
fi
if [[ "${FINAL_INSECURE}" == "true" ]]; then
    echo "  ✅ server.insecure: true"
else
    echo "  ⚠️  server.insecure: [${FINAL_INSECURE}]（预期 true）"
fi
echo ""

echo "--- CRD 验证 ---"
crd_count=$(kubectl get crd 2>/dev/null | grep -c "argoproj.io" || true)
if [[ "${crd_count}" -lt 4 ]]; then
    echo "  ⚠️  argoproj.io CRD 数量 ${crd_count} < 4"
else
    echo "  ✅ argoproj.io CRD: ${crd_count} 个"
fi
echo ""

echo "--- Pod 状态 ---"
kubectl -n "${NS}" get pods -o wide
echo ""

if [[ "${VERIFY_FAIL}" -gt 0 ]]; then
    echo "====================================================================="
    echo "  ⛔ 恢复验证有 ${VERIFY_FAIL} 项失败，请检查上方详情"
    echo "====================================================================="
    exit 1
fi

echo "====================================================================="
echo "  ✅ 恢复完成并验证通过"
echo "====================================================================="
    argocd-restore-prod.sh
    #!/usr/bin/env bash
# argocd-fix-manager-rbac-prod.sh
# 修复 argocd-manager ClusterRole/CRB 被误删导致的跨集群 SyncFailed
#
# 症状：
#   User "system:serviceaccount:kube-system:argocd-manager" cannot get resource
#   "serviceaccounts/services/..." in namespace "xxx" → SyncFailed
#
# 执行集群：生产目标集群（被外部 ArgoCD 管理的那个集群，不是源 ArgoCD 集群）
#
# 使用方法：
#   kubectl config use-context 
#   ./argocd-fix-manager-rbac-prod.sh
set -euo pipefail

echo "====================================================================="
echo "  修复 argocd-manager RBAC"
echo "  集群: $(kubectl config current-context 2>/dev/null || echo 'unknown')"
echo "  时间: $(date '+%Y-%m-%d %H:%M:%S')"
echo "====================================================================="
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 1: 确认现状
# ─────────────────────────────────────────────────────────────────────────────
echo "[1/3] 检查当前状态..."
echo -n "  ServiceAccount argocd-manager (kube-system): "
kubectl get sa argocd-manager -n kube-system &>/dev/null && echo "存在 ✅" || echo "不存在 ⚠️"

echo -n "  ClusterRole argocd-manager-role:             "
kubectl get clusterrole argocd-manager-role &>/dev/null && echo "存在 ✅" || echo "不存在（将重建）"

echo -n "  ClusterRoleBinding argocd-manager-role-binding: "
kubectl get clusterrolebinding argocd-manager-role-binding &>/dev/null && echo "存在 ✅" || echo "不存在（将重建）"
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 2: 重建 ClusterRole + ClusterRoleBinding
# ─────────────────────────────────────────────────────────────────────────────
echo "[2/3] 重建 ClusterRole / ClusterRoleBinding..."
kubectl apply -f - <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argocd-manager-role
rules:
- apiGroups: ['*']
  resources: ['*']
  verbs: ['*']
- nonResourceURLs: ['*']
  verbs: ['*']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argocd-manager-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: argocd-manager-role
subjects:
- kind: ServiceAccount
  name: argocd-manager
  namespace: kube-system
EOF
echo "  ✅ ClusterRole + ClusterRoleBinding 已重建"
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 3: 验证权限
# ─────────────────────────────────────────────────────────────────────────────
echo "[3/3] 验证权限恢复..."
for resource in serviceaccounts services deployments; do
    result=$(kubectl auth can-i get "${resource}" \
        --as=system:serviceaccount:kube-system:argocd-manager \
        --all-namespaces 2>/dev/null || echo "no")
    if [[ "${result}" == "yes" ]]; then
        echo "  ✅ get ${resource}: yes"
    else
        echo "  ❌ get ${resource}: ${result}"
    fi
done
echo ""

echo "====================================================================="
echo "  修复完成 ✅"
echo ""
echo "  下一步：在源 ArgoCD 触发重新同步"
echo "    argocd app sync "
echo "    或在 ArgoCD UI 点击 Sync / Refresh"
echo "====================================================================="
    argocd-fix-manager-rbac-prod.sh
    #!/usr/bin/env bash
# argocd-upgrade-post-v31-prod.sh
# ArgoCD v3.0.x → v3.1.x 升级后修复脚本（argocd 生产环境，第一跳）
#
# 生产集群说明：
#   - NS="argocd"（生产命名空间）
#   - replicas=2（升级后强制恢复，install.yaml 覆盖时可能降为 1）
#   - Redis HA：argocd-redis-ha-haproxy:6379
#   - argocd-manager ClusterRole 受保护（kubectl apply 不删除，此处仅校验现状）
#
# 使用方法：
#   1. kubectl apply -f argocd-install-v31-ha-prod.yaml -n argocd
#   2. 等待所有 Pod Running
#   3. 立即执行本脚本
set -euo pipefail

NS="argocd"
REDIS_HA_ADDR="argocd-redis-ha-haproxy:6379"

echo "====================================================================="
echo "  ArgoCD v3.0.x → v3.1.x 升级后修复（argocd 生产环境 — 第一跳）"
echo "  命名空间: ${NS}"
echo "  时间: $(date '+%Y-%m-%d %H:%M:%S')"
echo "====================================================================="
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# 工具函数
# ─────────────────────────────────────────────────────────────────────────────
has_node_taint() {
    kubectl get nodes -o jsonpath='{.items[*].spec.taints[*].key}' 2>/dev/null \
        | tr ' ' '\n' | grep -q "^${1}$"
}

check_pods_healthy() {
    local bad
    # 排除 Terminating：rolling upgrade 过渡期旧 Pod 正常处于此状态
    bad=$(kubectl -n "${NS}" get pods --no-headers 2>/dev/null \
        | grep -v "Running\|Completed\|Terminating" | grep -c "." || true)
    if [[ "${bad}" -gt 0 ]]; then
        echo "  ⚠️  有 ${bad} 个 Pod 未处于 Running 状态："
        kubectl -n "${NS}" get pods --no-headers | grep -v "Running\|Completed\|Terminating" || true
        echo ""
        echo "  ⛔ 回滚触发条件：CrashLoopBackOff > 5min / Degraded App > 30%"
        echo "  回滚命令：kubectl apply -f argocd-install-v3.0.6-ha-prod.yaml -n ${NS}"
        return 1
    fi
}

# ─────────────────────────────────────────────────────────────────────────────
# Step 0: 前置校验
# ─────────────────────────────────────────────────────────────────────────────
echo "[0/6] 前置校验..."

CURRENT_IMG=$(kubectl -n "${NS}" get deployment argocd-server \
    -o jsonpath='{.spec.template.spec.containers[?(@.name=="argocd-server")].image}' \
    2>/dev/null || echo "unknown")
echo "  当前 argocd-server 镜像: ${CURRENT_IMG}"
if echo "${CURRENT_IMG}" | grep -qE "v3\.0\."; then
    echo "  ⚠️  镜像仍是 v3.0.x，apply 未完成，请等待 rollout 后重试"
    exit 1
fi

# 前置 Pod 状态仅作诊断快照，不阻断脚本（redis config 尚未恢复，Pod 可能 CrashLoop）
check_pods_healthy || echo "  ⚠️  存在异常 Pod，将继续尝试恢复配置..."

# argocd-manager ClusterRole 保护校验（kubectl apply 不会删除此资源，此处仅记录现状）
echo "  [保护校验] argocd-manager ClusterRole..."
if kubectl get clusterrole argocd-manager-role &>/dev/null; then
    echo "  ✅ ClusterRole argocd-manager-role 存在，apply 不影响"
else
    echo "  ⚠️  ClusterRole argocd-manager-role 不存在！请在目标集群检查："
    echo "       kubectl config use-context "
    echo "       ./argocd-test/argocd-fix-manager-rbac-test.sh"
fi

# 确认 Redis HA HAProxy 服务存在
if ! kubectl -n "${NS}" get service argocd-redis-ha-haproxy &>/dev/null; then
    echo "  ⛔ argocd-redis-ha-haproxy Service 不存在！Redis HA 异常，中止修复"
    echo "  请先确认 Redis HA 状态: kubectl -n ${NS} get pods -l app=argocd-redis-ha"
    exit 1
fi
echo "  ✅ 前置校验通过"
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Steps 1+2: 原子恢复 redis.server + server.insecure
# ─────────────────────────────────────────────────────────────────────────────
echo "[1-2/6] 检查并原子恢复 redis.server + server.insecure..."

CURRENT_REDIS=$(kubectl -n "${NS}" get configmap argocd-cmd-params-cm \
    -o jsonpath='{.data.redis\.server}' 2>/dev/null || echo "")
INSECURE_VAL=$(kubectl -n "${NS}" get configmap argocd-cmd-params-cm \
    -o jsonpath='{.data.server\.insecure}' 2>/dev/null || echo "")

CONFIG_CHANGED=0
if [[ "${CURRENT_REDIS}" != "${REDIS_HA_ADDR}" || "${INSECURE_VAL}" != "true" ]]; then
    echo "  当前: redis.server=[${CURRENT_REDIS}]  server.insecure=[${INSECURE_VAL}]"
    kubectl -n "${NS}" patch configmap argocd-cmd-params-cm \
        --type merge \
        -p "{\"data\":{\"redis.server\":\"${REDIS_HA_ADDR}\",\"server.insecure\":\"true\"}}"
    echo "  ✅ 已原子写入: redis.server=${REDIS_HA_ADDR}  server.insecure=true"
    CONFIG_CHANGED=1
else
    echo "  ✅ redis.server + server.insecure 均已正确，无需修改"
fi
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 3: 生产环境强制恢复 replicas=2
# ─────────────────────────────────────────────────────────────────────────────
echo "[3/6] 检查并恢复 replicas=2（生产必须为 2）..."
REPLICAS_CHANGED=0
for deploy in argocd-server argocd-repo-server; do
    cur=$(kubectl -n "${NS}" get deployment "${deploy}" \
        -o jsonpath='{.spec.replicas}' 2>/dev/null || echo "0")
    if [[ "${cur}" -lt 2 ]]; then
        kubectl -n "${NS}" scale deployment "${deploy}" --replicas=2
        echo "  ✅ ${deploy}: replicas ${cur} → 2"
        REPLICAS_CHANGED=1
    else
        echo "  ✅ ${deploy}: replicas=${cur}（已满足 ≥ 2）"
    fi
done
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 4: tolerations
# ─────────────────────────────────────────────────────────────────────────────
echo "[4/6] 检查 tolerations..."
if has_node_taint "sretool"; then
    echo "  检测到节点存在 sretool:NoSchedule taint，恢复 tolerations..."
    TOLERATION_PATCH='{"spec":{"template":{"spec":{"tolerations":[{"key":"sretool","operator":"Equal","value":"true","effect":"NoSchedule"}]}}}}'
    kubectl -n "${NS}" patch deployment argocd-server      -p "${TOLERATION_PATCH}"
    kubectl -n "${NS}" patch deployment argocd-repo-server -p "${TOLERATION_PATCH}"
    echo "  ✅ tolerations 已恢复"
else
    echo "  ✅ 节点无 sretool taint，跳过"
fi
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 5: 重启所有组件，使 redis.server + insecure 配置生效
# ─────────────────────────────────────────────────────────────────────────────
echo "[5/6] 重启组件使配置生效..."
if [[ "${CONFIG_CHANGED}" -eq 0 && "${REPLICAS_CHANGED}" -eq 0 ]]; then
    echo "  ✅ CM 配置和 replicas 均已正确，跳过重启节省时间"
    echo "     如需强制重启: kubectl -n ${NS} rollout restart deployment/argocd-server deployment/argocd-repo-server"
else
    kubectl -n "${NS}" rollout restart deployment/argocd-server
    kubectl -n "${NS}" rollout restart deployment/argocd-repo-server
    kubectl -n "${NS}" rollout restart deployment/argocd-notifications-controller
    kubectl -n "${NS}" rollout restart statefulset/argocd-application-controller
    # applicationset-controller 可能未读 redis config，但一并重启保证 CM 变更生效
    (kubectl -n "${NS}" rollout restart deployment/argocd-applicationset-controller 2>/dev/null || true)
    ROLLOUT_PIDS=(); ROLLOUT_NAMES=()
    kubectl -n "${NS}" rollout status deployment/argocd-server                   --timeout=180s &
    ROLLOUT_PIDS+=($!); ROLLOUT_NAMES+=("argocd-server")
    kubectl -n "${NS}" rollout status deployment/argocd-repo-server              --timeout=180s &
    ROLLOUT_PIDS+=($!); ROLLOUT_NAMES+=("argocd-repo-server")
    kubectl -n "${NS}" rollout status deployment/argocd-notifications-controller --timeout=180s &
    ROLLOUT_PIDS+=($!); ROLLOUT_NAMES+=("argocd-notifications-controller")
    kubectl -n "${NS}" rollout status statefulset/argocd-application-controller  --timeout=300s &
    ROLLOUT_PIDS+=($!); ROLLOUT_NAMES+=("argocd-application-controller")
    (kubectl -n "${NS}" rollout status deployment/argocd-applicationset-controller --timeout=180s || true) &
    APPSET_PID=$!
    ROLLOUT_FAIL=0
    for i in "${!ROLLOUT_PIDS[@]}"; do
        if ! wait "${ROLLOUT_PIDS[$i]}"; then
            echo "  ❌ ${ROLLOUT_NAMES[$i]} rollout 超时/失败"
            ROLLOUT_FAIL=$((ROLLOUT_FAIL + 1))
        fi
    done
    wait "${APPSET_PID}" || true
    [[ "${ROLLOUT_FAIL}" -eq 0 ]] || { echo "  ⛔ ${ROLLOUT_FAIL} 个组件 rollout 失败，请检查上方详情"; exit 1; }
fi

check_pods_healthy
echo "  ✅ 所有组件重启完成，连接 Redis HA: ${REDIS_HA_ADDR}"
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 6: Redis HA 连通性验证
# ─────────────────────────────────────────────────────────────────────────────
echo "[6/6] Redis HA 连通性验证..."

HA_POD=$(kubectl -n "${NS}" get pod -l app.kubernetes.io/name=argocd-redis-ha \
    -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "")
# fallback: 兼容不同版本 label
[[ -z "${HA_POD}" ]] && HA_POD=$(kubectl -n "${NS}" get pod \
    --field-selector=status.phase=Running \
    -o name 2>/dev/null | grep "argocd-redis-ha-server" | head -1 | sed 's|pod/||' || echo "")
if [[ -n "${HA_POD}" ]]; then
    PING=$(kubectl -n "${NS}" exec "${HA_POD}" -c redis -- \
        redis-cli -h argocd-redis-ha-haproxy -p 6379 ping 2>/dev/null || echo "FAIL")
    if [[ "${PING}" == "PONG" ]]; then
        echo "  ✅ Redis HA HAProxy 连通 [${HA_POD} → argocd-redis-ha-haproxy:6379]: PONG"
    else
        echo "  ⚠️  Redis HA HAProxy ping 失败，请检查:"
        echo "  kubectl -n ${NS} get pods -l app=argocd-redis-ha-haproxy"
        echo "  kubectl -n ${NS} logs -l app=argocd-redis-ha-haproxy --tail=50"
        echo "  kubectl -n ${NS} logs ${HA_POD} -c redis --tail=50"
    fi
else
    echo "  ⚠️  未找到 Redis HA Pod（label: app=argocd-redis-ha），请手动验证"
fi
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# 最终验证
# ─────────────────────────────────────────────────────────────────────────────
echo "====================================================================="
echo "  最终验证"
echo "====================================================================="
echo ""
echo "--- Pod 状态 ---"
kubectl -n "${NS}" get pods -o wide

echo ""
echo "--- argocd-cmd-params-cm 关键字段 ---"
echo -n "  redis.server:    "; kubectl -n "${NS}" get configmap argocd-cmd-params-cm -o jsonpath='{.data.redis\.server}'; echo ""
echo -n "  server.insecure: "; kubectl -n "${NS}" get configmap argocd-cmd-params-cm -o jsonpath='{.data.server\.insecure}'; echo ""

echo ""
echo "--- replicas ---"
for deploy in argocd-server argocd-repo-server; do
    echo -n "  ${deploy}: "
    kubectl -n "${NS}" get deployment "${deploy}" -o jsonpath='{.spec.replicas}' 2>/dev/null; echo ""
done

echo ""
echo "====================================================================="
echo "  修复完成 ✅  (argocd 生产环境 — 第一跳)"
echo ""
echo "  [验证重点]"
echo "    ✅ install.yaml apply 后 redis.server 是否被重置 → 脚本已恢复"
echo "    ✅ Redis HA HAProxy 连通性"
echo "    ✅ server.insecure 已恢复"
echo "    ✅ replicas=2 已强制恢复"
echo ""
echo "  下一步: 第二跳升级"
echo "    kubectl apply -f argocd-install-v32-ha-prod.yaml -n ${NS}"
echo "    ./argocd-upgrade-post-v32-prod.sh"
echo ""
echo "  [回滚操作] 如升级后出现 CrashLoopBackOff > 5min 或 Degraded App > 30%："
echo "    # 1. 回滚镜像："
echo "    kubectl apply -f argocd-install-v3.0.6-ha-prod.yaml -n ${NS}"
echo "    # 2. 强制恢复 CM（install.yaml apply 会重置 argocd-cmd-params-cm，必须立即 patch）："
echo "    kubectl -n ${NS} patch configmap argocd-cmd-params-cm --type merge \\"
echo "        -p '{\"data\":{\"redis.server\":\"argocd-redis-ha-haproxy:6379\",\"server.insecure\":\"true\"}}'"
echo "    # 3. 恢复 replicas=2："
echo "    kubectl -n ${NS} scale deployment argocd-server argocd-repo-server --replicas=2"
echo "    # 4. 重启使 CM 生效："
echo "    kubectl -n ${NS} rollout restart deployment/argocd-server deployment/argocd-repo-server"
echo "    kubectl -n ${NS} rollout restart statefulset/argocd-application-controller"
echo "    kubectl -n ${NS} rollout status deployment/argocd-server --timeout=180s"
echo "    # 5. 若 argocd-manager RBAC 丢失（目标集群执行）："
echo "    #    kubectl config use-context "
echo "    #    ./argocd-test/argocd-fix-manager-rbac-test.sh"
echo "    # 6. 若 CR/ConfigMap 数据损坏，从备份完整恢复："
echo "    #    ./argocd-test/argocd-restore-test.sh argocd-backup-/"
echo "====================================================================="
    argocd-upgrade-post-v31-prod.sh
    #!/usr/bin/env bash
# argocd-upgrade-post-v32-prod.sh
# ArgoCD v3.1.x → v3.2.x 升级后修复脚本（argocd 生产环境，第二跳）
#
# 生产集群说明：
#   - NS="argocd"（生产命名空间）
#   - replicas=2（升级后强制恢复，install.yaml 覆盖时可能降为 1）
#   - Redis HA：argocd-redis-ha-haproxy:6379
#   - argocd-manager ClusterRole 受保护（kubectl apply 不删除，此处仅校验现状）
#
# 使用方法：
#   1. kubectl apply -f argocd-install-v32-ha-prod.yaml -n argocd
#   2. 等待所有 Pod Running
#   3. 立即执行本脚本
set -euo pipefail

NS="argocd"
REDIS_HA_ADDR="argocd-redis-ha-haproxy:6379"

echo "====================================================================="
echo "  ArgoCD v3.1.x → v3.2.x 升级后修复（argocd 生产环境 — 第二跳）"
echo "  命名空间: ${NS}"
echo "  时间: $(date '+%Y-%m-%d %H:%M:%S')"
echo "====================================================================="
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# 工具函数
# ─────────────────────────────────────────────────────────────────────────────
has_node_taint() {
    kubectl get nodes -o jsonpath='{.items[*].spec.taints[*].key}' 2>/dev/null \
        | tr ' ' '\n' | grep -q "^${1}$"
}

check_pods_healthy() {
    local bad
    # 排除 Terminating：rolling upgrade 过渡期旧 Pod 正常处于此状态
    bad=$(kubectl -n "${NS}" get pods --no-headers 2>/dev/null \
        | grep -v "Running\|Completed\|Terminating" | grep -c "." || true)
    if [[ "${bad}" -gt 0 ]]; then
        echo "  ⚠️  有 ${bad} 个 Pod 未处于 Running 状态："
        kubectl -n "${NS}" get pods --no-headers | grep -v "Running\|Completed\|Terminating" || true
        echo ""
        echo "  ⛔ 回滚触发条件：CrashLoopBackOff > 5min / Degraded App > 30%"
        echo "  回滚命令：kubectl apply -f argocd-install-v31-ha-prod.yaml -n ${NS}"
        return 1
    fi
}

# ─────────────────────────────────────────────────────────────────────────────
# Step 0: 前置校验
# ─────────────────────────────────────────────────────────────────────────────
echo "[0/6] 前置校验..."

CURRENT_IMG=$(kubectl -n "${NS}" get deployment argocd-server \
    -o jsonpath='{.spec.template.spec.containers[?(@.name=="argocd-server")].image}' \
    2>/dev/null || echo "unknown")
echo "  当前 argocd-server 镜像: ${CURRENT_IMG}"
if echo "${CURRENT_IMG}" | grep -qE "v3\.[01]\."; then
    echo "  ⚠️  镜像仍是 v3.0.x/v3.1.x，apply 未完成，请等待 rollout 后重试"
    exit 1
fi

# 前置 Pod 状态仅作诊断快照，不阻断脚本（redis config 尚未恢复，Pod 可能 CrashLoop）
check_pods_healthy || echo "  ⚠️  存在异常 Pod，将继续尝试恢复配置..."

# argocd-manager ClusterRole 保护校验（kubectl apply 不会删除此资源，此处仅记录现状）
echo "  [保护校验] argocd-manager ClusterRole..."
if kubectl get clusterrole argocd-manager-role &>/dev/null; then
    echo "  ✅ ClusterRole argocd-manager-role 存在，apply 不影响"
else
    echo "  ⚠️  ClusterRole argocd-manager-role 不存在！请在目标集群检查："
    echo "       kubectl config use-context "
    echo "       ./argocd-test/argocd-fix-manager-rbac-test.sh"
fi

# 确认 Redis HA HAProxy 服务存在
if ! kubectl -n "${NS}" get service argocd-redis-ha-haproxy &>/dev/null; then
    echo "  ⛔ argocd-redis-ha-haproxy Service 不存在！Redis HA 异常，中止修复"
    echo "  请先确认 Redis HA 状态: kubectl -n ${NS} get pods -l app=argocd-redis-ha"
    exit 1
fi
echo "  ✅ 前置校验通过"
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Steps 1+2: 原子恢复 redis.server + server.insecure
# ─────────────────────────────────────────────────────────────────────────────
echo "[1-2/6] 检查并原子恢复 redis.server + server.insecure..."

CURRENT_REDIS=$(kubectl -n "${NS}" get configmap argocd-cmd-params-cm \
    -o jsonpath='{.data.redis\.server}' 2>/dev/null || echo "")
INSECURE_VAL=$(kubectl -n "${NS}" get configmap argocd-cmd-params-cm \
    -o jsonpath='{.data.server\.insecure}' 2>/dev/null || echo "")

CONFIG_CHANGED=0
if [[ "${CURRENT_REDIS}" != "${REDIS_HA_ADDR}" || "${INSECURE_VAL}" != "true" ]]; then
    echo "  当前: redis.server=[${CURRENT_REDIS}]  server.insecure=[${INSECURE_VAL}]"
    kubectl -n "${NS}" patch configmap argocd-cmd-params-cm \
        --type merge \
        -p "{\"data\":{\"redis.server\":\"${REDIS_HA_ADDR}\",\"server.insecure\":\"true\"}}"
    echo "  ✅ 已原子写入: redis.server=${REDIS_HA_ADDR}  server.insecure=true"
    CONFIG_CHANGED=1
else
    echo "  ✅ redis.server + server.insecure 均已正确，无需修改"
fi
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 3: 生产环境强制恢复 replicas=2
# ─────────────────────────────────────────────────────────────────────────────
echo "[3/6] 检查并恢复 replicas=2（生产必须为 2）..."
REPLICAS_CHANGED=0
for deploy in argocd-server argocd-repo-server; do
    cur=$(kubectl -n "${NS}" get deployment "${deploy}" \
        -o jsonpath='{.spec.replicas}' 2>/dev/null || echo "0")
    if [[ "${cur}" -lt 2 ]]; then
        kubectl -n "${NS}" scale deployment "${deploy}" --replicas=2
        echo "  ✅ ${deploy}: replicas ${cur} → 2"
        REPLICAS_CHANGED=1
    else
        echo "  ✅ ${deploy}: replicas=${cur}（已满足 ≥ 2）"
    fi
done
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 4: tolerations
# ─────────────────────────────────────────────────────────────────────────────
echo "[4/6] 检查 tolerations..."
if has_node_taint "sretool"; then
    echo "  检测到节点存在 sretool:NoSchedule taint，恢复 tolerations..."
    TOLERATION_PATCH='{"spec":{"template":{"spec":{"tolerations":[{"key":"sretool","operator":"Equal","value":"true","effect":"NoSchedule"}]}}}}'
    kubectl -n "${NS}" patch deployment argocd-server      -p "${TOLERATION_PATCH}"
    kubectl -n "${NS}" patch deployment argocd-repo-server -p "${TOLERATION_PATCH}"
    echo "  ✅ tolerations 已恢复"
else
    echo "  ✅ 节点无 sretool taint，跳过"
fi
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 5: 重启所有组件（v3.2.x 额外检查 applicationset-controller RBAC）
# ─────────────────────────────────────────────────────────────────────────────
echo "[5/6] 重启组件使配置生效..."
if [[ "${CONFIG_CHANGED}" -eq 0 && "${REPLICAS_CHANGED}" -eq 0 ]]; then
    echo "  ✅ CM 配置和 replicas 均已正确，跳过重启节省时间"
    echo "     如需强制重启: kubectl -n ${NS} rollout restart deployment/argocd-server deployment/argocd-repo-server"
else
    kubectl -n "${NS}" rollout restart deployment/argocd-server
    kubectl -n "${NS}" rollout restart deployment/argocd-repo-server
    kubectl -n "${NS}" rollout restart deployment/argocd-notifications-controller
    kubectl -n "${NS}" rollout restart statefulset/argocd-application-controller
    # v3.2.x 额外：applicationset-controller RBAC 变更（新增 coordination.k8s.io/leases）
    ROLLOUT_PIDS=(); ROLLOUT_NAMES=()
    kubectl -n "${NS}" rollout status deployment/argocd-server                      --timeout=180s &
    ROLLOUT_PIDS+=($!); ROLLOUT_NAMES+=("argocd-server")
    kubectl -n "${NS}" rollout status deployment/argocd-repo-server                 --timeout=180s &
    ROLLOUT_PIDS+=($!); ROLLOUT_NAMES+=("argocd-repo-server")
    kubectl -n "${NS}" rollout status deployment/argocd-notifications-controller    --timeout=180s &
    ROLLOUT_PIDS+=($!); ROLLOUT_NAMES+=("argocd-notifications-controller")
    kubectl -n "${NS}" rollout status statefulset/argocd-application-controller     --timeout=300s &
    ROLLOUT_PIDS+=($!); ROLLOUT_NAMES+=("argocd-application-controller")
    # v3.2 中 applicationset-controller 可能被重命名，单独追踪，失败不阻断必要组件
    (kubectl -n "${NS}" rollout status deployment/argocd-applicationset-controller  --timeout=180s || true) &
    APPSET_PID=$!
    ROLLOUT_FAIL=0
    for i in "${!ROLLOUT_PIDS[@]}"; do
        if ! wait "${ROLLOUT_PIDS[$i]}"; then
            echo "  ❌ ${ROLLOUT_NAMES[$i]} rollout 超时/失败"
            ROLLOUT_FAIL=$((ROLLOUT_FAIL + 1))
        fi
    done
    wait "${APPSET_PID}" || true  # applicationset-controller 可选，失败不阻断
    [[ "${ROLLOUT_FAIL}" -eq 0 ]] || { echo "  ⛔ ${ROLLOUT_FAIL} 个必要组件 rollout 失败"; exit 1; }
fi

# v3.2 特有：检查 applicationset-controller RBAC 错误
APPSET_FORBIDDEN=$(kubectl -n "${NS}" logs \
    -l app.kubernetes.io/name=argocd-applicationset-controller \
    --tail=200 --since=30m 2>/dev/null | grep -c "Forbidden\|RBAC\|cannot" || true)
if [[ "${APPSET_FORBIDDEN}" -gt 0 ]]; then
    echo "  ⚠️  applicationset-controller 发现 ${APPSET_FORBIDDEN} 条 RBAC/Forbidden 日志："
    kubectl -n "${NS}" logs \
        -l app.kubernetes.io/name=argocd-applicationset-controller \
        --tail=200 --since=30m | grep "Forbidden\|RBAC\|cannot"
else
    echo "  ✅ applicationset-controller 无 RBAC 错误"
fi

check_pods_healthy
echo "  ✅ 所有组件重启完成，连接 Redis HA: ${REDIS_HA_ADDR}"
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# Step 6: Redis HA 连通性验证
# ─────────────────────────────────────────────────────────────────────────────
echo "[6/6] Redis HA 连通性验证..."

HA_POD=$(kubectl -n "${NS}" get pod -l app.kubernetes.io/name=argocd-redis-ha \
    -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "")
# fallback: 兼容不同版本 label
[[ -z "${HA_POD}" ]] && HA_POD=$(kubectl -n "${NS}" get pod \
    --field-selector=status.phase=Running \
    -o name 2>/dev/null | grep "argocd-redis-ha-server" | head -1 | sed 's|pod/||' || echo "")
if [[ -n "${HA_POD}" ]]; then
    PING=$(kubectl -n "${NS}" exec "${HA_POD}" -c redis -- \
        redis-cli -h argocd-redis-ha-haproxy -p 6379 ping 2>/dev/null || echo "FAIL")
    if [[ "${PING}" == "PONG" ]]; then
        echo "  ✅ Redis HA HAProxy 连通 [${HA_POD} → argocd-redis-ha-haproxy:6379]: PONG"
    else
        echo "  ⚠️  Redis HA HAProxy ping 失败，请检查:"
        echo "  kubectl -n ${NS} get pods -l app=argocd-redis-ha-haproxy"
        echo "  kubectl -n ${NS} logs -l app=argocd-redis-ha-haproxy --tail=50"
        echo "  kubectl -n ${NS} logs ${HA_POD} -c redis --tail=50"
    fi
else
    echo "  ⚠️  未找到 Redis HA Pod（label: app=argocd-redis-ha），请手动验证"
fi
echo ""

# ─────────────────────────────────────────────────────────────────────────────
# 最终验证
# ─────────────────────────────────────────────────────────────────────────────
echo "====================================================================="
echo "  最终验证"
echo "====================================================================="
echo ""
echo "--- Pod 状态 ---"
kubectl -n "${NS}" get pods -o wide

echo ""
echo "--- argocd-cmd-params-cm 关键字段 ---"
echo -n "  redis.server:    "; kubectl -n "${NS}" get configmap argocd-cmd-params-cm -o jsonpath='{.data.redis\.server}'; echo ""
echo -n "  server.insecure: "; kubectl -n "${NS}" get configmap argocd-cmd-params-cm -o jsonpath='{.data.server\.insecure}'; echo ""

echo ""
echo "--- replicas ---"
for deploy in argocd-server argocd-repo-server; do
    echo -n "  ${deploy}: "
    kubectl -n "${NS}" get deployment "${deploy}" -o jsonpath='{.spec.replicas}' 2>/dev/null; echo ""
done

echo ""
echo "====================================================================="
echo "  修复完成 ✅  (argocd 生产环境 — 第二跳)"
echo ""
echo "  [v3.2.x 行为变更验证要点]"
echo "    1. applicationset-controller RBAC: coordination.k8s.io/leases 权限"
echo "       检查: kubectl get clusterrole argocd-applicationset-controller -o yaml | grep coordination"
echo "    2. install.yaml apply 后 redis.server 被重置 → 脚本已恢复"
echo "    3. argocd-cmd-params-cm 中 server.insecure 被重置 → 脚本已恢复"
echo "    ✅ replicas=2 已强制恢复"
echo ""
echo "  [回滚操作] 如升级后出现 CrashLoopBackOff > 5min 或 Degraded App > 30%："
echo "    # 1. 回滚至 v3.1.x："
echo "    kubectl apply -f argocd-install-v31-ha-prod.yaml -n ${NS}"
echo "    # 2. 强制恢复 CM（install.yaml apply 会重置 argocd-cmd-params-cm，必须立即 patch）："
echo "    kubectl -n ${NS} patch configmap argocd-cmd-params-cm --type merge \\"
echo "        -p '{\"data\":{\"redis.server\":\"argocd-redis-ha-haproxy:6379\",\"server.insecure\":\"true\"}}'"
echo "    # 3. 恢复 replicas=2："
echo "    kubectl -n ${NS} scale deployment argocd-server argocd-repo-server --replicas=2"
echo "    # 4. 重启使 CM 生效："
echo "    kubectl -n ${NS} rollout restart deployment/argocd-server deployment/argocd-repo-server"
echo "    kubectl -n ${NS} rollout restart statefulset/argocd-application-controller"
echo "    kubectl -n ${NS} rollout status deployment/argocd-server --timeout=180s"
echo "    # 5. 若需完整回滚至 v3.0.6（重复步骤 1-4，使用 v3.0.6 yaml）："
echo "    #    kubectl apply -f argocd-install-v3.0.6-ha-prod.yaml -n ${NS}"
echo "    #    kubectl -n ${NS} patch configmap argocd-cmd-params-cm --type merge \\"
echo "    #        -p '{\"data\":{\"redis.server\":\"argocd-redis-ha-haproxy:6379\",\"server.insecure\":\"true\"}}'"
echo "    #    kubectl -n ${NS} scale deployment argocd-server argocd-repo-server --replicas=2"
echo "    #    kubectl -n ${NS} rollout restart deployment/argocd-server deployment/argocd-repo-server"
echo "    #    kubectl -n ${NS} rollout restart statefulset/argocd-application-controller"
echo "    # 6. 若 argocd-manager RBAC 丢失（目标集群执行）："
echo "    #    kubectl config use-context "
echo "    #    ./argocd-test/argocd-fix-manager-rbac-test.sh"
echo "    # 7. 若 CR/ConfigMap 数据损坏，从备份完整恢复："
echo "    #    ./argocd-test/argocd-restore-test.sh argocd-backup-/"
echo "====================================================================="
    argocd-upgrade-post-v32-prod.sh

     


原文地址: https://www.cveoy.top/t/topic/qHf2 著作权归作者所有。请勿转载和采集!

免费AI点我,无需注册和登录