[{"data":1,"prerenderedAt":3081},["ShallowReactive",2],{"\u002F2026-05-22-fa-012-gwbridge":3,"\u002F2026-05-22-fa-012-gwbridge-rel":1281},{"id":4,"title":5,"body":6,"column":1263,"date":1264,"description":12,"extension":1265,"hero_image":1266,"meta":1267,"navigation":1268,"path":1269,"seo":1270,"series_id":1271,"severity":1266,"stem":1272,"summary":1273,"tags":1274,"__hash__":1280},"posts\u002F2026-05-22-FA-012-gwbridge容量上限.md","一个默认配置，决定了我能有多少用户",{"type":7,"value":8,"toc":1255},"minimark",[9,13,17,20,23,26,29,32,35,38,41,47,55,65,68,149,152,174,177,180,185,192,244,247,250,253,283,286,291,294,319,322,327,330,335,338,341,406,409,412,417,420,423,426,462,465,468,496,499,502,509,514,517,520,526,578,581,586,593,596,615,618,621,626,629,634,637,689,696,707,714,717,804,810,815,821,875,878,881,886,889,892,895,900,903,906,911,914,920,923,930,935,941,944,951,956,959,1003,1006,1009,1027,1037,1043,1046,1051,1054,1068,1075,1080,1083,1088,1091,1094,1098,1104,1113,1129,1135,1138,1141,1167,1170,1190,1193,1196,1201,1216,1221,1224,1227,1251],[10,11,12],"p",{},"我以为这是十分钟的活：一行 apt 改动，给 runtime 镜像加 ffmpeg，打包、灰度、完。结果花了三小时多。不是因为代码复杂，而是在灰度的那一刻，整个基础设施的隐性容量上限暴露了——Docker Swarm 的 gwbridge 只有 253 个 IP，现在已经用满了。",[14,15,16],"h2",{"id":16},"现象",[10,18,19],{},"早上 7:30，用户报 AI 容器里没有 ffmpeg，无法剪视频。表面上这是一行 Dockerfile 改动——apt 列表里漏了 ffmpeg。",[10,21,22],{},"一小时内排查完毕：容器确实没装这个工具。改 Dockerfile、打包新镜像（1.0.18），准备灰度。",[10,24,25],{},"然后，在给一个用户灰度升级时，事情出了岔子。",[10,27,28],{},"Swarm 调度了一个新的 task 去 pull 新镜像，但 gwbridge（Docker overlay 网络中负责容器出站的虚拟网桥）拿不到 IP。Task 失败了。灰度用户被卡在 0\u002F1 状态。自动回滚也失败了。",[10,30,31],{},"同时看 prod 集群的 service 状态，有 10 个长期处于 cycling 启动失败、重试、又失败的状态——这些现象其实已经存在很久，但因为看起来像是\"偶发起不来\"，一直被当作瞬时故障忽视了。现在它们集中暴露了。",[10,33,34],{},"很快发现真问题：生产环境的 gwbridge 是 \u002F24 网段，253 个可用 IP 已经占满。当前有 251 个用户容器 + ingress network 的 1 个 endpoint + 其他系统组件 = 已用 253\u002F253。任何新容器启动时需要分配 endpoint，但池子空了——没有可用 IP，task 调度失败，开始 cycling。",[10,36,37],{},"那 10-13 个看起来\"偶发失败\"的 service？它们就是一直在试图找到那不存在的第 254 个 IP。",[14,39,40],{"id":40},"排查过程",[10,42,43],{},[44,45,46],"strong",{},"8:10 — 首先检查 service 状态",[10,48,49,50,54],{},"连接到生产服务器，运行 ",[51,52,53],"code",{},"docker service ls","，发现当前 service 的复制状态：",[56,57,62],"pre",{"className":58,"code":60,"language":61},[59],"language-text","ID          NAME              MODE        REPLICAS    IMAGE\n...\n251 个 RUNNING 状态的 service，每个 replicas=1\u002F1\n10 个 RUNNING 状态但 replicas=0\u002F1，反复 cycling\n","text",[51,63,60],{"__ignoreMap":64},"",[10,66,67],{},"所有用户的容器中有 251 个正常运行，另外 10 个卡在了反复重试的状态。再深入看 gwbridge 的配置：",[56,69,73],{"className":70,"code":71,"language":72,"meta":64,"style":64},"language-bash shiki shiki-themes github-light github-dark","$ docker network inspect gwbridge\n[{\n  \"Name\": \"docker_gwbridge\",\n  \"Subnet\": \"172.30.1.0\u002F24\",\n  \"Gateway\": \"172.30.1.1\"\n}]\n","bash",[51,74,75,97,104,119,132,143],{"__ignoreMap":64},[76,77,80,84,88,91,94],"span",{"class":78,"line":79},"line",1,[76,81,83],{"class":82},"sScJk","$",[76,85,87],{"class":86},"sZZnC"," docker",[76,89,90],{"class":86}," network",[76,92,93],{"class":86}," inspect",[76,95,96],{"class":86}," gwbridge\n",[76,98,100],{"class":78,"line":99},2,[76,101,103],{"class":102},"sVt8B","[{\n",[76,105,107,110,113,116],{"class":78,"line":106},3,[76,108,109],{"class":86},"  \"Name\"",[76,111,112],{"class":102},": ",[76,114,115],{"class":86},"\"docker_gwbridge\"",[76,117,118],{"class":102},",\n",[76,120,122,125,127,130],{"class":78,"line":121},4,[76,123,124],{"class":86},"  \"Subnet\"",[76,126,112],{"class":102},[76,128,129],{"class":86},"\"172.30.1.0\u002F24\"",[76,131,118],{"class":102},[76,133,135,138,140],{"class":78,"line":134},5,[76,136,137],{"class":86},"  \"Gateway\"",[76,139,112],{"class":102},[76,141,142],{"class":86},"\"172.30.1.1\"\n",[76,144,146],{"class":78,"line":145},6,[76,147,148],{"class":102},"}]\n",[10,150,151],{},"\u002F24 网段分配给了 gwbridge。接下来检查当前分配的 IP 究竟用到了哪里：",[56,153,155],{"className":70,"code":154,"language":72,"meta":64,"style":64},"$ docker network inspect gwbridge --verbose\n",[51,156,157],{"__ignoreMap":64},[76,158,159,161,163,165,167,170],{"class":78,"line":79},[76,160,83],{"class":82},[76,162,87],{"class":86},[76,164,90],{"class":86},[76,166,93],{"class":86},[76,168,169],{"class":86}," gwbridge",[76,171,173],{"class":172},"sj4cs"," --verbose\n",[10,175,176],{},"一遍遍历下来，251 个用户容器占了 endpoint，加上 ingress 及其他系统组件的占位，gwbridge 内 253 个可用 IP 已经无余量。\u002F24 网段理论上是 254 个 IP（.0 到 .255），但除去网络地址 .0 和广播地址 .255，可用的就是 253 个（.1 网关也在其中）。",[10,178,179],{},"已用 253\u002F253。没有余量。这就是为什么新容器无法获取 IP，灰度失败了。",[10,181,182],{},[44,183,184],{},"9:00 — 假设 1：改 daemon.json 配置",[10,186,187,188,191],{},"想过扩大 gwbridge 的网段。改 ",[51,189,190],{},"\u002Fetc\u002Fdocker\u002Fdaemon.json","：",[56,193,197],{"className":194,"code":195,"language":196,"meta":64,"style":64},"language-json shiki shiki-themes github-light github-dark","{\n  \"default-address-pools\": [{\n    \"base\": \"172.31.0.0\u002F16\",\n    \"size\": 22\n  }]\n}\n","json",[51,198,199,204,212,224,234,239],{"__ignoreMap":64},[76,200,201],{"class":78,"line":79},[76,202,203],{"class":102},"{\n",[76,205,206,209],{"class":78,"line":99},[76,207,208],{"class":172},"  \"default-address-pools\"",[76,210,211],{"class":102},": [{\n",[76,213,214,217,219,222],{"class":78,"line":106},[76,215,216],{"class":172},"    \"base\"",[76,218,112],{"class":102},[76,220,221],{"class":86},"\"172.31.0.0\u002F16\"",[76,223,118],{"class":102},[76,225,226,229,231],{"class":78,"line":121},[76,227,228],{"class":172},"    \"size\"",[76,230,112],{"class":102},[76,232,233],{"class":172},"22\n",[76,235,236],{"class":78,"line":134},[76,237,238],{"class":102},"  }]\n",[76,240,241],{"class":78,"line":145},[76,242,243],{"class":102},"}\n",[10,245,246],{},"这样，下次创建网络时，Swarm 应该用 \u002F22（1022 个 IP）而不是 \u002F24（253 个）。",[10,248,249],{},"重启 docker daemon，让配置生效。等待 60 秒，让集群自愈。",[10,251,252],{},"再看 gwbridge：",[56,254,256],{"className":70,"code":255,"language":72,"meta":64,"style":64},"$ docker info | grep \"Default Address Pools:\" -A 1\n",[51,257,258],{"__ignoreMap":64},[76,259,260,262,264,267,271,274,277,280],{"class":78,"line":79},[76,261,83],{"class":82},[76,263,87],{"class":86},[76,265,266],{"class":86}," info",[76,268,270],{"class":269},"szBVR"," |",[76,272,273],{"class":82}," grep",[76,275,276],{"class":86}," \"Default Address Pools:\"",[76,278,279],{"class":172}," -A",[76,281,282],{"class":172}," 1\n",[10,284,285],{},"没有输出。空。配置没读进去。",[10,287,288],{},[44,289,290],{},"9:15 — 尝试各种做法",[10,292,293],{},"想快速解决，试过几个办法：",[295,296,297,305,308],"ol",{},[298,299,300,301,304],"li",{},"删除 ",[51,302,303],{},"\u002Fvar\u002Flib\u002Fdocker\u002Fnetwork\u002Ffiles\u002Flocal-kv.db","（Docker 网络配置持久化库），强制重建。——不行。",[298,306,307],{},"确认 daemon.json 语法无误，重新加载 daemon 配置。——不行。",[298,309,310,311,314,315,318],{},"查阅 Docker 文档。发现一句关键的话：",[51,312,313],{},"default-address-pools"," 只在 ",[51,316,317],{},"swarm init"," 时消费一次。",[10,320,321],{},"意思是，这个配置项的生效时机是 Docker Swarm 初始化那一刻——在那时，Docker 会根据这个池子创建初始网络。但对于已经存在的网络，改这个配置不会有任何效果。gwbridge 是 5 天前 swarm init 时创建的。现在改 daemon.json 已经太晚。配置修改完全无法影响到已有网络的设置。",[10,323,324],{},[44,325,326],{},"9:30 — 改回配置，准备另一条路",[10,328,329],{},"把 daemon.json 恢复到原样，重启 docker。集群自愈回到\"252 healthy + 10 cycling\"状态。",[10,331,332],{},[44,333,334],{},"9:45 — Staging 验证",[10,336,337],{},"有个 staging 环境（同样的 Docker Swarm 架构，但用户数少得多）。用它做 dry-run。",[10,339,340],{},"方案：手动删除 gwbridge，再用新 subnet 重建。",[56,342,344],{"className":70,"code":343,"language":72,"meta":64,"style":64},"docker network rm docker_gwbridge\ndocker network create \\\n  --driver bridge \\\n  --subnet 172.31.0.0\u002F20 \\\n  --gateway 172.31.0.1 \\\n  docker_gwbridge\n",[51,345,346,359,371,381,391,401],{"__ignoreMap":64},[76,347,348,351,353,356],{"class":78,"line":79},[76,349,350],{"class":82},"docker",[76,352,90],{"class":86},[76,354,355],{"class":86}," rm",[76,357,358],{"class":86}," docker_gwbridge\n",[76,360,361,363,365,368],{"class":78,"line":99},[76,362,350],{"class":82},[76,364,90],{"class":86},[76,366,367],{"class":86}," create",[76,369,370],{"class":172}," \\\n",[76,372,373,376,379],{"class":78,"line":106},[76,374,375],{"class":172},"  --driver",[76,377,378],{"class":86}," bridge",[76,380,370],{"class":172},[76,382,383,386,389],{"class":78,"line":121},[76,384,385],{"class":172},"  --subnet",[76,387,388],{"class":86}," 172.31.0.0\u002F20",[76,390,370],{"class":172},[76,392,393,396,399],{"class":78,"line":134},[76,394,395],{"class":172},"  --gateway",[76,397,398],{"class":172}," 172.31.0.1",[76,400,370],{"class":172},[76,402,403],{"class":78,"line":145},[76,404,405],{"class":86},"  docker_gwbridge\n",[10,407,408],{},"Staging 上，105 秒内完成整个过程，所有 service 自动重新调度，容器全部拿回 IP，状态恢复。",[10,410,411],{},"确认这个方案可行。",[10,413,414],{},[44,415,416],{},"9:50 — Canary 试错",[10,418,419],{},"但真正动手前，想在 prod 上做个\"单用户 canary\"——选一个 idle 用户，试试 scale 0（停容器，释放 endpoint）然后 scale 1（重启容器，重新分配 endpoint）。这样可以在生产环境验证删除重建方案是否安全。",[10,421,422],{},"选了用户 B。",[10,424,425],{},"先 scale 0：",[56,427,429],{"className":70,"code":428,"language":72,"meta":64,"style":64},"docker service scale svc-\u003C用户B>=0\n",[51,430,431],{"__ignoreMap":64},[76,432,433,435,438,441,444,447,450,453,456,459],{"class":78,"line":79},[76,434,350],{"class":82},[76,436,437],{"class":86}," service",[76,439,440],{"class":86}," scale",[76,442,443],{"class":86}," svc-",[76,445,446],{"class":269},"\u003C",[76,448,449],{"class":86},"用户",[76,451,452],{"class":102},"B",[76,454,455],{"class":269},">",[76,457,458],{"class":86},"=",[76,460,461],{"class":172},"0\n",[10,463,464],{},"成功。容器停了，gwbridge 上的 endpoint 释放了。一个 IP 应该回归可用。",[10,466,467],{},"再 scale 1：",[56,469,471],{"className":70,"code":470,"language":72,"meta":64,"style":64},"docker service scale svc-\u003C用户B>=1\n",[51,472,473],{"__ignoreMap":64},[76,474,475,477,479,481,483,485,487,489,491,493],{"class":78,"line":79},[76,476,350],{"class":82},[76,478,437],{"class":86},[76,480,440],{"class":86},[76,482,443],{"class":86},[76,484,446],{"class":269},[76,486,449],{"class":86},[76,488,452],{"class":102},[76,490,455],{"class":269},[76,492,458],{"class":86},[76,494,495],{"class":172},"1\n",[10,497,498],{},"预期：gwbridge 应该有 1 个空 IP 现在可用，container 会拿到它。",[10,500,501],{},"实际：task 又启动失败了。为什么？",[10,503,504,505,508],{},"看日志发现：刚释放的那个 IP（比如 172.30.1.100），立刻被其他 cycling 的 task 抢走了。Swarm 内部有失联的 service 在不停地尝试获取 endpoint，它们的重试机制比 ",[51,506,507],{},"scale 1"," 的新 task 更激进。这种负反馈导致用户 B 的容器仍然拿不到 IP。",[10,510,511],{},[44,512,513],{},"9:55 — 关键假设验证",[10,515,516],{},"这个失败给了一个启发：能否先把那 13 个 cycling service 全部 scale 0，让它们停止抢占 IP？",[10,518,519],{},"这样就有 13 个 IP 可用，足够 gwbridge 扩容时临时周转。",[10,521,522,523,191],{},"给 13 个失联 service 全部执行 ",[51,524,525],{},"scale 0",[56,527,529],{"className":70,"code":528,"language":72,"meta":64,"style":64},"docker service ls --filter \"label=health=fail\" --format \"{{.Name}}\" | \\\n  xargs -I {} docker service scale {}=0\n",[51,530,531,556],{"__ignoreMap":64},[76,532,533,535,537,540,543,546,549,552,554],{"class":78,"line":79},[76,534,350],{"class":82},[76,536,437],{"class":86},[76,538,539],{"class":86}," ls",[76,541,542],{"class":172}," --filter",[76,544,545],{"class":86}," \"label=health=fail\"",[76,547,548],{"class":172}," --format",[76,550,551],{"class":86}," \"{{.Name}}\"",[76,553,270],{"class":269},[76,555,370],{"class":172},[76,557,558,561,564,567,569,571,573,576],{"class":78,"line":99},[76,559,560],{"class":82},"  xargs",[76,562,563],{"class":172}," -I",[76,565,566],{"class":86}," {}",[76,568,87],{"class":86},[76,570,437],{"class":86},[76,572,440],{"class":86},[76,574,575],{"class":86}," {}=",[76,577,461],{"class":172},[10,579,580],{},"都成功了。集群稳定，回到\"252 healthy + 0 failing\"（那 13 个现在是 0\u002F0）。",[10,582,583],{},[44,584,585],{},"10:10 — Staging 性能优化验证",[10,587,588,589,592],{},"刚意识到，扩容过程中，swarm 会试图 pull 新镜像。因为网络不好，pull 可能花 60+ 秒。能否通过修改 ",[51,590,591],{},"\u002Fetc\u002Fhosts"," 把 docker.io 黑洞掉，让 Swarm 直接 fallback 到本地镜像？",[10,594,595],{},"在 staging 上测试：在容器启动前加一行 hosts 黑洞：",[56,597,599],{"className":70,"code":598,"language":72,"meta":64,"style":64},"echo \"127.0.0.1 docker.io\" >> \u002Fetc\u002Fhosts\n",[51,600,601],{"__ignoreMap":64},[76,602,603,606,609,612],{"class":78,"line":79},[76,604,605],{"class":172},"echo",[76,607,608],{"class":86}," \"127.0.0.1 docker.io\"",[76,610,611],{"class":269}," >>",[76,613,614],{"class":86}," \u002Fetc\u002Fhosts\n",[10,616,617],{},"效果：scale 0 → scale 1 的过程从 63 秒降到 1 秒。",[10,619,620],{},"确认这个优化可行。会用到生产环境。",[10,622,623],{},[44,624,625],{},"10:25 — 完整备份",[10,627,628],{},"在真正动手删 gwbridge 前，备份所有 265 个 service 的 spec JSON（2.1MB）。以防万一 raft 数据库损坏，这是唯一的重建 service 的办法。",[10,630,631],{},[44,632,633],{},"10:42 — 真正扩容，第一次失败",[10,635,636],{},"开始执行删除并重建流程。先删掉旧 gwbridge，再用新网段重建：",[56,638,639],{"className":70,"code":343,"language":72,"meta":64,"style":64},[51,640,641,651,661,669,677,685],{"__ignoreMap":64},[76,642,643,645,647,649],{"class":78,"line":79},[76,644,350],{"class":82},[76,646,90],{"class":86},[76,648,355],{"class":86},[76,650,358],{"class":86},[76,652,653,655,657,659],{"class":78,"line":99},[76,654,350],{"class":82},[76,656,90],{"class":86},[76,658,367],{"class":86},[76,660,370],{"class":172},[76,662,663,665,667],{"class":78,"line":106},[76,664,375],{"class":172},[76,666,378],{"class":86},[76,668,370],{"class":172},[76,670,671,673,675],{"class":78,"line":121},[76,672,385],{"class":172},[76,674,388],{"class":86},[76,676,370],{"class":172},[76,678,679,681,683],{"class":78,"line":134},[76,680,395],{"class":172},[76,682,398],{"class":172},[76,684,370],{"class":172},[76,686,687],{"class":78,"line":145},[76,688,405],{"class":86},[10,690,691,692,695],{},"第二条命令报错：",[51,693,694],{},"\"Error response from daemon: Pool overlaps with other one on this address space\"","。",[10,697,698,699,702,703,706],{},"原因很讽刺：在 9:15 分的探索期间，改了 daemon.json 为 ",[51,700,701],{},"172.31.0.0\u002F16 size 22","，然后重启了 docker。启动时，docker daemon 会重建系统默认的 bridge 网络。这次它从那个 \u002F16 池子里切了一个 \u002F22 出来——恰好是 ",[51,704,705],{},"172.31.0.0\u002F22","。这块子网现在被 bridge 网络占用了。",[10,708,709,710,713],{},"我想给 gwbridge 分配的 ",[51,711,712],{},"172.31.0.0\u002F20"," 直接包含了这个 \u002F22，两个网络的地址空间产生了重叠，所以冲突。必须换一个不冲突的网段。",[10,715,716],{},"临时决策：换一个网段。探测几个候选 subnet 看哪个能创建：",[56,718,720],{"className":70,"code":719,"language":72,"meta":64,"style":64},"for SUBNET in 172.31.0.0\u002F22 10.30.0.0\u002F20 192.168.32.0\u002F20; do\n  docker network create --subnet \"$SUBNET\" _test 2>&1 | head -1\n  docker network rm _test 2>\u002Fdev\u002Fnull\ndone\n",[51,721,722,748,783,799],{"__ignoreMap":64},[76,723,724,727,730,733,736,739,742,745],{"class":78,"line":79},[76,725,726],{"class":269},"for",[76,728,729],{"class":102}," SUBNET ",[76,731,732],{"class":269},"in",[76,734,735],{"class":86}," 172.31.0.0\u002F22",[76,737,738],{"class":86}," 10.30.0.0\u002F20",[76,740,741],{"class":86}," 192.168.32.0\u002F20",[76,743,744],{"class":102},"; ",[76,746,747],{"class":269},"do\n",[76,749,750,753,755,757,760,763,766,769,772,775,777,780],{"class":78,"line":99},[76,751,752],{"class":82},"  docker",[76,754,90],{"class":86},[76,756,367],{"class":86},[76,758,759],{"class":172}," --subnet",[76,761,762],{"class":86}," \"",[76,764,765],{"class":102},"$SUBNET",[76,767,768],{"class":86},"\"",[76,770,771],{"class":86}," _test",[76,773,774],{"class":269}," 2>&1",[76,776,270],{"class":269},[76,778,779],{"class":82}," head",[76,781,782],{"class":172}," -1\n",[76,784,785,787,789,791,793,796],{"class":78,"line":106},[76,786,752],{"class":82},[76,788,90],{"class":86},[76,790,355],{"class":86},[76,792,771],{"class":86},[76,794,795],{"class":269}," 2>",[76,797,798],{"class":86},"\u002Fdev\u002Fnull\n",[76,800,801],{"class":78,"line":121},[76,802,803],{"class":269},"done\n",[10,805,806,809],{},[51,807,808],{},"10.30.0.0\u002F20"," 可用。改用这个。",[10,811,812],{},[44,813,814],{},"10:45 — 恢复并重建",[10,816,817,818,820],{},"删 gwbridge，用 ",[51,819,808],{}," 重建：",[56,822,824],{"className":70,"code":823,"language":72,"meta":64,"style":64},"docker network rm docker_gwbridge\ndocker network create \\\n  --driver bridge \\\n  --subnet 10.30.0.0\u002F20 \\\n  --gateway 10.30.0.1 \\\n  docker_gwbridge\n",[51,825,826,836,846,854,862,871],{"__ignoreMap":64},[76,827,828,830,832,834],{"class":78,"line":79},[76,829,350],{"class":82},[76,831,90],{"class":86},[76,833,355],{"class":86},[76,835,358],{"class":86},[76,837,838,840,842,844],{"class":78,"line":99},[76,839,350],{"class":82},[76,841,90],{"class":86},[76,843,367],{"class":86},[76,845,370],{"class":172},[76,847,848,850,852],{"class":78,"line":106},[76,849,375],{"class":172},[76,851,378],{"class":86},[76,853,370],{"class":172},[76,855,856,858,860],{"class":78,"line":121},[76,857,385],{"class":172},[76,859,738],{"class":86},[76,861,370],{"class":172},[76,863,864,866,869],{"class":78,"line":134},[76,865,395],{"class":172},[76,867,868],{"class":172}," 10.30.0.1",[76,870,370],{"class":172},[76,872,873],{"class":78,"line":145},[76,874,405],{"class":86},[10,876,877],{},"成功。",[10,879,880],{},"Swarm 立刻开始自动重新调度那 13 个被 scale 0 的 service。在接下来的 1-2 分钟内，所有 270 个 service 都拿回了 IP。gwbridge 当前用量从 0 涨回 267\u002F4094。",[10,882,883],{},[44,884,885],{},"10:55 — 孤儿清理",[10,887,888],{},"有 1 个 service 卡住了，一直 0\u002F1。",[10,890,891],{},"检查发现，旧的 docker-proxy 进程还在占着 host port 18183。container 已经不存在了，但进程没清。端口被占，新容器绑定不了。",[10,893,894],{},"手动 kill 这个 PID，释放端口。Swarm 重试，成功。",[10,896,897],{},[44,898,899],{},"11:00 — 数据库对账",[10,901,902],{},"DB 里有 13 条 Instance 记录状态仍是 PROVISIONING，实际容器已经在 RUNNING。写个快速脚本，把它们更新到 RUNNING。",[10,904,905],{},"同时删了 1 条孤儿 Instance 记录（某用户容器在异常重启时丢了）。",[10,907,908],{},[44,909,910],{},"11:05 — 第一次 ffmpeg 灰度，失败",[10,912,913],{},"现在网络扩容完毕，可以开始灰度 ffmpeg 了。",[10,915,916,917,695],{},"用 digest 形式指定镜像：",[51,918,919],{},"cpa\u002Fopenclaw-runtime:1.0.18@sha256:fcebbc...",[10,921,922],{},"想法是 digest 是 immutable 的，Swarm 可以直接用，不必重新 pull。",[10,924,925,926,929],{},"实际：Swarm 看到 ",[51,927,928],{},"@sha256:..."," 这种格式后，认为这是\"远程仓库 immutable 引用\"，强行去 docker.io pull。Pull 失败（网络不稳定），没有 fallback 到本地已有的 image（本地有 1.0.18 tag，但 digest 不同）。Task fatal error。Rollback。",[10,931,932],{},[44,933,934],{},"11:08 — 第二次灰度，成功",[10,936,937,938,695],{},"改用 tag-only 形式：",[51,939,940],{},"cpa\u002Fopenclaw-runtime:1.0.18",[10,942,943],{},"Swarm 走 pull 流程，失败后自动 fallback 本地同 tag 的 image。成功。",[10,945,946,947,950],{},"容器启动，",[51,948,949],{},"ffmpeg -version"," 输出 5.1.9。实际剪了一个 1 秒黑屏的 mp4 验证。",[10,952,953],{},[44,954,955],{},"11:18 — v4 全量发布",[10,957,958],{},"网络问题解决后，准备一键发布所有用户的容器升级。提交全部 270 个 service 的 update（1.0.17 → 1.0.18）。用 xargs 并发提交：",[56,960,962],{"className":70,"code":961,"language":72,"meta":64,"style":64},"cat service-list.txt | xargs -P 10 -I {} docker service update --image cpa\u002Fopenclaw-runtime:1.0.18 {}\n",[51,963,964],{"__ignoreMap":64},[76,965,966,969,972,974,977,980,983,985,987,989,991,994,997,1000],{"class":78,"line":79},[76,967,968],{"class":82},"cat",[76,970,971],{"class":86}," service-list.txt",[76,973,270],{"class":269},[76,975,976],{"class":82}," xargs",[76,978,979],{"class":172}," -P",[76,981,982],{"class":172}," 10",[76,984,563],{"class":172},[76,986,566],{"class":86},[76,988,87],{"class":86},[76,990,437],{"class":86},[76,992,993],{"class":86}," update",[76,995,996],{"class":172}," --image",[76,998,999],{"class":86}," cpa\u002Fopenclaw-runtime:1.0.18",[76,1001,1002],{"class":86}," {}\n",[10,1004,1005],{},"大约 1 秒完成 270 个 update 提交。",[10,1007,1008],{},"然后脚本进入 Phase 4（收敛判断）。检查所有 service 是否都运行了新镜像：",[56,1010,1012],{"className":70,"code":1011,"language":72,"meta":64,"style":64},"docker service ls --format \"{{.Image}}\"\n",[51,1013,1014],{"__ignoreMap":64},[76,1015,1016,1018,1020,1022,1024],{"class":78,"line":79},[76,1017,350],{"class":82},[76,1019,437],{"class":86},[76,1021,539],{"class":86},[76,1023,548],{"class":172},[76,1025,1026],{"class":86}," \"{{.Image}}\"\n",[10,1028,1029,1030,1033,1034,1036],{},"1 秒钟后看到全部输出都变成了 ",[51,1031,1032],{},"1.0.18","，脚本误判完成，把之前加的 ",[51,1035,591],{}," 黑洞过早还原。",[10,1038,1039,1040,1042],{},"问题来了：Swarm rolling update 是异步的。",[51,1041,53],{}," 看到的是 spec image（声明），不是 task 实际跑的 image（事实）。Spec 在你 update 那一秒就变了，但容器真正切换需要时间——它要先 stop 旧 task，再 pull（或 fallback）新镜像，再 start 新 task。这个过程通常需要数十秒。",[10,1044,1045],{},"结果，脚本还原了 hosts 黑洞，但大批容器正在切换过程中，需要 pull 镜像。此时网络开始响应慢。30 秒的 pull 超时开始积累，导致部分容器切换失败。",[10,1047,1048],{},[44,1049,1050],{},"11:21 — 紧急救援",[10,1052,1053],{},"检测到 hosts 还原过早，立刻加回黑洞：",[56,1055,1056],{"className":70,"code":598,"language":72,"meta":64,"style":64},[51,1057,1058],{"__ignoreMap":64},[76,1059,1060,1062,1064,1066],{"class":78,"line":79},[76,1061,605],{"class":172},[76,1063,608],{"class":86},[76,1065,611],{"class":269},[76,1067,614],{"class":86},[10,1069,1070,1071,1074],{},"等真实收敛。看 ",[51,1072,1073],{},"docker ps --filter \"name=svc-\" --format \"{{.Image}}\"","（task 层的实际镜像），不是 service ls 的 spec。",[10,1076,1077],{},[44,1078,1079],{},"11:25 — 真实收敛",[10,1081,1082],{},"270\u002F270 容器全部跑上 1.0.18。",[10,1084,1085],{},[44,1086,1087],{},"11:26 — 最终验证",[10,1089,1090],{},"抽 3 个用户容器实测 ffmpeg，panel\u002Fcanvas health check = 200，gwbridge 用量 270\u002F4094 (~6.6%)。",[10,1092,1093],{},"全部完成。",[14,1095,1097],{"id":1096},"根因三层问题叠加","根因：三层问题叠加",[10,1099,1100,1103],{},[44,1101,1102],{},"表面","：容器没装 ffmpeg。改 Dockerfile 就行。",[10,1105,1106,1109,1110,1112],{},[44,1107,1108],{},"深层 1 — 架构容量瓶颈","：系统设计是\"1 用户 = 1 容器\"，所以用户数上限 = 可用 IP 数。gwbridge 是 5 天前 ",[51,1111,317],{}," 时以 docker 默认值 \u002F24 创建的，253 个 IP。现在 251 个用户 + ingress 等系统组件已占满。",[10,1114,1115,1118,1119,1122,1123,1125,1126,1128],{},[44,1116,1117],{},"深层 2 — 配置陷阱","：某人改了 ",[51,1120,1121],{},"daemon.json"," 的 ",[51,1124,313],{},"，本想扩容。但这个配置只在 ",[51,1127,317],{}," 那一刻被读取一次。已有网络（gwbridge）不会因为修改配置而改变。这是个文档没突出强调的时机陷阱。",[10,1130,1131,1134],{},[44,1132,1133],{},"深层 3 — 无告警机制","：基础设施容量（IP 池、CPU、内存）应该主动巡检。这里 gwbridge 沉默地跑到 100% 满，用户侧只看到\"个别容器偶发起不来\"，很难追溯到网络容量。",[14,1136,1137],{"id":1137},"修复选择与执行",[10,1139,1140],{},"为什么选\"手动删除重建 gwbridge\"而不是其他：",[295,1142,1143,1149,1155,1161],{},[298,1144,1145,1148],{},[44,1146,1147],{},"改 daemon.json 已验证无效","：现有网络不会因配置变化而改变",[298,1150,1151,1154],{},[44,1152,1153],{},"重启 docker 风险太高","：所有 270 个容器停机 10+ 分钟",[298,1156,1157,1160],{},[44,1158,1159],{},"逐个迁移","：工作量巨大，容易出错",[298,1162,1163,1166],{},[44,1164,1165],{},"删除重建","：只涉及网络对象层面，不影响容器生命周期。Swarm 自动重调度，风险可控",[10,1168,1169],{},"验证阶梯：",[295,1171,1172,1178,1184],{},[298,1173,1174,1177],{},[44,1175,1176],{},"Staging dry-run","（105 秒完成）：确认 service 自动重调度、endpoint 重新分配、容器正常启动",[298,1179,1180,1183],{},[44,1181,1182],{},"单用户 canary","（发现\"释放 IP 被抢\"的问题）：改成先 scale 0 那 13 个 cycling service，再做扩容",[298,1185,1186,1189],{},[44,1187,1188],{},"全量执行","（5 分钟）：删 gwbridge，用 10.30.0.0\u002F20 重建，自动调度完成",[10,1191,1192],{},"结果：253 → 4094 IP（16 倍扩容），13 个失联用户恢复，270 个 service 全部正常。",[14,1194,1195],{"id":1195},"影响与防回归",[10,1197,1198,191],{},[44,1199,1200],{},"停机影响",[1202,1203,1204,1207,1210,1213],"ul",{},[298,1205,1206],{},"gwbridge 重建期间 5-10 分钟全集群不可用",[298,1208,1209],{},"灰度期间部分用户约 25 分钟间歇掉线",[298,1211,1212],{},"全量发布期间大批 task 切换，约 7 分钟影响",[298,1214,1215],{},"数据零丢失",[10,1217,1218,191],{},[44,1219,1220],{},"防回归措施",[10,1222,1223],{},"目前无已实施方案。Action Items 中有 P1 的\"监控告警：gwbridge IP 池 > 70%\"，尚未部署。",[14,1225,1226],{"id":1226},"教训",[295,1228,1229,1235,1245],{},[298,1230,1231,1234],{},[44,1232,1233],{},"基础设施容量要主动巡检，不要等爆表","。隐性容量上限通常以\"偶发故障\"的形式暴露，此时已接近 100%。应定期扫描 IP 池用量，设 70% 和 90% 的告警。",[298,1236,1237,695,1240,314,1242,1244],{},[44,1238,1239],{},"配置生效时机必须验证，不要假设",[51,1241,313],{},[51,1243,317],{}," 消费一次。类似的时机陷阱在分布式系统很常见（初始化 vs 运行时 vs 重启）。每项基础设施配置要写脚本验证一次：改配置 → 观察系统反映（docker info \u002F ps \u002F 日志），不要假设。",[298,1246,1247,1250],{},[44,1248,1249],{},"验证阶梯很重要","。Staging dry-run 测试可行性且暴露意外行为（这次的\"释放 IP 被抢\"）。单用户 canary 在生产真实复现，成本低。全量发布时信心最足。",[1252,1253,1254],"style",{},"html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}",{"title":64,"searchDepth":99,"depth":99,"links":1256},[1257,1258,1259,1260,1261,1262],{"id":16,"depth":99,"text":16},{"id":40,"depth":99,"text":40},{"id":1096,"depth":99,"text":1097},{"id":1137,"depth":99,"text":1137},{"id":1195,"depth":99,"text":1195},{"id":1226,"depth":99,"text":1226},"故障档案","2026-05-22","md",null,{},true,"\u002F2026-05-22-fa-012-gwbridge",{"title":5,"description":12},"FA-012","2026-05-22-FA-012-gwbridge容量上限","一行 apt 改动引出的 16 倍网络扩容：docker daemon.json 的配置陷阱如何偷偷限制了业务增长。",[1275,1276,1277,1278,1279],"Docker Swarm","网络容量","基础设施","隐性容量上限","配置陷阱","VVmDdWNROPXksGoBzLZgxfDwrGQwJ3EaEOK5I1GncQE",[1282,1690,2147],{"id":1283,"title":1284,"body":1285,"column":1263,"date":1675,"description":1289,"extension":1265,"hero_image":1266,"meta":1676,"navigation":1268,"path":1677,"seo":1678,"series_id":1679,"severity":1266,"stem":1680,"summary":1681,"tags":1682,"__hash__":1689},"posts\u002F2026-07-23-FA-018-共享存储的subPath陷阱.md","一个 PVC 装所有用户，当时看不出有什么问题",{"type":7,"value":1286,"toc":1666},[1287,1290,1293,1297,1300,1303,1310,1351,1354,1364,1368,1374,1377,1388,1391,1395,1398,1401,1404,1422,1425,1432,1435,1450,1454,1457,1460,1471,1482,1485,1488,1491,1559,1562,1569,1572,1575,1601,1604,1615,1618,1621,1660,1663],[10,1288,1289],{},"平台从 Docker Swarm 迁移到 Kubernetes，存储架构做了一个看似理想的调整：用共享 PVC + subPath 替代原来的\"每用户独占 PVC\"。方案优势显而易见——管理简单（1 个 PVC 对 574 个）、自动扩展、成本低。当时看不出有什么缺点。",[10,1291,1292],{},"但实施中遇到了四个隐蔽的问题。",[14,1294,1296],{"id":1295},"权限错配bind-型数据的属主陷阱","权限错配：bind 型数据的属主陷阱",[10,1298,1299],{},"旧系统中，用户数据通过两种方式持久化：bind mount 和 Docker volume。迁移时需要从这两个来源完整导出数据。",[10,1301,1302],{},"对于 bind mount 型数据，文件属主通常是 10001（或其他固定 UID）。当我登上旧集群的宿主机，以普通用户身份尝试读取这些文件时，只能看到 13 个文件。权限拒绝。同一目录里其实有 4778 个文件，但大多数因为属主权限不匹配而不可见。",[10,1304,1305,1306,1309],{},"只有用 sudo 才能完整读到。所以迁移脚本必须用 ",[51,1307,1308],{},"sudo tar"," 来打包：",[56,1311,1313],{"className":70,"code":1312,"language":72,"meta":64,"style":64},"sudo tar -czf \u002Ftmp\u002Fmig\u002F${uid}\u002Fdata.tgz \\\n  -C \u002Fmnt\u002Fyun-claw\u002Fusers\u002F${uid} .\n",[51,1314,1315,1337],{"__ignoreMap":64},[76,1316,1317,1320,1323,1326,1329,1332,1335],{"class":78,"line":79},[76,1318,1319],{"class":82},"sudo",[76,1321,1322],{"class":86}," tar",[76,1324,1325],{"class":172}," -czf",[76,1327,1328],{"class":86}," \u002Ftmp\u002Fmig\u002F",[76,1330,1331],{"class":102},"${uid}",[76,1333,1334],{"class":86},"\u002Fdata.tgz",[76,1336,370],{"class":172},[76,1338,1339,1342,1345,1348],{"class":78,"line":99},[76,1340,1341],{"class":172},"  -C",[76,1343,1344],{"class":86}," \u002Fmnt\u002Fyun-claw\u002Fusers\u002F",[76,1346,1347],{"class":102},"${uid} ",[76,1349,1350],{"class":86},".\n",[10,1352,1353],{},"这不是脚本设计的问题，而是 Kubernetes 环境下容器权限与宿主权限映射的一个陷阱。容器内运行的网关进程可能以容器用户身份运行，但宿主上的文件属主是不同的 UID。当两个权限体系接不上，访问就会失败。",[10,1355,1356,1357,1360,1361,1363],{},"新方案中，用户数据通过 subPath 挂到容器的 ",[51,1358,1359],{},"\u002Fdata"," 目录。如果 subPath 指向的目录是由 Kubernetes 在宿主机上创建的，属主可能是 root 或其他用户，容器内进程（以容器指定的 UID 运行）仍然会遭遇权限问题。这个风险需要在 Dockerfile 和容器启动时明确处理：容器内应该预先创建 ",[51,1362,1359],{}," 并设置正确的属主，或者使用 securityContext 的 fsGroup 或 runAsUser 来强制权限。",[14,1365,1367],{"id":1366},"subpath-目录创建的时机与属主处理","subPath 目录创建的时机与属主处理",[10,1369,1370,1371],{},"Kubernetes 对 subPath 的处理有个微妙之处：如果挂载的 subPath 目录在宿主机上不存在，kubelet 会自动创建它。但 ",[76,1372,1373],{},"待补：这个自动创建的目录的属主是什么？是 root 还是其他用户？容器的 securityContext 能否保证挂载后的权限符合预期？",[10,1375,1376],{},"设计文档中没有明确说明这一点。在实施前，需要验证：",[1202,1378,1379,1382,1385],{},[298,1380,1381],{},"首次 subPath 目录不存在时，kubelet 创建它并将其属主设置为谁",[298,1383,1384],{},"容器的 securityContext（fsGroup、runAsUser）是否能在挂载时生效",[298,1386,1387],{},"是否应该预先在共享 PVC 上创建所有 subPath 目录并设置好属主，而不是依赖 Kubernetes 的隐式创建",[10,1389,1390],{},"没有明确的答案，就容易踩坑。一个保险做法是在 destroy 用户账户时清空 subPath 目录，同时在 provision 时让一个 init-container 验证或修复目录属主，确保容器进程有写权限。",[14,1392,1394],{"id":1393},"单-pvc-中的节点级故障域","单 PVC 中的节点级故障域",[10,1396,1397],{},"这是最严重的问题，也是与 FA-015（节点假 Ready）的关联点。",[10,1399,1400],{},"当一个节点的 NFS 客户端或到 SFS 的网络链路发生 hang 时，所有依赖共享 PVC 的 Pod 都会受影响。这不是共享 PVC 本身的问题，而是故障隔离的问题。",[10,1402,1403],{},"具体现象：",[1202,1405,1406,1413,1416,1419],{},[298,1407,1408,1409,1412],{},"如果 NFS 挂载使用了硬 mount（",[51,1410,1411],{},"hard"," 参数是 NFS 的默认值），而网络故障或存储服务中断，NFS 客户端会无限重试",[298,1414,1415],{},"重试期间，所有试图访问 NFS 的进程都会被阻塞在 I\u002FO 上（D 状态，不可中断）",[298,1417,1418],{},"如果 kubelet 或 containerd 的某个操作（如创建容器的卷挂载步骤）陷入 I\u002FO 阻塞，整个节点就会冻死",[298,1420,1421],{},"该节点上的所有实例（无论是否在访问存储）都会因为无法创建或删除 Pod 而故障",[10,1423,1424],{},"这与独立 PVC 的效果截然不同。旧架构中，每个用户有独立 PVC，意味着如果某个 PVC 对应的存储有问题，只会影响这一个用户。其他用户的 PVC 可能分散在不同的存储卷甚至不同的节点上，相对独立。",[10,1426,1427,1428,1431],{},"新架构下，单个共享 PVC 承载所有用户，一旦这个 PVC 对应的网络链路或挂载出问题，",[44,1429,1430],{},"同节点上所有实例都会被殃及","。如果该节点碰巧堆积了 50 个实例，一次节点级的存储 hang 就会导致 50 个实例集体故障，且外表看起来是\"节点 Ready，但实例卡住\"——kubelet 的心跳正常，Pod 状态却卡在 ContainerCreating 或 Terminating。",[10,1433,1434],{},"这正是 FA-015 在 2026-06-05 观察到的现象：节点假 Ready，但大量实例网关无响应。防护措施包括：",[1202,1436,1437,1444,1447],{},[298,1438,1439,1440,1443],{},"NFS 挂载参数改为软 mount（",[51,1441,1442],{},"soft,timeo=100,retrans=3","），让 I\u002FO 超时而不是无限等待",[298,1445,1446],{},"节点加健康检测，检查 ContainerCreating 堆积数和网关探测失败率，及早发现节点冻死",[298,1448,1449],{},"实例分散部署，用 topologySpreadConstraints 避免单节点堆积",[14,1451,1453],{"id":1452},"无-per-subpath-硬配额","无 per-subPath 硬配额",[10,1455,1456],{},"共享存储的架构决定了无法在存储层面按 subPath 限制用户配额。SFS（及 NFS 一般）没有 per-directory 的硬配额功能，只能在整个卷级别限制。",[10,1458,1459],{},"这意味着：",[1202,1461,1462,1465,1468],{},[298,1463,1464],{},"无法阻止某个用户的数据膨胀而挤占其他用户的空间",[298,1466,1467],{},"无法在存储层面实现\"超额用户的写入失败\"",[298,1469,1470],{},"只能在应用层进行监控、告警和逻辑控制",[10,1472,1473,1474,1477,1478,1481],{},"新系统的方案是应用层监控：定期 ",[51,1475,1476],{},"du -sb \u002Fdata"," 扫描每个用户的目录大小，落库到 Instance 表，",[51,1479,1480],{},"\u002Fstatus"," 接口返回超额标志位。前端据此提示用户\"存储已满\"。这是纯监控，不阻断。如果用户无视提示继续写入，直到共享卷真的满了，才会出现\"所有用户集体写入失败\"的惨淡局面。",[10,1483,1484],{},"这个限制是方案选择的代价。",[14,1486,1487],{"id":1487},"为什么仍然选了这个方案",[10,1489,1490],{},"尽管有这些问题，平台仍然采用了共享 PVC + subPath 的方案。原因是对比了替代方案：",[1492,1493,1494,1513],"table",{},[1495,1496,1497],"thead",{},[1498,1499,1500,1504,1507,1510],"tr",{},[1501,1502,1503],"th",{},"方案",[1501,1505,1506],{},"优点",[1501,1508,1509],{},"缺点",[1501,1511,1512],{},"故障隔离",[1514,1515,1516,1531,1545],"tbody",{},[1498,1517,1518,1522,1525,1528],{},[1519,1520,1521],"td",{},"独立 PVC（旧架构）",[1519,1523,1524],{},"天然的每用户隔离；故障域清晰",[1519,1526,1527],{},"管理复杂（574 个 PVC）；扩展性差；成本高",[1519,1529,1530],{},"✓ 最好",[1498,1532,1533,1536,1539,1542],{},[1519,1534,1535],{},"共享 PVC + subPath（新架构）",[1519,1537,1538],{},"管理简单（1 个 PVC）；自动扩展；成本低",[1519,1540,1541],{},"故障隔离性差；无硬配额；权限管理复杂",[1519,1543,1544],{},"✗ 较差",[1498,1546,1547,1550,1553,1556],{},[1519,1548,1549],{},"对象存储（S3\u002FOSS）",[1519,1551,1552],{},"真正的多租户隔离；天然分布",[1519,1554,1555],{},"延迟高；成本更高；应用改造大",[1519,1557,1558],{},"✓ 最好，但代价大",[10,1560,1561],{},"独立 PVC 的管理开销是关键问题。574 个用户意味着 574 个 PVC 对象、574 条 PV 绑定、574 个存储卷。每次实例创建、删除或迁移都要涉及 PVC 的生命周期管理。扩展到 5000 用户时，这种开销会成为瓶颈。对象存储虽然隔离性最好，但需要应用层改造（兼容 S3 API、处理延迟、调整备份策略），而且成本更高。",[10,1563,1564,1565,1568],{},"共享 PVC 方案的核心优势是",[44,1566,1567],{},"运维简洁","：增删用户只需改 subPath（一行 YAML），不涉及存储层操作。代价是故障隔离性从\"用户级\"降到\"节点级\"。",[14,1570,1571],{"id":1571},"适用边界",[10,1573,1574],{},"这个方案适合以下场景：",[1202,1576,1577,1583,1589,1595],{},[298,1578,1579,1582],{},[44,1580,1581],{},"用户数量有上限","（几百到几千）。用户过多时，共享卷的单点压力会成问题。",[298,1584,1585,1588],{},[44,1586,1587],{},"用户数据量可控","（单用户通常 GB 级）。如果单用户数据量达 TB，一次 du 扫描会拖累整体。",[298,1590,1591,1594],{},[44,1592,1593],{},"可以接受节点级故障隔离","。只要网络和存储配置足够稳定（硬化 NFS 参数、冗余链路），节点 hang 的概率不会很高。",[298,1596,1597,1600],{},[44,1598,1599],{},"能够实施应用层监控和告警","。无硬配额，就必须有实时监控。",[10,1602,1603],{},"不适合的场景包括：",[1202,1605,1606,1609,1612],{},[298,1607,1608],{},"超大规模用户（万级以上）",[298,1610,1611],{},"用户数据特别不均衡（少数用户占大头）的场景",[298,1613,1614],{},"对故障隔离要求极高的系统（比如金融交易）",[14,1616,1617],{"id":1617},"防护与调整",[10,1619,1620],{},"基于这四个问题，实施中做了以下调整：",[295,1622,1623,1632,1642,1648,1654],{},[298,1624,1625,1628,1629,1631],{},[44,1626,1627],{},"权限处理","：容器 Dockerfile 中预先创建 ",[51,1630,1359],{}," 并设置正确的属主；启动脚本检查并修复权限。",[298,1633,1634,1637,1638,1641],{},[44,1635,1636],{},"NFS 参数硬化","：StorageClass 的挂载选项改为 ",[51,1639,1640],{},"soft,timeo=100,retrans=3,intr","，避免硬 mount 导致节点冻。",[298,1643,1644,1647],{},[44,1645,1646],{},"节点健康检测","：cron 每 60 秒检查各节点的 ContainerCreating 堆积和网关探测失败率，及早发现故障。",[298,1649,1650,1653],{},[44,1651,1652],{},"实例分散","：StatefulSet 加 topologySpreadConstraints，避免单节点堆积 50+ 个实例。",[298,1655,1656,1659],{},[44,1657,1658],{},"应用层配额","：\u002Fstatus 接口返回 diskUsedMi 和 overQuota 标志位，前端告警用户。",[10,1661,1662],{},"这些措施不能消除风险，但可以大幅降低故障概率和影响范围。",[1252,1664,1665],{},"html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":64,"searchDepth":99,"depth":99,"links":1667},[1668,1669,1670,1671,1672,1673,1674],{"id":1295,"depth":99,"text":1296},{"id":1366,"depth":99,"text":1367},{"id":1393,"depth":99,"text":1394},{"id":1452,"depth":99,"text":1453},{"id":1487,"depth":99,"text":1487},{"id":1571,"depth":99,"text":1571},{"id":1617,"depth":99,"text":1617},"2026-07-23",{},"\u002F2026-07-23-fa-018-subpath",{"title":1284,"description":1289},"FA-018","2026-07-23-FA-018-共享存储的subPath陷阱","K8s 共享 PVC + subPath 方案在实施中暴露的四个陷阱：权限不匹配、目录创建时机、节点级故障域、无 per-subPath 配额。",[1683,1684,1685,1686,1687,1688,1512],"Kubernetes","存储","subPath","PVC","NFS","权限","4pJJyzbII4OOc9jJS74HUpUfyVc9CfxdE5ovso8ylSE",{"id":1691,"title":1692,"body":1693,"column":1263,"date":2131,"description":2132,"extension":1265,"hero_image":1266,"meta":2133,"navigation":1268,"path":2134,"seo":2135,"series_id":2136,"severity":1266,"stem":2137,"summary":2138,"tags":2139,"__hash__":2146},"posts\u002F2026-07-11-FA-017-长任务异步化.md","三个超时机制，一起把长任务掐死了",{"type":7,"value":1694,"toc":2123},[1695,1705,1708,1711,1717,1723,1729,1736,1742,1746,1749,1752,1758,1761,1772,1778,1782,1785,1792,1798,1801,1816,1820,1823,1834,1883,1898,1905,1909,1912,1915,1933,1936,1944,1947,1962,1965,1968,2114,2117,2120],[10,1696,1697,1698,1701,1702,695],{},"数字人口播的「分析」功能从视频提取口播文稿、分镜脚本、结构卖点，需要下载视频、压缩、调 Gemini 视觉 API，耗时从几十秒到几分钟。最初同步实现：前端调 ",[51,1699,1700],{},"POST \u002Fapi\u002Fworkflow\u002Fdub\u002Fanalyze-parsed","，后端直接跑完再返回。这个方案暴露的问题不是单点，而是",[44,1703,1704],{},"三个独立的缺陷叠加",[10,1706,1707],{},"网关超时、CDN 空闲切断、客户端重试——这三个没有一个是\"bug\"，每个都是合理的系统设计。但组合在一起就是连锁灾难：客户端看不到结果（超时或连接断），认为失败了重试，后端却已经扣过费了；或者分析其实已跑完，但因为响应发不出去，重试又跑了一遍。退款时无法幂等判断，某些用户最终被多扣了好几倍。",[10,1709,1710],{},"具体来说，三个问题各自是什么：",[10,1712,1713,1716],{},[44,1714,1715],{},"第一个问题是网关超时","。云平台的入口网关（API Gateway）有一个固定的读超时，通常设为几十秒。当后端分析耗时超过这个阈值，网关主动断连接，客户端收到 504 或连接重置，但后端的分析进程并不知道客户端已经走了，继续跑完整个流程。如果分析结果的退款逻辑放在 try-catch 的 finally 里，此时 HTTP 响应已发不出去，客户就看到\"分析失败\"，但款已经扣了。",[10,1718,1719,1722],{},[44,1720,1721],{},"第二个问题是 CDN 空闲切断","。不少 CDN 和负载均衡层有这样的策略：如果一条 HTTP 连接在某个时间段内没有数据往返，就认为它已死而主动关闭。这与 FA-013 那次踩到的问题一样——长连接因为没有心跳数据而被 CDN 中间件切断，导致请求被突然中断。这个机制在分析请求上也会造成麻烦：如果分析用时较长，连接就可能被切，响应发不出去。",[10,1724,1725,1728],{},[44,1726,1727],{},"第三个问题是客户端重试导致重复扣费","。当客户端没有收到响应（网关超时或 CDN 切断），一般会重试这个请求。问题在于这次重试是一条完全独立的请求，后端会把它当作新任务处理，再次扣费。而且如果第一次的分析实际上已经跑完了，就会出现\"分析跑了两遍、扣费扣了两遍、用户只看到了一个结果\"的情况。如果第一次分析中途被 CDN 切了，第二次重试重新开始，那扣费可能是三倍。而退款的时候，因为业务流程上没有幂等设计，很难追溯哪个 operationId 应该被退，哪个不应该。",[10,1730,1731,1732,1735],{},"这三个问题是",[44,1733,1734],{},"叠加的","，不是独立的。它们共同指向一个根本的架构问题。",[10,1737,1738,1739,695],{},"根本原因一个：",[44,1740,1741],{},"不该让耗时任务挂在 HTTP 连接上",[14,1743,1745],{"id":1744},"异步任务是必需的不是可选的","异步任务是必需的，不是可选的",[10,1747,1748],{},"解决方案很简单：把分析改成异步任务模式。",[10,1750,1751],{},"前端提交分析请求直接返回任务 ID（HTTP 耗时极短：校验参数、建数据库记录、预扣费）。后端后台 worker 跑分析重逻辑，完成后持久化结果。前端轮询查询状态，得到结果后停止。",[56,1753,1756],{"className":1754,"code":1755,"language":61},[59],"客户端              后端\n  |                 |\n  +--提交---------->|\n  |             创建任务，返回 ID\n  |\u003C--返回 ID-----+\n  |                 |（后台运行分析）\n  |                 |\n  +--轮询--------->|\n  |             查询状态\n  |\u003C--返回状态---+\n  |                 |\n  |（重复轮询）\n  +--轮询--------->|\n  |             查询状态\n  |\u003C--返回结果---+\n  |                 |\n",[51,1757,1755],{"__ignoreMap":64},[10,1759,1760],{},"好处显而易见：",[1202,1762,1763,1766,1769],{},[298,1764,1765],{},"单次 HTTP 请求的耗时从分钟级降到秒级，不会触发网关或 CDN 的超时。",[298,1767,1768],{},"后端分析的运行生命周期与 HTTP 连接完全解耦。即使连接在中途被切，已经启动的分析会继续在后台跑，结果最终还是会被保存。",[298,1770,1771],{},"客户端重试不会产生新的分析任务。只要实现幂等判重，同一个请求在短时间内重复提交也只会产生一个任务。",[10,1773,1774,1775,695],{},"但异步化不是万灵药，它把问题转移到了三个新的维度上：",[44,1776,1777],{},"幂等、持久化、失败语义",[14,1779,1781],{"id":1780},"幂等同一请求只产生一个任务","幂等：同一请求只产生一个任务",[10,1783,1784],{},"假设客户端因为网络抖动，在短时间内连续发了两次\"开始分析\"请求，后端收到两条独立的 HTTP 请求。如果没有幂等设计，就会创建两个任务、扣两次费。",[10,1786,1787,1788,1791],{},"幂等的实现最直接的办法是在",[44,1789,1790],{},"提交阶段就判重","：为这个用户、这个资源设置一个\"只有一个进行中的分析任务\"的约束。素材中的设计是这样做的：",[56,1793,1796],{"className":1794,"code":1795,"language":61},[59],"if 该用户已有 status=running && kind=analyze_parsed 的任务:\n    return 409 Conflict（分析已在进行中）\n",[51,1797,1795],{"__ignoreMap":64},[10,1799,1800],{},"这样双击提交同一个视频，第一次返回 201 + taskId，第二次返回 409，客户端看到 409 就知道不要再建新任务，拿之前返回的 ID 去轮询。",[10,1802,1803,1804,1807,1808,1811,1812,1815],{},"另一个幂等层次是",[44,1805,1806],{},"计费操作的幂等","。预扣费的时候用一个全局唯一的 ",[51,1809,1810],{},"operationId","（比如 ",[51,1813,1814],{},"dub-analyze:{uuid}","），这个 ID 关联了这次预扣。如果扣费 API 被重复调用，因为 operationId 重复，系统只会扣一次。退款时也用同一个 operationId，保证退款与预扣能正确配对。",[14,1817,1819],{"id":1818},"持久化进程重启后任务不丢","持久化：进程重启后任务不丢",[10,1821,1822],{},"后台分析 worker 可能因为发版、机器重启、容器被杀等原因在中途退出。如果任务的状态只保存在内存里，进程一死任务就丢了。",[10,1824,1825,1826,1829,1830,1833],{},"持久化的关键是",[44,1827,1828],{},"任务状态表","。素材中复用了既有的 ",[51,1831,1832],{},"SkyhumanTask"," 表，包含字段：",[1202,1835,1836,1842,1848,1853,1859,1865,1871,1877],{},[298,1837,1838,1841],{},[51,1839,1840],{},"id",": 任务 ID",[298,1843,1844,1847],{},[51,1845,1846],{},"status",": 进度（running \u002F completed \u002F failed）",[298,1849,1850,1852],{},[51,1851,1810],{},": 幂等 ID",[298,1854,1855,1858],{},[51,1856,1857],{},"chargedPoints",": 预扣费用",[298,1860,1861,1864],{},[51,1862,1863],{},"resultPayload",": 分析结果的 JSON",[298,1866,1867,1870],{},[51,1868,1869],{},"error",": 失败原因",[298,1872,1873,1876],{},[51,1874,1875],{},"updatedAt",": 最后更新时间戳",[298,1878,1879,1882],{},[51,1880,1881],{},"sourceObjectKey",": 源视频对象路径（用于幂等判重和重试）",[10,1884,1885,1886,1889,1890,1893,1894,1897],{},"任务创建时插一条 ",[51,1887,1888],{},"status=running"," 的记录。后台 worker 定期通过 heartbeat（每 30 秒做一次空 update，仅触碰 updatedAt）来证明自己还活着。分析完成时更新为 ",[51,1891,1892],{},"status=completed, resultPayload=...","；失败时更新为 ",[51,1895,1896],{},"status=failed, error=..."," 并触发退款。",[10,1899,1900,1901,1904],{},"这样即使 worker 进程突然死了，任务记录还在数据库里。新启起的 worker 可以扫描表里的 ",[51,1902,1903],{},"status=running && updatedAt 过期"," 的任务，判断它们已经没有 worker 在跑了，执行清理逻辑。",[14,1906,1908],{"id":1907},"失败语义区分任务失败和查询失败","失败语义：区分\"任务失败\"和\"查询失败\"",[10,1910,1911],{},"异步设计引入了一个新的错误维度：客户端查询任务状态时得到的 404，可能是\"这个任务 ID 不存在\"（用户输错了），也可能是\"任务 ID 之前存在但已被清理\"（进程清理了过期数据）。这两种情况的含义很不一样。",[10,1913,1914],{},"更重要的是，前端轮询收到的错误需要区分是否应该重试：",[1202,1916,1917,1927],{},[298,1918,1919,1922,1923,1926],{},[44,1920,1921],{},"任务本身失败","（",[51,1924,1925],{},"status=failed, error=\"Gemini API 返回错误\"","）：这种情况不该重试，因为重试也会失败。应该直接向用户展示失败信息，并根据 operationId 触发退款。",[298,1928,1929,1932],{},[44,1930,1931],{},"查询接口失败","（500、网络超时）：这种情况应该重试查询，因为任务本身可能还在继续跑。",[10,1934,1935],{},"素材中的设计把这两类区分得很清楚：",[1202,1937,1938,1941],{},[298,1939,1940],{},"任务最终的状态（completed 或 failed）持久化到数据库，客户端查询会拿到明确的业务语义。",[298,1942,1943],{},"查询接口的 HTTP 错误（5xx）是技术层故障，客户端应该重试。",[10,1945,1946],{},"同时，后端的 reaper 机制（定期扫表清理超期任务）也遵循这个语义：",[1202,1948,1949,1955],{},[298,1950,1951,1952,1954],{},"如果一个 ",[51,1953,1888],{}," 的任务超过 90 秒没有 heartbeat 更新，说明 worker 已死，reaper 会强制置为 failed 并退款。",[298,1956,1957,1958,1961],{},"但这个强制置为 failed 不应该抛错，因为此时可能有其他进程也想更新这个任务。更新语句应该带条件：",[51,1959,1960],{},"UPDATE SkyhumanTask SET status=failed WHERE id=? AND status=running","，这样如果 reaper 和其他 worker 同时操作，只有一方会成功。",[14,1963,1964],{"id":1964},"具体实现模式",[10,1966,1967],{},"数字人口播的分析异步化采用了这样的模式：",[295,1969,1970,2015,2061,2084],{},[298,1971,1972,1975,1976],{},[44,1973,1974],{},"提交阶段","（同步）：",[1202,1977,1978,1981,1984,1990,1996,2003,2009],{},[298,1979,1980],{},"校验用户和资源",[298,1982,1983],{},"判重：是否已有进行中的任务（409）",[298,1985,1986,1989],{},[51,1987,1988],{},"getObject"," 拉视频，检测时长（若≤0 返 400 不扣费）",[298,1991,1992,1995],{},[51,1993,1994],{},"chargeResource"," 预扣费用，获得 operationId",[298,1997,1998,1999,2002],{},"创建 ",[51,2000,2001],{},"SkyhumanTask{ status:running, operationId, sourceObjectKey }"," 记录",[298,2004,2005,2008],{},[51,2006,2007],{},"scheduleTask"," 丢到后台队列",[298,2010,2011,2012],{},"返回 ",[51,2013,2014],{},"{ taskId }",[298,2016,2017,191,2020],{},[44,2018,2019],{},"后台运行阶段",[1202,2021,2022,2025,2028,2035,2041,2051,2058],{},[298,2023,2024],{},"Worker 取出任务记录",[298,2026,2027],{},"开启 heartbeat 定时器（每 30s update 一次 updatedAt）",[298,2029,2030,2031,2034],{},"执行 ",[51,2032,2033],{},"analyzeDubVisionOnly","（下载→压缩→调用 Gemini→解析）",[298,2036,2037,2038],{},"成功：",[51,2039,2040],{},"update( status:completed, resultPayload )",[298,2042,2043,2044,2047,2048],{},"失败：",[51,2045,2046],{},"update( status:failed, error )"," + ",[51,2049,2050],{},"refundResource(operationId)",[298,2052,2053,2054,2057],{},"所有 update 都带条件 ",[51,2055,2056],{},"WHERE id=? AND status=running","（防被 reaper 覆盖）",[298,2059,2060],{},"Finally 清理 heartbeat 定时器",[298,2062,2063,2066,2067],{},[44,2064,2065],{},"查询阶段","（前端轮询）：",[1202,2068,2069,2078,2081],{},[298,2070,2071,2074,2075],{},[51,2072,2073],{},"GET \u002Fapi\u002Fworkflow\u002Fdub\u002Ftasks\u002F:id"," 返回 ",[51,2076,2077],{},"{ status, resultPayload, error }",[298,2079,2080],{},"若 status=completed 或 failed，停止轮询",[298,2082,2083],{},"若查询接口返回 5xx，则后续重试",[298,2085,2086,2089,2090],{},[44,2087,2088],{},"清理阶段","（reaper）：",[1202,2091,2092,2095,2102,2105,2111],{},[298,2093,2094],{},"后台定期扫表",[298,2096,2097,2098,2101],{},"对于 ",[51,2099,2100],{},"status=running && updatedAt > 90s"," 的任务，判定 worker 已死",[298,2103,2104],{},"若 providerTaskId（这里没有）存在，调用上游的 finalize API",[298,2106,2107,2108,2110],{},"若不存在，直接 ",[51,2109,2050],{}," + 置 failed",[298,2112,2113],{},"本设计无 providerTaskId，所以 reaper 自动走退款分支，无需改动 reaper 代码",[10,2115,2116],{},"这个模式的关键是三道防线：提交时判重（409）、后台心跳（定期 touch）、reaper 兜底（自动退款）。任何一个环节出问题，都有后续环节来补救。",[14,2118,2119],{"id":2119},"防线缺一不可",[10,2121,2122],{},"长耗时任务不能挂在 HTTP 连接上不只是超时问题，而是连接脆弱性与重复处理的共谋。异步化表面是\"返回 ID、后台跑\"这一步，真正的难点在三个维度的同时保证：幂等（不重复扣费）、持久化（不丢数据）、失败语义（不破坏重试逻辑）。缺一就会在某个场景暴露。",{"title":64,"searchDepth":99,"depth":99,"links":2124},[2125,2126,2127,2128,2129,2130],{"id":1744,"depth":99,"text":1745},{"id":1780,"depth":99,"text":1781},{"id":1818,"depth":99,"text":1819},{"id":1907,"depth":99,"text":1908},{"id":1964,"depth":99,"text":1964},{"id":2119,"depth":99,"text":2119},"2026-07-11","数字人口播的「分析」功能从视频提取口播文稿、分镜脚本、结构卖点，需要下载视频、压缩、调 Gemini 视觉 API，耗时从几十秒到几分钟。最初同步实现：前端调 POST \u002Fapi\u002Fworkflow\u002Fdub\u002Fanalyze-parsed，后端直接跑完再返回。这个方案暴露的问题不是单点，而是三个独立的缺陷叠加。",{},"\u002F2026-07-11-fa-017",{"title":1692,"description":2132},"FA-017","2026-07-11-FA-017-长任务异步化","长耗时分析请求（几十秒到分钟级）用同步 HTTP 导致网关超时、CDN切断、客户端重试重复扣费。改异步任务需同时解决幂等、持久化、失败语义。",[2140,2141,2142,2143,2144,2145],"异步任务","HTTP超时","CDN","幂等","任务队列","微服务","ARFDvBMGX3tw-E3Rj_gw1RYEjrykplO6u1dxtdS7Uto",{"id":2148,"title":2149,"body":2150,"column":1263,"date":3066,"description":3067,"extension":1265,"hero_image":1266,"meta":3068,"navigation":1268,"path":3069,"seo":3070,"series_id":3071,"severity":1266,"stem":3072,"summary":3073,"tags":3074,"__hash__":3080},"posts\u002F2026-06-17-FA-016-WS永久断开三条路径.md","日志里没有重连记录——它根本没在重连",{"type":7,"value":2151,"toc":3054},[2152,2163,2170,2177,2180,2183,2188,2195,2270,2276,2345,2356,2360,2363,2487,2498,2502,2508,2563,2566,2569,2572,2587,2590,2593,2596,2603,2861,2871,2874,2884,2897,2907,2910,2913,2920,2971,2974,2977,2980,2983,2986,3031,3048,3051],[10,2153,2154,2155,2158,2159,2162],{},"从 2026-05-11 开始，多台便携包客户反馈 WebSocket 断线后面板卡死。网关进程（",[51,2156,2157],{},"openclaw_gateway.exe","）明确在运行，",[51,2160,2161],{},"\u002Fhealth"," 返回 200，但面板显示\"已停止重连，请手动刷新\"——用户必须手动重启才能恢复。",[10,2164,2165,2166,2169],{},"关键线索在控制台：看不到任何 ",[51,2167,2168],{},"[ws] 计划重连"," 的日志。",[10,2171,2172,2173,2176],{},"通常断线后客户端会频繁尝试重连，日志里应该是满屏的重连记录。没有日志意味着问题不在重连失败，而在",[44,2174,2175],{},"压根没启动重连机制","。说明客户端进入了某个永久休眠状态。",[14,2178,2179],{"id":2179},"三条独立的永久断开路径",[10,2181,2182],{},"代码审查发现了三条完全不同的、独立的永久断开路径。任意一条命中，就会停止调度任何重连计时器。",[2184,2185,2187],"h3",{"id":2186},"路径-a凭据刷新异常","路径 A：凭据刷新异常",[10,2189,2190,2191,2194],{},"在 ",[51,2192,2193],{},"_scheduleReconnect"," 方法末尾，每次普通重连都会调用：",[56,2196,2200],{"className":2197,"code":2198,"language":2199,"meta":64,"style":64},"language-javascript shiki shiki-themes github-light github-dark","this._reconnectTimer = setTimeout(() => {\n  if (!this._intentionalClose) {\n    this._refreshCredentialsAndReconnect(0)\n  }\n}, delay)\n","javascript",[51,2201,2202,2224,2240,2260,2265],{"__ignoreMap":64},[76,2203,2204,2207,2210,2212,2215,2218,2221],{"class":78,"line":79},[76,2205,2206],{"class":172},"this",[76,2208,2209],{"class":102},"._reconnectTimer ",[76,2211,458],{"class":269},[76,2213,2214],{"class":82}," setTimeout",[76,2216,2217],{"class":102},"(() ",[76,2219,2220],{"class":269},"=>",[76,2222,2223],{"class":102}," {\n",[76,2225,2226,2229,2232,2235,2237],{"class":78,"line":99},[76,2227,2228],{"class":269},"  if",[76,2230,2231],{"class":102}," (",[76,2233,2234],{"class":269},"!",[76,2236,2206],{"class":172},[76,2238,2239],{"class":102},"._intentionalClose) {\n",[76,2241,2242,2245,2248,2251,2254,2257],{"class":78,"line":106},[76,2243,2244],{"class":172},"    this",[76,2246,2247],{"class":102},".",[76,2249,2250],{"class":82},"_refreshCredentialsAndReconnect",[76,2252,2253],{"class":102},"(",[76,2255,2256],{"class":172},"0",[76,2258,2259],{"class":102},")\n",[76,2261,2262],{"class":78,"line":121},[76,2263,2264],{"class":102},"  }\n",[76,2266,2267],{"class":78,"line":134},[76,2268,2269],{"class":102},"}, delay)\n",[10,2271,2272,2273,2275],{},"而 ",[51,2274,2250],{}," 是异步方法，catch 分支没有任何后续处理：",[56,2277,2279],{"className":2197,"code":2278,"language":2199,"meta":64,"style":64},"} catch (e) {\n  console.error('[ws] 刷新凭据失败:', e)\n  this._setConnected(false, 'error', `凭据刷新失败: ${e}`)\n}\n",[51,2280,2281,2292,2307,2341],{"__ignoreMap":64},[76,2282,2283,2286,2289],{"class":78,"line":79},[76,2284,2285],{"class":102},"} ",[76,2287,2288],{"class":269},"catch",[76,2290,2291],{"class":102}," (e) {\n",[76,2293,2294,2297,2299,2301,2304],{"class":78,"line":99},[76,2295,2296],{"class":102},"  console.",[76,2298,1869],{"class":82},[76,2300,2253],{"class":102},[76,2302,2303],{"class":86},"'[ws] 刷新凭据失败:'",[76,2305,2306],{"class":102},", e)\n",[76,2308,2309,2312,2314,2317,2319,2322,2325,2328,2330,2333,2336,2339],{"class":78,"line":106},[76,2310,2311],{"class":172},"  this",[76,2313,2247],{"class":102},[76,2315,2316],{"class":82},"_setConnected",[76,2318,2253],{"class":102},[76,2320,2321],{"class":172},"false",[76,2323,2324],{"class":102},", ",[76,2326,2327],{"class":86},"'error'",[76,2329,2324],{"class":102},[76,2331,2332],{"class":86},"`凭据刷新失败: ${",[76,2334,2335],{"class":102},"e",[76,2337,2338],{"class":86},"}`",[76,2340,2259],{"class":102},[76,2342,2343],{"class":78,"line":121},[76,2344,243],{"class":102},[10,2346,2347,2348,2351,2352,2355],{},"如果 ",[51,2349,2350],{},"api.readOpenclawConfig()"," 或 ",[51,2353,2354],{},"api.autoPairDevice()"," 抛错——磁盘 IO 抖、Rust 端处理器忙、Tauri IPC 队列阻塞——这次重连就永久终止。便携磁盘的 IO 抖动 + 配置文件读写争抢时最容易发生。",[2184,2357,2359],{"id":2358},"路径-b认证失败后的强制关闭","路径 B：认证失败后的强制关闭",[10,2361,2362],{},"当 WebSocket 收到 1008 unauthorized 响应时的处理逻辑：",[56,2364,2366],{"className":2197,"code":2365,"language":2199,"meta":64,"style":64},"if (this._authRetryCount \u003C 2) {\n  this._authRetryCount++\n  this._refreshCredentialsAndReconnect()\n  return\n}\nthis._setConnected(false, 'auth_failed', `认证失败: ${e.reason}。请检查 Gateway Token 配置。`)\nthis._intentionalClose = true   \u002F\u002F ← 永久关闭标记\nthis._flushPending()\nreturn\n",[51,2367,2368,2388,2398,2409,2414,2418,2452,2469,2481],{"__ignoreMap":64},[76,2369,2370,2373,2375,2377,2380,2382,2385],{"class":78,"line":79},[76,2371,2372],{"class":269},"if",[76,2374,2231],{"class":102},[76,2376,2206],{"class":172},[76,2378,2379],{"class":102},"._authRetryCount ",[76,2381,446],{"class":269},[76,2383,2384],{"class":172}," 2",[76,2386,2387],{"class":102},") {\n",[76,2389,2390,2392,2395],{"class":78,"line":99},[76,2391,2311],{"class":172},[76,2393,2394],{"class":102},"._authRetryCount",[76,2396,2397],{"class":269},"++\n",[76,2399,2400,2402,2404,2406],{"class":78,"line":106},[76,2401,2311],{"class":172},[76,2403,2247],{"class":102},[76,2405,2250],{"class":82},[76,2407,2408],{"class":102},"()\n",[76,2410,2411],{"class":78,"line":121},[76,2412,2413],{"class":269},"  return\n",[76,2415,2416],{"class":78,"line":134},[76,2417,243],{"class":102},[76,2419,2420,2422,2424,2426,2428,2430,2432,2435,2437,2440,2442,2444,2447,2450],{"class":78,"line":145},[76,2421,2206],{"class":172},[76,2423,2247],{"class":102},[76,2425,2316],{"class":82},[76,2427,2253],{"class":102},[76,2429,2321],{"class":172},[76,2431,2324],{"class":102},[76,2433,2434],{"class":86},"'auth_failed'",[76,2436,2324],{"class":102},[76,2438,2439],{"class":86},"`认证失败: ${",[76,2441,2335],{"class":102},[76,2443,2247],{"class":86},[76,2445,2446],{"class":102},"reason",[76,2448,2449],{"class":86},"}。请检查 Gateway Token 配置。`",[76,2451,2259],{"class":102},[76,2453,2455,2457,2460,2462,2465],{"class":78,"line":2454},7,[76,2456,2206],{"class":172},[76,2458,2459],{"class":102},"._intentionalClose ",[76,2461,458],{"class":269},[76,2463,2464],{"class":172}," true",[76,2466,2468],{"class":2467},"sJ8bj","   \u002F\u002F ← 永久关闭标记\n",[76,2470,2472,2474,2476,2479],{"class":78,"line":2471},8,[76,2473,2206],{"class":172},[76,2475,2247],{"class":102},[76,2477,2478],{"class":82},"_flushPending",[76,2480,2408],{"class":102},[76,2482,2484],{"class":78,"line":2483},9,[76,2485,2486],{"class":269},"return\n",[10,2488,2489,2490,2493,2494,2497],{},"设置 ",[51,2491,2492],{},"_intentionalClose = true"," 之后，所有重连逻辑都被 ",[51,2495,2496],{},"if (!this._intentionalClose)"," 短路。即使只是 Gateway 重启窗口碰好出现 token 短暂不匹配，也会一次性把客户端打死，必须刷新页面才能恢复。",[2184,2499,2501],{"id":2500},"路径-c快速重连配额耗尽","路径 C：快速重连配额耗尽",[10,2503,2504,2505,191],{},"定义了常数 ",[51,2506,2507],{},"MAX_RECONNECT_ATTEMPTS = 60",[56,2509,2511],{"className":2197,"code":2510,"language":2199,"meta":64,"style":64},"if (this._reconnectAttempts >= MAX_RECONNECT_ATTEMPTS) {\n  this._setConnected(false, 'error', `连接失败，已停止重连。请手动刷新页面重试。`)\n  return\n}\n",[51,2512,2513,2532,2555,2559],{"__ignoreMap":64},[76,2514,2515,2517,2519,2521,2524,2527,2530],{"class":78,"line":79},[76,2516,2372],{"class":269},[76,2518,2231],{"class":102},[76,2520,2206],{"class":172},[76,2522,2523],{"class":102},"._reconnectAttempts ",[76,2525,2526],{"class":269},">=",[76,2528,2529],{"class":172}," MAX_RECONNECT_ATTEMPTS",[76,2531,2387],{"class":102},[76,2533,2534,2536,2538,2540,2542,2544,2546,2548,2550,2553],{"class":78,"line":99},[76,2535,2311],{"class":172},[76,2537,2247],{"class":102},[76,2539,2316],{"class":82},[76,2541,2253],{"class":102},[76,2543,2321],{"class":172},[76,2545,2324],{"class":102},[76,2547,2327],{"class":86},[76,2549,2324],{"class":102},[76,2551,2552],{"class":86},"`连接失败，已停止重连。请手动刷新页面重试。`",[76,2554,2259],{"class":102},[76,2556,2557],{"class":78,"line":106},[76,2558,2413],{"class":269},[76,2560,2561],{"class":78,"line":121},[76,2562,243],{"class":102},[10,2564,2565],{},"客户机晚上挂机，Gateway 因为便携磁盘 GC 或 Windows 休眠争抢资源短暂掉线，客户端尝试 60 次仍未成功重连，就永久停摆。早上用户回来面板已成死链。",[14,2567,2568],{"id":2568},"为什么三条缺陷同时存在",[10,2570,2571],{},"这是逐步累加的历史债：",[1202,2573,2574,2581,2584],{},[298,2575,2576,2577,2580],{},"路径 B 是在修复\"避免无限自动配对循环\"时添加的保护，但用错了对象——",[51,2578,2579],{},"_intentionalClose=true"," 本来是用户主动断开的语义，不该用在被动失败上",[298,2582,2583],{},"路径 C 是早期\"避免无穷重试\"的防护，但 60 次后完全放弃而不留任何复活路径是绝对错误",[298,2585,2586],{},"路径 A 是在\"凭据 reload\"改动时把所有重连都改走 refreshCredentials，没留意 catch 分支已经成了终态",[10,2588,2589],{},"三条各自独立、都能单独打死客户端，组合在一起命中率非常高。",[14,2591,2592],{"id":2592},"修复方案",[10,2594,2595],{},"核心思想是永不彻底放弃。任何\"快速重连配额耗尽\"的分支都转入慢轮询，留出窗口让用户改配置或等待 Gateway 恢复后能自动复连。",[10,2597,2598,2599,2602],{},"新增辅助方法 ",[51,2600,2601],{},"_schedulePoll(delayMs, kind)"," 作为慢轮询的触发器：",[56,2604,2606],{"className":2197,"code":2605,"language":2199,"meta":64,"style":64},"const AUTH_RETRY_LIMIT = 2\nconst SLOW_POLL_DELAY_AUTH = 60_000      \u002F\u002F 认证持续失败：1 分钟探一次\nconst SLOW_POLL_DELAY_GENERAL = 300_000  \u002F\u002F 一般持续失败：5 分钟探一次\n\n_schedulePoll(delayMs, kind) {\n  this._clearReconnectTimer()\n  this._reconnectAttempts = 0\n  if (kind === 'auth') this._authRetryCount = 0\n  this._reconnectState = 'scheduled'\n  this._pendingReconnect = true\n  this._reconnectTimer = setTimeout(() => {\n    this._reconnectTimer = null\n    if (this._intentionalClose) return\n    this._reconnectState = 'attempting'\n    if (kind === 'auth') {\n      this._refreshCredentialsAndReconnect(0)\n    } else {\n      this._doConnect()\n    }\n  }, delayMs)\n}\n",[51,2607,2608,2622,2637,2652,2657,2665,2676,2687,2711,2723,2736,2753,2765,2780,2792,2805,2821,2832,2844,2850,2856],{"__ignoreMap":64},[76,2609,2610,2613,2616,2619],{"class":78,"line":79},[76,2611,2612],{"class":269},"const",[76,2614,2615],{"class":172}," AUTH_RETRY_LIMIT",[76,2617,2618],{"class":269}," =",[76,2620,2621],{"class":172}," 2\n",[76,2623,2624,2626,2629,2631,2634],{"class":78,"line":99},[76,2625,2612],{"class":269},[76,2627,2628],{"class":172}," SLOW_POLL_DELAY_AUTH",[76,2630,2618],{"class":269},[76,2632,2633],{"class":172}," 60_000",[76,2635,2636],{"class":2467},"      \u002F\u002F 认证持续失败：1 分钟探一次\n",[76,2638,2639,2641,2644,2646,2649],{"class":78,"line":106},[76,2640,2612],{"class":269},[76,2642,2643],{"class":172}," SLOW_POLL_DELAY_GENERAL",[76,2645,2618],{"class":269},[76,2647,2648],{"class":172}," 300_000",[76,2650,2651],{"class":2467},"  \u002F\u002F 一般持续失败：5 分钟探一次\n",[76,2653,2654],{"class":78,"line":121},[76,2655,2656],{"emptyLinePlaceholder":1268},"\n",[76,2658,2659,2662],{"class":78,"line":134},[76,2660,2661],{"class":82},"_schedulePoll",[76,2663,2664],{"class":102},"(delayMs, kind) {\n",[76,2666,2667,2669,2671,2674],{"class":78,"line":145},[76,2668,2311],{"class":172},[76,2670,2247],{"class":102},[76,2672,2673],{"class":82},"_clearReconnectTimer",[76,2675,2408],{"class":102},[76,2677,2678,2680,2682,2684],{"class":78,"line":2454},[76,2679,2311],{"class":172},[76,2681,2523],{"class":102},[76,2683,458],{"class":269},[76,2685,2686],{"class":172}," 0\n",[76,2688,2689,2691,2694,2697,2700,2703,2705,2707,2709],{"class":78,"line":2471},[76,2690,2228],{"class":269},[76,2692,2693],{"class":102}," (kind ",[76,2695,2696],{"class":269},"===",[76,2698,2699],{"class":86}," 'auth'",[76,2701,2702],{"class":102},") ",[76,2704,2206],{"class":172},[76,2706,2379],{"class":102},[76,2708,458],{"class":269},[76,2710,2686],{"class":172},[76,2712,2713,2715,2718,2720],{"class":78,"line":2483},[76,2714,2311],{"class":172},[76,2716,2717],{"class":102},"._reconnectState ",[76,2719,458],{"class":269},[76,2721,2722],{"class":86}," 'scheduled'\n",[76,2724,2726,2728,2731,2733],{"class":78,"line":2725},10,[76,2727,2311],{"class":172},[76,2729,2730],{"class":102},"._pendingReconnect ",[76,2732,458],{"class":269},[76,2734,2735],{"class":172}," true\n",[76,2737,2739,2741,2743,2745,2747,2749,2751],{"class":78,"line":2738},11,[76,2740,2311],{"class":172},[76,2742,2209],{"class":102},[76,2744,458],{"class":269},[76,2746,2214],{"class":82},[76,2748,2217],{"class":102},[76,2750,2220],{"class":269},[76,2752,2223],{"class":102},[76,2754,2756,2758,2760,2762],{"class":78,"line":2755},12,[76,2757,2244],{"class":172},[76,2759,2209],{"class":102},[76,2761,458],{"class":269},[76,2763,2764],{"class":172}," null\n",[76,2766,2768,2771,2773,2775,2778],{"class":78,"line":2767},13,[76,2769,2770],{"class":269},"    if",[76,2772,2231],{"class":102},[76,2774,2206],{"class":172},[76,2776,2777],{"class":102},"._intentionalClose) ",[76,2779,2486],{"class":269},[76,2781,2783,2785,2787,2789],{"class":78,"line":2782},14,[76,2784,2244],{"class":172},[76,2786,2717],{"class":102},[76,2788,458],{"class":269},[76,2790,2791],{"class":86}," 'attempting'\n",[76,2793,2795,2797,2799,2801,2803],{"class":78,"line":2794},15,[76,2796,2770],{"class":269},[76,2798,2693],{"class":102},[76,2800,2696],{"class":269},[76,2802,2699],{"class":86},[76,2804,2387],{"class":102},[76,2806,2808,2811,2813,2815,2817,2819],{"class":78,"line":2807},16,[76,2809,2810],{"class":172},"      this",[76,2812,2247],{"class":102},[76,2814,2250],{"class":82},[76,2816,2253],{"class":102},[76,2818,2256],{"class":172},[76,2820,2259],{"class":102},[76,2822,2824,2827,2830],{"class":78,"line":2823},17,[76,2825,2826],{"class":102},"    } ",[76,2828,2829],{"class":269},"else",[76,2831,2223],{"class":102},[76,2833,2835,2837,2839,2842],{"class":78,"line":2834},18,[76,2836,2810],{"class":172},[76,2838,2247],{"class":102},[76,2840,2841],{"class":82},"_doConnect",[76,2843,2408],{"class":102},[76,2845,2847],{"class":78,"line":2846},19,[76,2848,2849],{"class":102},"    }\n",[76,2851,2853],{"class":78,"line":2852},20,[76,2854,2855],{"class":102},"  }, delayMs)\n",[76,2857,2859],{"class":78,"line":2858},21,[76,2860,243],{"class":102},[10,2862,2863,2864,2866,2867,2870],{},"慢轮询命中后失败会重新进入 ",[51,2865,2193],{}," 快速重连周期（因为 ",[51,2868,2869],{},"_reconnectAttempts"," 重置为 0），相当于\"快速 60 次 → 慢一次 → 快速 60 次 → 慢一次\"的循环，永不放弃。",[10,2872,2873],{},"三条修复并行：",[10,2875,2876,2879,2880,2883],{},[44,2877,2878],{},"Fix A","：普通重连直接走 ",[51,2881,2882],{},"_doConnect()","，凭据刷新隔离到认证失败分支，避免配置读取错误牵连普通重连。",[10,2885,2886,2889,2890,2893,2894,2896],{},[44,2887,2888],{},"Fix B","：认证失败耗尽后转入 ",[51,2891,2892],{},"_schedulePoll('auth')","，不再设置 ",[51,2895,2579],{},"，给认证恢复留出 60 秒的探测窗口。",[10,2898,2899,2902,2903,2906],{},[44,2900,2901],{},"Fix C","：重连次数超过 MAX_RECONNECT_ATTEMPTS 后转入 ",[51,2904,2905],{},"_schedulePoll('general')","，设置 UI 状态为\"连接持续失败，300 秒后重试\"而不是终态。",[10,2908,2909],{},"慢轮询失败后重新进入快速重连周期，形成\"快速 60 次 → 慢一次 → 快速 60 次\"的循环，永不放弃。",[14,2911,2912],{"id":2912},"防回归验证",[10,2914,2915,2916,2919],{},"新增 ",[51,2917,2918],{},"wakou-full\u002Fsrc\u002Flib\u002Fws-client.slow-poll.test.js","，覆盖 7 个 case：",[1202,2921,2922,2930,2942,2945,2954,2961,2966],{},[298,2923,2924,2925,2927,2928],{},"Fix A：普通重连命中 ",[51,2926,2841],{},"，不调用 ",[51,2929,2250],{},[298,2931,2932,2933,2936,2937,2939,2940],{},"Fix C：MAX_RECONNECT_ATTEMPTS 后状态是 ",[51,2934,2935],{},"reconnecting","（非终态 ",[51,2938,1869],{},"），5 分钟后触发 ",[51,2941,2841],{},[298,2943,2944],{},"Fix C：第二轮慢轮询失败后还能再调度快速重连",[298,2946,2947,2948,2950,2951],{},"Fix B：",[51,2949,2892],{}," 等待 60 秒调用 ",[51,2952,2953],{},"_refreshCredentialsAndReconnect(0)",[298,2955,2947,2956,2958,2959],{},[51,2957,2905],{}," 等待后调用 ",[51,2960,2841],{},[298,2962,2963,2965],{},[51,2964,2579],{}," 时慢轮询应跳过",[298,2967,2968,2970],{},[51,2969,2250],{}," 抛错路径调度慢轮询",[10,2972,2973],{},"测试全部通过（7\u002F7）。",[14,2975,2976],{"id":2976},"外部因素的叠加",[10,2978,2979],{},"这个时期同步发生了另一个外部问题：CDN 对 WebSocket 空闲连接会在 4-9 分钟后强制切断。客户端收到连接中断后根据指数退避策略重连，如果恰好命中上述三条路径之一，就进入永久断开状态。这解释了为什么故障特别在长时间挂机后高发——空闲足够长，CDN 必定切断，而后续重连很容易落入某条缺陷路径。两个问题各自独立，但组合效果是\"挂机过夜必死\"。",[14,2981,2982],{"id":2982},"排查验证",[10,2984,2985],{},"故障修复后，可以用以下方式验证重连状态：",[56,2987,2989],{"className":2197,"code":2988,"language":2199,"meta":64,"style":64},"__clawpanelWsClient.getConnectionInfo()\n\u002F\u002F {\n\u002F\u002F   connected: false,\n\u002F\u002F   reconnectState: 'scheduled' | 'attempting',\n\u002F\u002F   reconnectAttempts: 0~60,\n\u002F\u002F   ...\n\u002F\u002F }\n",[51,2990,2991,3001,3006,3011,3016,3021,3026],{"__ignoreMap":64},[76,2992,2993,2996,2999],{"class":78,"line":79},[76,2994,2995],{"class":102},"__clawpanelWsClient.",[76,2997,2998],{"class":82},"getConnectionInfo",[76,3000,2408],{"class":102},[76,3002,3003],{"class":78,"line":99},[76,3004,3005],{"class":2467},"\u002F\u002F {\n",[76,3007,3008],{"class":78,"line":106},[76,3009,3010],{"class":2467},"\u002F\u002F   connected: false,\n",[76,3012,3013],{"class":78,"line":121},[76,3014,3015],{"class":2467},"\u002F\u002F   reconnectState: 'scheduled' | 'attempting',\n",[76,3017,3018],{"class":78,"line":134},[76,3019,3020],{"class":2467},"\u002F\u002F   reconnectAttempts: 0~60,\n",[76,3022,3023],{"class":78,"line":145},[76,3024,3025],{"class":2467},"\u002F\u002F   ...\n",[76,3027,3028],{"class":78,"line":2454},[76,3029,3030],{"class":2467},"\u002F\u002F }\n",[10,3032,3033,3036,3037,3040,3041,3036,3044,3047],{},[51,3034,3035],{},"reconnectState='scheduled'"," 且 ",[51,3038,3039],{},"reconnectAttempts=0"," 表示在慢轮询窗口正常工作；",[51,3042,3043],{},"reconnectState='idle'",[51,3045,3046],{},"connected=false"," 是老的永久断开路径。",[10,3049,3050],{},"重连机制里避免静默终态是最基本的要求。任何异常路径都必须有可见的状态提示和后续操作入口，否则用户端看到的就是死机。这次故障的根本教训是，不能让客户端在任何情况下进入\"不可恢复\"的状态而没有任何提示。慢轮询的引入给了所有失败情景一个\"最后的机会\"，即使前面的快速重连机制彻底耗尽了，用户等待足够长的时间后系统仍有自动恢复的可能。",[1252,3052,3053],{},"html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}",{"title":64,"searchDepth":99,"depth":99,"links":3055},[3056,3061,3062,3063,3064,3065],{"id":2179,"depth":99,"text":2179,"children":3057},[3058,3059,3060],{"id":2186,"depth":106,"text":2187},{"id":2358,"depth":106,"text":2359},{"id":2500,"depth":106,"text":2501},{"id":2568,"depth":99,"text":2568},{"id":2592,"depth":99,"text":2592},{"id":2912,"depth":99,"text":2912},{"id":2976,"depth":99,"text":2976},{"id":2982,"depth":99,"text":2982},"2026-06-17","从 2026-05-11 开始，多台便携包客户反馈 WebSocket 断线后面板卡死。网关进程（openclaw_gateway.exe）明确在运行，\u002Fhealth 返回 200，但面板显示\"已停止重连，请手动刷新\"——用户必须手动重启才能恢复。",{},"\u002F2026-06-17-fa-016-ws",{"title":2149,"description":3067},"FA-016","2026-06-17-FA-016-WS永久断开三条路径","客户端重连逻辑中三处独立的\"静默终止\"缺陷导致 WS 永久断开，任何一条路径命中就不再调度重连计时器。",[3075,3076,3077,3078,3079],"WebSocket","Node.js","客户端重连机制","异常处理","便携包","qaISKpAwjYnhiSdxEMYYnJtJ-A8QaOq-djOEJ0W9-bM",1785406912288]