[{"data":1,"prerenderedAt":2407},["ShallowReactive",2],{"\u002F2026-05-25-fa-013-p0":3,"\u002F2026-05-25-fa-013-p0-rel":616},{"id":4,"title":5,"body":6,"column":597,"date":598,"description":12,"extension":599,"hero_image":600,"meta":601,"navigation":602,"path":603,"seo":604,"series_id":605,"severity":600,"stem":606,"summary":607,"tags":608,"__hash__":615},"posts\u002F2026-05-25-FA-013-九缺陷叠加P0.md","一天，247 次 OOM",{"type":7,"value":8,"toc":589},"minimark",[9,13,16,19,24,27,72,75,83,86,89,95,98,105,193,204,207,210,213,216,279,282,285,288,291,309,315,336,339,342,348,354,364,367,370,373,544,547,550,553,556,559,562,582,585],[10,11,12],"p",{},"247 次。",[10,14,15],{},"一个容器一天被 cgroup OOM 杀了 247 次。每次 kill 加 supervisord 拉起新进程，22 秒的启动窗口里所有 API 调用都在超时。累计掉线时间超过 1.5 小时。而这只是冰山一角——整个故障链条里还有八个独立的缺陷在同步放大。",[10,17,18],{},"2026-05-24 下午 17:30 起，大量用户反馈掉线。表面上是三种完全不同的症状：网关无响应、插件拿不到 token、WebSocket 频繁断线重连。如果顺着单一假设深挖——比如\"是不是 token 服务故障\"——永远也理不清楚。症状看起来毫无关联，但其实源于同一条触发链的不同环节失效。",[20,21,23],"h2",{"id":22},"为什么症状分散根因却统一","为什么症状分散，根因却统一",[10,25,26],{},"SSH 进服务器第一件事：",[28,29,34],"pre",{"className":30,"code":31,"language":32,"meta":33,"style":33},"language-bash shiki shiki-themes github-light github-dark","sudo dmesg -T | grep \"Killed process.*openclaw\" | head -20\n","bash","",[35,36,37],"code",{"__ignoreMap":33},[38,39,42,46,50,54,58,61,64,66,69],"span",{"class":40,"line":41},"line",1,[38,43,45],{"class":44},"sScJk","sudo",[38,47,49],{"class":48},"sZZnC"," dmesg",[38,51,53],{"class":52},"sj4cs"," -T",[38,55,57],{"class":56},"szBVR"," |",[38,59,60],{"class":44}," grep",[38,62,63],{"class":48}," \"Killed process.*openclaw\"",[38,65,57],{"class":56},[38,67,68],{"class":44}," head",[38,70,71],{"class":52}," -20\n",[10,73,74],{},"输出：",[28,76,81],{"className":77,"code":79,"language":80},[78],"language-text","[21:35:14] Killed process 1409836 (openclaw) total-vm:1570176kB, anon-rss:512000kB\n[21:35:15] Killed process 1409924 (openclaw)\n[21:35:52] Killed process 1413381 (openclaw)\n","text",[35,82,79],{"__ignoreMap":33},[10,84,85],{},"2 分钟 15 次 kill。内存限制 500MB 的容器在高负载下不够用，系统开始清理。gateway 进程死后，supervisord 重启它，22 秒内启动完成才能服务请求。这个窗口里 panel API 全部超时。现象一：网关掉线。",[10,87,88],{},"但这还解释不了\"token 过期\"。进容器看日志，大量这样的：",[28,90,93],{"className":91,"code":92,"language":80},[78],"[21:39:39] [reload] config watcher error: EMFILE: too many open files\n[21:44:10] [skills] watcher error: EMFILE: too many open files\n",[35,94,92],{"__ignoreMap":33},[10,96,97],{},"同时 registerCpaConnectorMcp 失败。查 DB 对比，容器的 openclaw.json 里的 token 与数据库的 active token 不一致。这说明配置写入本身出了问题。",[10,99,100,101,104],{},"检查代码 ",[35,102,103],{},"apps\u002Fapi\u002Fsrc\u002Fconnector\u002Fmcp-register.ts","：",[28,106,110],{"className":107,"code":108,"language":109,"meta":33,"style":33},"language-typescript shiki shiki-themes github-light github-dark","if (r.exitCode !== 0) {\n  logger.warn({...}, 'mcp register exited non-zero');\n  return { status: 'error', reason: `exit=${r.exitCode}` };\n}\n","typescript",[35,111,112,130,154,187],{"__ignoreMap":33},[38,113,114,117,121,124,127],{"class":40,"line":41},[38,115,116],{"class":56},"if",[38,118,120],{"class":119},"sVt8B"," (r.exitCode ",[38,122,123],{"class":56},"!==",[38,125,126],{"class":52}," 0",[38,128,129],{"class":119},") {\n",[38,131,133,136,139,142,145,148,151],{"class":40,"line":132},2,[38,134,135],{"class":119},"  logger.",[38,137,138],{"class":44},"warn",[38,140,141],{"class":119},"({",[38,143,144],{"class":56},"...",[38,146,147],{"class":119},"}, ",[38,149,150],{"class":48},"'mcp register exited non-zero'",[38,152,153],{"class":119},");\n",[38,155,157,160,163,166,169,172,175,178,181,184],{"class":40,"line":156},3,[38,158,159],{"class":56},"  return",[38,161,162],{"class":119}," { status: ",[38,164,165],{"class":48},"'error'",[38,167,168],{"class":119},", reason: ",[38,170,171],{"class":48},"`exit=${",[38,173,174],{"class":119},"r",[38,176,177],{"class":48},".",[38,179,180],{"class":119},"exitCode",[38,182,183],{"class":48},"}`",[38,185,186],{"class":119}," };\n",[38,188,190],{"class":40,"line":189},4,[38,191,192],{"class":119},"}\n",[10,194,195,196,199,200,203],{},"openclaw CLI 用 exit code 1 表示\"完成但有警告\"。代码把所有非零当失败。虽然 stderr 里有 ",[35,197,198],{},"Config overwrite"," 痕迹（文件其实已写入），但这个误判触发了 ",[35,201,202],{},"clearReconcileCooldown","，下次 WebSocket 重连时又重试一遍。每次重试就是一次 SIGTERM。最终统计：1366 次 mcp register 报失败（大部分是假失败），对应 1366 次虚假 SIGTERM。现象二：网关频繁掉线。",[10,205,206],{},"现象三：WebSocket 4-9 分钟就断一次。查 CDN 日志，Tencent EdgeOne 对 idle 连接有强制切断策略。客户端自动重连，形成流量风暴。",[10,208,209],{},"三个症状，三个独立的源头，但都指向同一条链路的不同节点失效。正因为如此，修复不能是单一 patch——需要同时拆除这条链上的所有地雷。",[20,211,212],{"id":212},"九个缺陷与它们的叠加效应",[10,214,215],{},"做完全景取证后，缺陷清单出现了：",[217,218,219,227,233,239,245,251,261,267,273],"ol",{},[220,221,222,226],"li",{},[223,224,225],"strong",{},"B1"," 容器内存限制 500MB 严重不足 → gateway 进程 OOM kill → 22s 启动窗口 → panel 调用超时",[220,228,229,232],{},[223,230,231],{},"B2"," CLI exit code 1 被误判为失败 → 1366 次虚假 SIGTERM → 指数级恶化掉线",[220,234,235,238],{},[223,236,237],{},"B3"," ulimit nofile=1024 太低 → EMFILE → file watcher 失败 → 配置无法热重载",[220,240,241,244],{},[223,242,243],{},"B4"," mcp reconcile cooldown 在 skip 时也被清空 → 每次 WS 重连重试 → 无效 SIGTERM 更多",[220,246,247,250],{},[223,248,249],{},"B5"," registry kick-on-replace → 多客户端互踢 → 工具调用中途断裂",[220,252,253,256,257,260],{},[223,254,255],{},"B6"," ",[35,258,259],{},"\u002Finstance\u002Fprovision"," 不幂等 → 新用户注册失败 → 14 次 AlreadyExists 错误\u002F天",[220,262,263,266],{},[223,264,265],{},"B7"," event loop 阻塞 7-18 秒 → tools\u002Fcall 5 分钟超时 → 整机压力指数级增加",[220,268,269,272],{},[223,270,271],{},"B8"," CDN 对 WebSocket idle 4-9 分钟强制切断 → 客户端循环重连 → 流量风暴",[220,274,275,278],{},[223,276,277],{},"B9"," 7 个用户仍跑老 EXE → 跟 Tauri 1.0.8 共存互踢 → 特定用户反复掉线",[10,280,281],{},"单独看，B3（文件描述符限制）只会导致某些 watcher 失败。B4（cooldown 清空）只是重试次数多一些。但当 B1（内存溢出）把整机资源挤到极限时，B2（虚假 SIGTERM）的流量放大、B4 的无限重试、B8 的 CDN 断线形成了一个正反馈循环。12GiB swap 活跃交换，主线程不断触发 page fault，event loop 阻塞最高达 18.8 秒。整个服务器的 system load 从正常的 2-3 飙升到 18.96。",[10,283,284],{},"不止受害的那个 247 次 OOM 的容器宕掉了。所有 372 个容器的网关进程都在 event loop 阻塞中挣扎，形成连锁故障。",[20,286,287],{"id":287},"凌晨部署与并行推进",[10,289,290],{},"修复分两条线并行。",[10,292,293,296,297,300,301,304,305,308],{},[223,294,295],{},"基础设施层","（立即上线）：部署 WebSocket 直连入口，绕过 CDN 的 idle timeout。DNS 添加新直连域名的 A 记录，Caddy 仅暴露 ",[35,298,299],{},"\u002Fapi\u002Fconnector\u002Fws"," 和 ",[35,302,303],{},"\u002Fhealth","。客户端 1.0.9 OTA 时读 ",[35,306,307],{},"ws_endpoint"," 字段，动态切换通路。",[10,310,311,314],{},[223,312,313],{},"业务代码层","（业务低谷部署）：五个 patch，530 行改动：",[316,317,318,321,324,327,330],"ul",{},[220,319,320],{},"B1：500MB → 2GB hard limit",[220,322,323],{},"B2：exit code 判定加 stderr marker 检查",[220,325,326],{},"B3：ulimit nofile 1024 → 65536",[220,328,329],{},"B4：cooldown skip 时不清",[220,331,332,333,335],{},"B6：",[35,334,259],{}," 加 AlreadyExists 幂等处理",[10,337,338],{},"P1 级（稍后）：B5 改为拒绝新连接而非踢掉旧连接；B9 的老 EXE 用户做 revoke 和自检。",[10,340,341],{},"凌晨 02:00 开始。前置备份 30 分钟（PG、repo、Caddyfile、image、swarm spec、用户数据）。",[10,343,344,347],{},[223,345,346],{},"阶段 1 代码部署","：pnpm build，rsync dist 到服务器，原子切换（mv），restart cpa-api。全程 5 秒，用户感知的掉线比预估的 30 秒更短。",[10,349,350,353],{},[223,351,352],{},"阶段 2 监控验证","：部署后 5 分钟内检查指标。B2 生效（1366 失败 → 9h 内 132）、B4 生效（cooldown 日志出现）、B6 生效（新注册零错误）。全部通过。",[10,355,356,359,360,363],{},[223,357,358],{},"阶段 3 滚动升级","：优先升级 6 个重灾户（含 247 次那个），5.5 分钟完成。升级后检查 dmesg，无 OOM kill。剩余 ~366 个容器一次 10 个并行，间隔 30 秒。单容器 ",[35,361,362],{},"docker service update"," 耗时 60 秒（含 swarm scheduler delay）。总耗时 6h42min（预估 3 小时，实际慢一倍）。但系统压力从升级到 ~80 个容器时就开始回落。",[10,365,366],{},"0 failed，0 回滚。",[20,368,369],{"id":369},"数据说话",[10,371,372],{},"部署 24 小时后对比：",[374,375,376,395],"table",{},[377,378,379],"thead",{},[380,381,382,386,389,392],"tr",{},[383,384,385],"th",{},"指标",[383,387,388],{},"部署前",[383,390,391],{},"部署后 9h",[383,393,394],{},"改善",[396,397,398,415,431,447,463,480,497,511,528],"tbody",{},[380,399,400,404,407,412],{},[401,402,403],"td",{},"容器 OOM kill（重灾户）",[401,405,406],{},"247 次\u002F天",[401,408,409],{},[223,410,411],{},"0",[401,413,414],{},"100%",[380,416,417,420,423,426],{},[401,418,419],{},"Swap 使用",[401,421,422],{},"12 GiB",[401,424,425],{},"9 MiB",[401,427,428],{},[223,429,430],{},"-99.9%",[380,432,433,436,439,442],{},[401,434,435],{},"System load（1min）",[401,437,438],{},"18.96",[401,440,441],{},"8.46",[401,443,444],{},[223,445,446],{},"-55%",[380,448,449,452,455,458],{},[401,450,451],{},"可用内存",[401,453,454],{},"58 GiB",[401,456,457],{},"95 GiB",[401,459,460],{},[223,461,462],{},"+64%",[380,464,465,471,474,477],{},[401,466,467,470],{},[35,468,469],{},"mcp register"," 真失败",[401,472,473],{},"1366\u002F天（混+假）",[401,475,476],{},"132\u002F9h",[401,478,479],{},"~88% ↓",[380,481,482,487,489,494],{},[401,483,484,486],{},[35,485,469],{}," 假失败救活",[401,488,411],{},[401,490,491],{},[223,492,493],{},"26 次",[401,495,496],{},"新指标",[380,498,499,502,505,509],{},[401,500,501],{},"AlreadyExists 新用户错误",[401,503,504],{},"14\u002F天",[401,506,507],{},[223,508,411],{},[401,510,414],{},[380,512,513,519,522,526],{},[401,514,515,518],{},[35,516,517],{},"connection replaced"," 工具中断",[401,520,521],{},"18\u002F天",[401,523,524],{},[223,525,411],{},[401,527,414],{},[380,529,530,535,538,541],{},[401,531,532],{},[35,533,534],{},"tools\u002Fcall exception",[401,536,537],{},"35\u002F天",[401,539,540],{},"2",[401,542,543],{},"~95% ↓",[10,545,546],{},"用户反馈中\"网关掉线\"消失，\"插件用不了\"归零。",[20,548,549],{"id":549},"防回归与巡检",[10,551,552],{},"五个 patch 配套 19 个单元测试 case（vitest，135ms 跑完），覆盖 B1\u002FB2\u002FB3\u002FB6 的临界条件。新增 PG 自动备份 cron（每日 03:00），swarm spec 全量快照存档用于灾难重建。部署 checklist 和回滚脚本纳入标准流程。",[10,554,555],{},"但多缺陷叠加的\"系统级阈值\"仍缺持续巡检——这不能用传统的单点告警解决。需要在后续单独设计。",[20,557,558],{"id":558},"调查多缺陷故障的方法",[10,560,561],{},"多缺陷叠加时，症状看起来统一但根因分散。不能顺着单一假设深挖（比如假设\"一定是 token 问题\"），会被带进死胡同。正确的做法：",[217,563,564,570,576],{},[220,565,566,569],{},[223,567,568],{},"全景取证","：把所有异常信号（OOM、EMFILE、token 不一致、event loop 阻塞、WebSocket thrash）列出来，不预判因果关系。",[220,571,572,575],{},[223,573,574],{},"分类而非追踪","：不要从某个症状开始追链条，而是把信号按系统层级分组，看哪些是独立根因、哪些是被放大的现象。",[220,577,578,581],{},[223,579,580],{},"按影响面排优先级","：单个 bug 的修复收益不一定最大。有时修复跨层级放大的系统级缺陷（比如错误的 exit code 判定 + 无 cooldown 保护）的收益，远超修复看起来最直接的单个 bug（比如内存限制）。",[10,583,584],{},"这次故障的真正杀伤力不来自 B1 的内存溢出，而来自 B2（虚假 SIGTERM）+ B4（cooldown 清空）的组合——它们把单个容器的故障放大成了全站连锁反应。",[586,587,588],"style",{},"html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}",{"title":33,"searchDepth":132,"depth":132,"links":590},[591,592,593,594,595,596],{"id":22,"depth":132,"text":23},{"id":212,"depth":132,"text":212},{"id":287,"depth":132,"text":287},{"id":369,"depth":132,"text":369},{"id":549,"depth":132,"text":549},{"id":558,"depth":132,"text":558},"故障档案","2026-05-25","md",null,{},true,"\u002F2026-05-25-fa-013-p0",{"title":5,"description":12},"FA-013","2026-05-25-FA-013-九缺陷叠加P0","一场波及全站用户掉线的故障根因不是单一 bug，而是九个独立缺陷在内存压力下叠加放大；修复需要基础设施和业务代码两条线并行推进。",[609,610,611,612,613,614],"容器调度","内存管理","WebSocket","分布式系统","故障诊断","cgroup","-HmhblzI7fSTtvtqjX9kzYVJF2SMwjtWQlD5yvkx0O8",[617,1019,1477],{"id":618,"title":619,"body":620,"column":597,"date":1004,"description":624,"extension":599,"hero_image":600,"meta":1005,"navigation":602,"path":1006,"seo":1007,"series_id":1008,"severity":600,"stem":1009,"summary":1010,"tags":1011,"__hash__":1018},"posts\u002F2026-07-23-FA-018-共享存储的subPath陷阱.md","一个 PVC 装所有用户，当时看不出有什么问题",{"type":7,"value":621,"toc":995},[622,625,628,632,635,638,645,686,689,699,703,709,712,723,726,730,733,736,739,757,760,767,770,785,789,792,795,806,817,820,823,826,888,891,898,901,904,930,933,944,947,950,989,992],[10,623,624],{},"平台从 Docker Swarm 迁移到 Kubernetes，存储架构做了一个看似理想的调整：用共享 PVC + subPath 替代原来的\"每用户独占 PVC\"。方案优势显而易见——管理简单（1 个 PVC 对 574 个）、自动扩展、成本低。当时看不出有什么缺点。",[10,626,627],{},"但实施中遇到了四个隐蔽的问题。",[20,629,631],{"id":630},"权限错配bind-型数据的属主陷阱","权限错配：bind 型数据的属主陷阱",[10,633,634],{},"旧系统中，用户数据通过两种方式持久化：bind mount 和 Docker volume。迁移时需要从这两个来源完整导出数据。",[10,636,637],{},"对于 bind mount 型数据，文件属主通常是 10001（或其他固定 UID）。当我登上旧集群的宿主机，以普通用户身份尝试读取这些文件时，只能看到 13 个文件。权限拒绝。同一目录里其实有 4778 个文件，但大多数因为属主权限不匹配而不可见。",[10,639,640,641,644],{},"只有用 sudo 才能完整读到。所以迁移脚本必须用 ",[35,642,643],{},"sudo tar"," 来打包：",[28,646,648],{"className":30,"code":647,"language":32,"meta":33,"style":33},"sudo tar -czf \u002Ftmp\u002Fmig\u002F${uid}\u002Fdata.tgz \\\n  -C \u002Fmnt\u002Fyun-claw\u002Fusers\u002F${uid} .\n",[35,649,650,672],{"__ignoreMap":33},[38,651,652,654,657,660,663,666,669],{"class":40,"line":41},[38,653,45],{"class":44},[38,655,656],{"class":48}," tar",[38,658,659],{"class":52}," -czf",[38,661,662],{"class":48}," \u002Ftmp\u002Fmig\u002F",[38,664,665],{"class":119},"${uid}",[38,667,668],{"class":48},"\u002Fdata.tgz",[38,670,671],{"class":52}," \\\n",[38,673,674,677,680,683],{"class":40,"line":132},[38,675,676],{"class":52},"  -C",[38,678,679],{"class":48}," \u002Fmnt\u002Fyun-claw\u002Fusers\u002F",[38,681,682],{"class":119},"${uid} ",[38,684,685],{"class":48},".\n",[10,687,688],{},"这不是脚本设计的问题，而是 Kubernetes 环境下容器权限与宿主权限映射的一个陷阱。容器内运行的网关进程可能以容器用户身份运行，但宿主上的文件属主是不同的 UID。当两个权限体系接不上，访问就会失败。",[10,690,691,692,695,696,698],{},"新方案中，用户数据通过 subPath 挂到容器的 ",[35,693,694],{},"\u002Fdata"," 目录。如果 subPath 指向的目录是由 Kubernetes 在宿主机上创建的，属主可能是 root 或其他用户，容器内进程（以容器指定的 UID 运行）仍然会遭遇权限问题。这个风险需要在 Dockerfile 和容器启动时明确处理：容器内应该预先创建 ",[35,697,694],{}," 并设置正确的属主，或者使用 securityContext 的 fsGroup 或 runAsUser 来强制权限。",[20,700,702],{"id":701},"subpath-目录创建的时机与属主处理","subPath 目录创建的时机与属主处理",[10,704,705,706],{},"Kubernetes 对 subPath 的处理有个微妙之处：如果挂载的 subPath 目录在宿主机上不存在，kubelet 会自动创建它。但 ",[38,707,708],{},"待补：这个自动创建的目录的属主是什么？是 root 还是其他用户？容器的 securityContext 能否保证挂载后的权限符合预期？",[10,710,711],{},"设计文档中没有明确说明这一点。在实施前，需要验证：",[316,713,714,717,720],{},[220,715,716],{},"首次 subPath 目录不存在时，kubelet 创建它并将其属主设置为谁",[220,718,719],{},"容器的 securityContext（fsGroup、runAsUser）是否能在挂载时生效",[220,721,722],{},"是否应该预先在共享 PVC 上创建所有 subPath 目录并设置好属主，而不是依赖 Kubernetes 的隐式创建",[10,724,725],{},"没有明确的答案，就容易踩坑。一个保险做法是在 destroy 用户账户时清空 subPath 目录，同时在 provision 时让一个 init-container 验证或修复目录属主，确保容器进程有写权限。",[20,727,729],{"id":728},"单-pvc-中的节点级故障域","单 PVC 中的节点级故障域",[10,731,732],{},"这是最严重的问题，也是与 FA-015（节点假 Ready）的关联点。",[10,734,735],{},"当一个节点的 NFS 客户端或到 SFS 的网络链路发生 hang 时，所有依赖共享 PVC 的 Pod 都会受影响。这不是共享 PVC 本身的问题，而是故障隔离的问题。",[10,737,738],{},"具体现象：",[316,740,741,748,751,754],{},[220,742,743,744,747],{},"如果 NFS 挂载使用了硬 mount（",[35,745,746],{},"hard"," 参数是 NFS 的默认值），而网络故障或存储服务中断，NFS 客户端会无限重试",[220,749,750],{},"重试期间，所有试图访问 NFS 的进程都会被阻塞在 I\u002FO 上（D 状态，不可中断）",[220,752,753],{},"如果 kubelet 或 containerd 的某个操作（如创建容器的卷挂载步骤）陷入 I\u002FO 阻塞，整个节点就会冻死",[220,755,756],{},"该节点上的所有实例（无论是否在访问存储）都会因为无法创建或删除 Pod 而故障",[10,758,759],{},"这与独立 PVC 的效果截然不同。旧架构中，每个用户有独立 PVC，意味着如果某个 PVC 对应的存储有问题，只会影响这一个用户。其他用户的 PVC 可能分散在不同的存储卷甚至不同的节点上，相对独立。",[10,761,762,763,766],{},"新架构下，单个共享 PVC 承载所有用户，一旦这个 PVC 对应的网络链路或挂载出问题，",[223,764,765],{},"同节点上所有实例都会被殃及","。如果该节点碰巧堆积了 50 个实例，一次节点级的存储 hang 就会导致 50 个实例集体故障，且外表看起来是\"节点 Ready，但实例卡住\"——kubelet 的心跳正常，Pod 状态却卡在 ContainerCreating 或 Terminating。",[10,768,769],{},"这正是 FA-015 在 2026-06-05 观察到的现象：节点假 Ready，但大量实例网关无响应。防护措施包括：",[316,771,772,779,782],{},[220,773,774,775,778],{},"NFS 挂载参数改为软 mount（",[35,776,777],{},"soft,timeo=100,retrans=3","），让 I\u002FO 超时而不是无限等待",[220,780,781],{},"节点加健康检测，检查 ContainerCreating 堆积数和网关探测失败率，及早发现节点冻死",[220,783,784],{},"实例分散部署，用 topologySpreadConstraints 避免单节点堆积",[20,786,788],{"id":787},"无-per-subpath-硬配额","无 per-subPath 硬配额",[10,790,791],{},"共享存储的架构决定了无法在存储层面按 subPath 限制用户配额。SFS（及 NFS 一般）没有 per-directory 的硬配额功能，只能在整个卷级别限制。",[10,793,794],{},"这意味着：",[316,796,797,800,803],{},[220,798,799],{},"无法阻止某个用户的数据膨胀而挤占其他用户的空间",[220,801,802],{},"无法在存储层面实现\"超额用户的写入失败\"",[220,804,805],{},"只能在应用层进行监控、告警和逻辑控制",[10,807,808,809,812,813,816],{},"新系统的方案是应用层监控：定期 ",[35,810,811],{},"du -sb \u002Fdata"," 扫描每个用户的目录大小，落库到 Instance 表，",[35,814,815],{},"\u002Fstatus"," 接口返回超额标志位。前端据此提示用户\"存储已满\"。这是纯监控，不阻断。如果用户无视提示继续写入，直到共享卷真的满了，才会出现\"所有用户集体写入失败\"的惨淡局面。",[10,818,819],{},"这个限制是方案选择的代价。",[20,821,822],{"id":822},"为什么仍然选了这个方案",[10,824,825],{},"尽管有这些问题，平台仍然采用了共享 PVC + subPath 的方案。原因是对比了替代方案：",[374,827,828,844],{},[377,829,830],{},[380,831,832,835,838,841],{},[383,833,834],{},"方案",[383,836,837],{},"优点",[383,839,840],{},"缺点",[383,842,843],{},"故障隔离",[396,845,846,860,874],{},[380,847,848,851,854,857],{},[401,849,850],{},"独立 PVC（旧架构）",[401,852,853],{},"天然的每用户隔离；故障域清晰",[401,855,856],{},"管理复杂（574 个 PVC）；扩展性差；成本高",[401,858,859],{},"✓ 最好",[380,861,862,865,868,871],{},[401,863,864],{},"共享 PVC + subPath（新架构）",[401,866,867],{},"管理简单（1 个 PVC）；自动扩展；成本低",[401,869,870],{},"故障隔离性差；无硬配额；权限管理复杂",[401,872,873],{},"✗ 较差",[380,875,876,879,882,885],{},[401,877,878],{},"对象存储（S3\u002FOSS）",[401,880,881],{},"真正的多租户隔离；天然分布",[401,883,884],{},"延迟高；成本更高；应用改造大",[401,886,887],{},"✓ 最好，但代价大",[10,889,890],{},"独立 PVC 的管理开销是关键问题。574 个用户意味着 574 个 PVC 对象、574 条 PV 绑定、574 个存储卷。每次实例创建、删除或迁移都要涉及 PVC 的生命周期管理。扩展到 5000 用户时，这种开销会成为瓶颈。对象存储虽然隔离性最好，但需要应用层改造（兼容 S3 API、处理延迟、调整备份策略），而且成本更高。",[10,892,893,894,897],{},"共享 PVC 方案的核心优势是",[223,895,896],{},"运维简洁","：增删用户只需改 subPath（一行 YAML），不涉及存储层操作。代价是故障隔离性从\"用户级\"降到\"节点级\"。",[20,899,900],{"id":900},"适用边界",[10,902,903],{},"这个方案适合以下场景：",[316,905,906,912,918,924],{},[220,907,908,911],{},[223,909,910],{},"用户数量有上限","（几百到几千）。用户过多时，共享卷的单点压力会成问题。",[220,913,914,917],{},[223,915,916],{},"用户数据量可控","（单用户通常 GB 级）。如果单用户数据量达 TB，一次 du 扫描会拖累整体。",[220,919,920,923],{},[223,921,922],{},"可以接受节点级故障隔离","。只要网络和存储配置足够稳定（硬化 NFS 参数、冗余链路），节点 hang 的概率不会很高。",[220,925,926,929],{},[223,927,928],{},"能够实施应用层监控和告警","。无硬配额，就必须有实时监控。",[10,931,932],{},"不适合的场景包括：",[316,934,935,938,941],{},[220,936,937],{},"超大规模用户（万级以上）",[220,939,940],{},"用户数据特别不均衡（少数用户占大头）的场景",[220,942,943],{},"对故障隔离要求极高的系统（比如金融交易）",[20,945,946],{"id":946},"防护与调整",[10,948,949],{},"基于这四个问题，实施中做了以下调整：",[217,951,952,961,971,977,983],{},[220,953,954,957,958,960],{},[223,955,956],{},"权限处理","：容器 Dockerfile 中预先创建 ",[35,959,694],{}," 并设置正确的属主；启动脚本检查并修复权限。",[220,962,963,966,967,970],{},[223,964,965],{},"NFS 参数硬化","：StorageClass 的挂载选项改为 ",[35,968,969],{},"soft,timeo=100,retrans=3,intr","，避免硬 mount 导致节点冻。",[220,972,973,976],{},[223,974,975],{},"节点健康检测","：cron 每 60 秒检查各节点的 ContainerCreating 堆积和网关探测失败率，及早发现故障。",[220,978,979,982],{},[223,980,981],{},"实例分散","：StatefulSet 加 topologySpreadConstraints，避免单节点堆积 50+ 个实例。",[220,984,985,988],{},[223,986,987],{},"应用层配额","：\u002Fstatus 接口返回 diskUsedMi 和 overQuota 标志位，前端告警用户。",[10,990,991],{},"这些措施不能消除风险，但可以大幅降低故障概率和影响范围。",[586,993,994],{},"html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":33,"searchDepth":132,"depth":132,"links":996},[997,998,999,1000,1001,1002,1003],{"id":630,"depth":132,"text":631},{"id":701,"depth":132,"text":702},{"id":728,"depth":132,"text":729},{"id":787,"depth":132,"text":788},{"id":822,"depth":132,"text":822},{"id":900,"depth":132,"text":900},{"id":946,"depth":132,"text":946},"2026-07-23",{},"\u002F2026-07-23-fa-018-subpath",{"title":619,"description":624},"FA-018","2026-07-23-FA-018-共享存储的subPath陷阱","K8s 共享 PVC + subPath 方案在实施中暴露的四个陷阱：权限不匹配、目录创建时机、节点级故障域、无 per-subPath 配额。",[1012,1013,1014,1015,1016,1017,843],"Kubernetes","存储","subPath","PVC","NFS","权限","4pJJyzbII4OOc9jJS74HUpUfyVc9CfxdE5ovso8ylSE",{"id":1020,"title":1021,"body":1022,"column":597,"date":1461,"description":1462,"extension":599,"hero_image":600,"meta":1463,"navigation":602,"path":1464,"seo":1465,"series_id":1466,"severity":600,"stem":1467,"summary":1468,"tags":1469,"__hash__":1476},"posts\u002F2026-07-11-FA-017-长任务异步化.md","三个超时机制，一起把长任务掐死了",{"type":7,"value":1023,"toc":1453},[1024,1035,1038,1041,1047,1053,1059,1066,1072,1076,1079,1082,1088,1091,1102,1108,1112,1115,1122,1128,1131,1146,1150,1153,1164,1213,1228,1235,1239,1242,1245,1263,1266,1274,1277,1292,1295,1298,1444,1447,1450],[10,1025,1026,1027,1030,1031,1034],{},"数字人口播的「分析」功能从视频提取口播文稿、分镜脚本、结构卖点，需要下载视频、压缩、调 Gemini 视觉 API，耗时从几十秒到几分钟。最初同步实现：前端调 ",[35,1028,1029],{},"POST \u002Fapi\u002Fworkflow\u002Fdub\u002Fanalyze-parsed","，后端直接跑完再返回。这个方案暴露的问题不是单点，而是",[223,1032,1033],{},"三个独立的缺陷叠加","。",[10,1036,1037],{},"网关超时、CDN 空闲切断、客户端重试——这三个没有一个是\"bug\"，每个都是合理的系统设计。但组合在一起就是连锁灾难：客户端看不到结果（超时或连接断），认为失败了重试，后端却已经扣过费了；或者分析其实已跑完，但因为响应发不出去，重试又跑了一遍。退款时无法幂等判断，某些用户最终被多扣了好几倍。",[10,1039,1040],{},"具体来说，三个问题各自是什么：",[10,1042,1043,1046],{},[223,1044,1045],{},"第一个问题是网关超时","。云平台的入口网关（API Gateway）有一个固定的读超时，通常设为几十秒。当后端分析耗时超过这个阈值，网关主动断连接，客户端收到 504 或连接重置，但后端的分析进程并不知道客户端已经走了，继续跑完整个流程。如果分析结果的退款逻辑放在 try-catch 的 finally 里，此时 HTTP 响应已发不出去，客户就看到\"分析失败\"，但款已经扣了。",[10,1048,1049,1052],{},[223,1050,1051],{},"第二个问题是 CDN 空闲切断","。不少 CDN 和负载均衡层有这样的策略：如果一条 HTTP 连接在某个时间段内没有数据往返，就认为它已死而主动关闭。这与 FA-013 那次踩到的问题一样——长连接因为没有心跳数据而被 CDN 中间件切断，导致请求被突然中断。这个机制在分析请求上也会造成麻烦：如果分析用时较长，连接就可能被切，响应发不出去。",[10,1054,1055,1058],{},[223,1056,1057],{},"第三个问题是客户端重试导致重复扣费","。当客户端没有收到响应（网关超时或 CDN 切断），一般会重试这个请求。问题在于这次重试是一条完全独立的请求，后端会把它当作新任务处理，再次扣费。而且如果第一次的分析实际上已经跑完了，就会出现\"分析跑了两遍、扣费扣了两遍、用户只看到了一个结果\"的情况。如果第一次分析中途被 CDN 切了，第二次重试重新开始，那扣费可能是三倍。而退款的时候，因为业务流程上没有幂等设计，很难追溯哪个 operationId 应该被退，哪个不应该。",[10,1060,1061,1062,1065],{},"这三个问题是",[223,1063,1064],{},"叠加的","，不是独立的。它们共同指向一个根本的架构问题。",[10,1067,1068,1069,1034],{},"根本原因一个：",[223,1070,1071],{},"不该让耗时任务挂在 HTTP 连接上",[20,1073,1075],{"id":1074},"异步任务是必需的不是可选的","异步任务是必需的，不是可选的",[10,1077,1078],{},"解决方案很简单：把分析改成异步任务模式。",[10,1080,1081],{},"前端提交分析请求直接返回任务 ID（HTTP 耗时极短：校验参数、建数据库记录、预扣费）。后端后台 worker 跑分析重逻辑，完成后持久化结果。前端轮询查询状态，得到结果后停止。",[28,1083,1086],{"className":1084,"code":1085,"language":80},[78],"客户端              后端\n  |                 |\n  +--提交---------->|\n  |             创建任务，返回 ID\n  |\u003C--返回 ID-----+\n  |                 |（后台运行分析）\n  |                 |\n  +--轮询--------->|\n  |             查询状态\n  |\u003C--返回状态---+\n  |                 |\n  |（重复轮询）\n  +--轮询--------->|\n  |             查询状态\n  |\u003C--返回结果---+\n  |                 |\n",[35,1087,1085],{"__ignoreMap":33},[10,1089,1090],{},"好处显而易见：",[316,1092,1093,1096,1099],{},[220,1094,1095],{},"单次 HTTP 请求的耗时从分钟级降到秒级，不会触发网关或 CDN 的超时。",[220,1097,1098],{},"后端分析的运行生命周期与 HTTP 连接完全解耦。即使连接在中途被切，已经启动的分析会继续在后台跑，结果最终还是会被保存。",[220,1100,1101],{},"客户端重试不会产生新的分析任务。只要实现幂等判重，同一个请求在短时间内重复提交也只会产生一个任务。",[10,1103,1104,1105,1034],{},"但异步化不是万灵药，它把问题转移到了三个新的维度上：",[223,1106,1107],{},"幂等、持久化、失败语义",[20,1109,1111],{"id":1110},"幂等同一请求只产生一个任务","幂等：同一请求只产生一个任务",[10,1113,1114],{},"假设客户端因为网络抖动，在短时间内连续发了两次\"开始分析\"请求，后端收到两条独立的 HTTP 请求。如果没有幂等设计，就会创建两个任务、扣两次费。",[10,1116,1117,1118,1121],{},"幂等的实现最直接的办法是在",[223,1119,1120],{},"提交阶段就判重","：为这个用户、这个资源设置一个\"只有一个进行中的分析任务\"的约束。素材中的设计是这样做的：",[28,1123,1126],{"className":1124,"code":1125,"language":80},[78],"if 该用户已有 status=running && kind=analyze_parsed 的任务:\n    return 409 Conflict（分析已在进行中）\n",[35,1127,1125],{"__ignoreMap":33},[10,1129,1130],{},"这样双击提交同一个视频，第一次返回 201 + taskId，第二次返回 409，客户端看到 409 就知道不要再建新任务，拿之前返回的 ID 去轮询。",[10,1132,1133,1134,1137,1138,1141,1142,1145],{},"另一个幂等层次是",[223,1135,1136],{},"计费操作的幂等","。预扣费的时候用一个全局唯一的 ",[35,1139,1140],{},"operationId","（比如 ",[35,1143,1144],{},"dub-analyze:{uuid}","），这个 ID 关联了这次预扣。如果扣费 API 被重复调用，因为 operationId 重复，系统只会扣一次。退款时也用同一个 operationId，保证退款与预扣能正确配对。",[20,1147,1149],{"id":1148},"持久化进程重启后任务不丢","持久化：进程重启后任务不丢",[10,1151,1152],{},"后台分析 worker 可能因为发版、机器重启、容器被杀等原因在中途退出。如果任务的状态只保存在内存里，进程一死任务就丢了。",[10,1154,1155,1156,1159,1160,1163],{},"持久化的关键是",[223,1157,1158],{},"任务状态表","。素材中复用了既有的 ",[35,1161,1162],{},"SkyhumanTask"," 表，包含字段：",[316,1165,1166,1172,1178,1183,1189,1195,1201,1207],{},[220,1167,1168,1171],{},[35,1169,1170],{},"id",": 任务 ID",[220,1173,1174,1177],{},[35,1175,1176],{},"status",": 进度（running \u002F completed \u002F failed）",[220,1179,1180,1182],{},[35,1181,1140],{},": 幂等 ID",[220,1184,1185,1188],{},[35,1186,1187],{},"chargedPoints",": 预扣费用",[220,1190,1191,1194],{},[35,1192,1193],{},"resultPayload",": 分析结果的 JSON",[220,1196,1197,1200],{},[35,1198,1199],{},"error",": 失败原因",[220,1202,1203,1206],{},[35,1204,1205],{},"updatedAt",": 最后更新时间戳",[220,1208,1209,1212],{},[35,1210,1211],{},"sourceObjectKey",": 源视频对象路径（用于幂等判重和重试）",[10,1214,1215,1216,1219,1220,1223,1224,1227],{},"任务创建时插一条 ",[35,1217,1218],{},"status=running"," 的记录。后台 worker 定期通过 heartbeat（每 30 秒做一次空 update，仅触碰 updatedAt）来证明自己还活着。分析完成时更新为 ",[35,1221,1222],{},"status=completed, resultPayload=...","；失败时更新为 ",[35,1225,1226],{},"status=failed, error=..."," 并触发退款。",[10,1229,1230,1231,1234],{},"这样即使 worker 进程突然死了，任务记录还在数据库里。新启起的 worker 可以扫描表里的 ",[35,1232,1233],{},"status=running && updatedAt 过期"," 的任务，判断它们已经没有 worker 在跑了，执行清理逻辑。",[20,1236,1238],{"id":1237},"失败语义区分任务失败和查询失败","失败语义：区分\"任务失败\"和\"查询失败\"",[10,1240,1241],{},"异步设计引入了一个新的错误维度：客户端查询任务状态时得到的 404，可能是\"这个任务 ID 不存在\"（用户输错了），也可能是\"任务 ID 之前存在但已被清理\"（进程清理了过期数据）。这两种情况的含义很不一样。",[10,1243,1244],{},"更重要的是，前端轮询收到的错误需要区分是否应该重试：",[316,1246,1247,1257],{},[220,1248,1249,1252,1253,1256],{},[223,1250,1251],{},"任务本身失败","（",[35,1254,1255],{},"status=failed, error=\"Gemini API 返回错误\"","）：这种情况不该重试，因为重试也会失败。应该直接向用户展示失败信息，并根据 operationId 触发退款。",[220,1258,1259,1262],{},[223,1260,1261],{},"查询接口失败","（500、网络超时）：这种情况应该重试查询，因为任务本身可能还在继续跑。",[10,1264,1265],{},"素材中的设计把这两类区分得很清楚：",[316,1267,1268,1271],{},[220,1269,1270],{},"任务最终的状态（completed 或 failed）持久化到数据库，客户端查询会拿到明确的业务语义。",[220,1272,1273],{},"查询接口的 HTTP 错误（5xx）是技术层故障，客户端应该重试。",[10,1275,1276],{},"同时，后端的 reaper 机制（定期扫表清理超期任务）也遵循这个语义：",[316,1278,1279,1285],{},[220,1280,1281,1282,1284],{},"如果一个 ",[35,1283,1218],{}," 的任务超过 90 秒没有 heartbeat 更新，说明 worker 已死，reaper 会强制置为 failed 并退款。",[220,1286,1287,1288,1291],{},"但这个强制置为 failed 不应该抛错，因为此时可能有其他进程也想更新这个任务。更新语句应该带条件：",[35,1289,1290],{},"UPDATE SkyhumanTask SET status=failed WHERE id=? AND status=running","，这样如果 reaper 和其他 worker 同时操作，只有一方会成功。",[20,1293,1294],{"id":1294},"具体实现模式",[10,1296,1297],{},"数字人口播的分析异步化采用了这样的模式：",[217,1299,1300,1345,1391,1414],{},[220,1301,1302,1305,1306],{},[223,1303,1304],{},"提交阶段","（同步）：",[316,1307,1308,1311,1314,1320,1326,1333,1339],{},[220,1309,1310],{},"校验用户和资源",[220,1312,1313],{},"判重：是否已有进行中的任务（409）",[220,1315,1316,1319],{},[35,1317,1318],{},"getObject"," 拉视频，检测时长（若≤0 返 400 不扣费）",[220,1321,1322,1325],{},[35,1323,1324],{},"chargeResource"," 预扣费用，获得 operationId",[220,1327,1328,1329,1332],{},"创建 ",[35,1330,1331],{},"SkyhumanTask{ status:running, operationId, sourceObjectKey }"," 记录",[220,1334,1335,1338],{},[35,1336,1337],{},"scheduleTask"," 丢到后台队列",[220,1340,1341,1342],{},"返回 ",[35,1343,1344],{},"{ taskId }",[220,1346,1347,104,1350],{},[223,1348,1349],{},"后台运行阶段",[316,1351,1352,1355,1358,1365,1371,1381,1388],{},[220,1353,1354],{},"Worker 取出任务记录",[220,1356,1357],{},"开启 heartbeat 定时器（每 30s update 一次 updatedAt）",[220,1359,1360,1361,1364],{},"执行 ",[35,1362,1363],{},"analyzeDubVisionOnly","（下载→压缩→调用 Gemini→解析）",[220,1366,1367,1368],{},"成功：",[35,1369,1370],{},"update( status:completed, resultPayload )",[220,1372,1373,1374,1377,1378],{},"失败：",[35,1375,1376],{},"update( status:failed, error )"," + ",[35,1379,1380],{},"refundResource(operationId)",[220,1382,1383,1384,1387],{},"所有 update 都带条件 ",[35,1385,1386],{},"WHERE id=? AND status=running","（防被 reaper 覆盖）",[220,1389,1390],{},"Finally 清理 heartbeat 定时器",[220,1392,1393,1396,1397],{},[223,1394,1395],{},"查询阶段","（前端轮询）：",[316,1398,1399,1408,1411],{},[220,1400,1401,1404,1405],{},[35,1402,1403],{},"GET \u002Fapi\u002Fworkflow\u002Fdub\u002Ftasks\u002F:id"," 返回 ",[35,1406,1407],{},"{ status, resultPayload, error }",[220,1409,1410],{},"若 status=completed 或 failed，停止轮询",[220,1412,1413],{},"若查询接口返回 5xx，则后续重试",[220,1415,1416,1419,1420],{},[223,1417,1418],{},"清理阶段","（reaper）：",[316,1421,1422,1425,1432,1435,1441],{},[220,1423,1424],{},"后台定期扫表",[220,1426,1427,1428,1431],{},"对于 ",[35,1429,1430],{},"status=running && updatedAt > 90s"," 的任务，判定 worker 已死",[220,1433,1434],{},"若 providerTaskId（这里没有）存在，调用上游的 finalize API",[220,1436,1437,1438,1440],{},"若不存在，直接 ",[35,1439,1380],{}," + 置 failed",[220,1442,1443],{},"本设计无 providerTaskId，所以 reaper 自动走退款分支，无需改动 reaper 代码",[10,1445,1446],{},"这个模式的关键是三道防线：提交时判重（409）、后台心跳（定期 touch）、reaper 兜底（自动退款）。任何一个环节出问题，都有后续环节来补救。",[20,1448,1449],{"id":1449},"防线缺一不可",[10,1451,1452],{},"长耗时任务不能挂在 HTTP 连接上不只是超时问题，而是连接脆弱性与重复处理的共谋。异步化表面是\"返回 ID、后台跑\"这一步，真正的难点在三个维度的同时保证：幂等（不重复扣费）、持久化（不丢数据）、失败语义（不破坏重试逻辑）。缺一就会在某个场景暴露。",{"title":33,"searchDepth":132,"depth":132,"links":1454},[1455,1456,1457,1458,1459,1460],{"id":1074,"depth":132,"text":1075},{"id":1110,"depth":132,"text":1111},{"id":1148,"depth":132,"text":1149},{"id":1237,"depth":132,"text":1238},{"id":1294,"depth":132,"text":1294},{"id":1449,"depth":132,"text":1449},"2026-07-11","数字人口播的「分析」功能从视频提取口播文稿、分镜脚本、结构卖点，需要下载视频、压缩、调 Gemini 视觉 API，耗时从几十秒到几分钟。最初同步实现：前端调 POST \u002Fapi\u002Fworkflow\u002Fdub\u002Fanalyze-parsed，后端直接跑完再返回。这个方案暴露的问题不是单点，而是三个独立的缺陷叠加。",{},"\u002F2026-07-11-fa-017",{"title":1021,"description":1462},"FA-017","2026-07-11-FA-017-长任务异步化","长耗时分析请求（几十秒到分钟级）用同步 HTTP 导致网关超时、CDN切断、客户端重试重复扣费。改异步任务需同时解决幂等、持久化、失败语义。",[1470,1471,1472,1473,1474,1475],"异步任务","HTTP超时","CDN","幂等","任务队列","微服务","ARFDvBMGX3tw-E3Rj_gw1RYEjrykplO6u1dxtdS7Uto",{"id":1478,"title":1479,"body":1480,"column":597,"date":2393,"description":2394,"extension":599,"hero_image":600,"meta":2395,"navigation":602,"path":2396,"seo":2397,"series_id":2398,"severity":600,"stem":2399,"summary":2400,"tags":2401,"__hash__":2406},"posts\u002F2026-06-17-FA-016-WS永久断开三条路径.md","日志里没有重连记录——它根本没在重连",{"type":7,"value":1481,"toc":2381},[1482,1492,1499,1506,1509,1512,1517,1524,1599,1605,1672,1683,1687,1690,1814,1825,1829,1835,1890,1893,1896,1899,1914,1917,1920,1923,1930,2188,2198,2201,2211,2224,2234,2237,2240,2247,2298,2301,2304,2307,2310,2313,2358,2375,2378],[10,1483,1484,1485,1488,1489,1491],{},"从 2026-05-11 开始，多台便携包客户反馈 WebSocket 断线后面板卡死。网关进程（",[35,1486,1487],{},"openclaw_gateway.exe","）明确在运行，",[35,1490,303],{}," 返回 200，但面板显示\"已停止重连，请手动刷新\"——用户必须手动重启才能恢复。",[10,1493,1494,1495,1498],{},"关键线索在控制台：看不到任何 ",[35,1496,1497],{},"[ws] 计划重连"," 的日志。",[10,1500,1501,1502,1505],{},"通常断线后客户端会频繁尝试重连，日志里应该是满屏的重连记录。没有日志意味着问题不在重连失败，而在",[223,1503,1504],{},"压根没启动重连机制","。说明客户端进入了某个永久休眠状态。",[20,1507,1508],{"id":1508},"三条独立的永久断开路径",[10,1510,1511],{},"代码审查发现了三条完全不同的、独立的永久断开路径。任意一条命中，就会停止调度任何重连计时器。",[1513,1514,1516],"h3",{"id":1515},"路径-a凭据刷新异常","路径 A：凭据刷新异常",[10,1518,1519,1520,1523],{},"在 ",[35,1521,1522],{},"_scheduleReconnect"," 方法末尾，每次普通重连都会调用：",[28,1525,1529],{"className":1526,"code":1527,"language":1528,"meta":33,"style":33},"language-javascript shiki shiki-themes github-light github-dark","this._reconnectTimer = setTimeout(() => {\n  if (!this._intentionalClose) {\n    this._refreshCredentialsAndReconnect(0)\n  }\n}, delay)\n","javascript",[35,1530,1531,1554,1570,1588,1593],{"__ignoreMap":33},[38,1532,1533,1536,1539,1542,1545,1548,1551],{"class":40,"line":41},[38,1534,1535],{"class":52},"this",[38,1537,1538],{"class":119},"._reconnectTimer ",[38,1540,1541],{"class":56},"=",[38,1543,1544],{"class":44}," setTimeout",[38,1546,1547],{"class":119},"(() ",[38,1549,1550],{"class":56},"=>",[38,1552,1553],{"class":119}," {\n",[38,1555,1556,1559,1562,1565,1567],{"class":40,"line":132},[38,1557,1558],{"class":56},"  if",[38,1560,1561],{"class":119}," (",[38,1563,1564],{"class":56},"!",[38,1566,1535],{"class":52},[38,1568,1569],{"class":119},"._intentionalClose) {\n",[38,1571,1572,1575,1577,1580,1583,1585],{"class":40,"line":156},[38,1573,1574],{"class":52},"    this",[38,1576,177],{"class":119},[38,1578,1579],{"class":44},"_refreshCredentialsAndReconnect",[38,1581,1582],{"class":119},"(",[38,1584,411],{"class":52},[38,1586,1587],{"class":119},")\n",[38,1589,1590],{"class":40,"line":189},[38,1591,1592],{"class":119},"  }\n",[38,1594,1596],{"class":40,"line":1595},5,[38,1597,1598],{"class":119},"}, delay)\n",[10,1600,1601,1602,1604],{},"而 ",[35,1603,1579],{}," 是异步方法，catch 分支没有任何后续处理：",[28,1606,1608],{"className":1526,"code":1607,"language":1528,"meta":33,"style":33},"} catch (e) {\n  console.error('[ws] 刷新凭据失败:', e)\n  this._setConnected(false, 'error', `凭据刷新失败: ${e}`)\n}\n",[35,1609,1610,1621,1636,1668],{"__ignoreMap":33},[38,1611,1612,1615,1618],{"class":40,"line":41},[38,1613,1614],{"class":119},"} ",[38,1616,1617],{"class":56},"catch",[38,1619,1620],{"class":119}," (e) {\n",[38,1622,1623,1626,1628,1630,1633],{"class":40,"line":132},[38,1624,1625],{"class":119},"  console.",[38,1627,1199],{"class":44},[38,1629,1582],{"class":119},[38,1631,1632],{"class":48},"'[ws] 刷新凭据失败:'",[38,1634,1635],{"class":119},", e)\n",[38,1637,1638,1641,1643,1646,1648,1651,1654,1656,1658,1661,1664,1666],{"class":40,"line":156},[38,1639,1640],{"class":52},"  this",[38,1642,177],{"class":119},[38,1644,1645],{"class":44},"_setConnected",[38,1647,1582],{"class":119},[38,1649,1650],{"class":52},"false",[38,1652,1653],{"class":119},", ",[38,1655,165],{"class":48},[38,1657,1653],{"class":119},[38,1659,1660],{"class":48},"`凭据刷新失败: ${",[38,1662,1663],{"class":119},"e",[38,1665,183],{"class":48},[38,1667,1587],{"class":119},[38,1669,1670],{"class":40,"line":189},[38,1671,192],{"class":119},[10,1673,1674,1675,1678,1679,1682],{},"如果 ",[35,1676,1677],{},"api.readOpenclawConfig()"," 或 ",[35,1680,1681],{},"api.autoPairDevice()"," 抛错——磁盘 IO 抖、Rust 端处理器忙、Tauri IPC 队列阻塞——这次重连就永久终止。便携磁盘的 IO 抖动 + 配置文件读写争抢时最容易发生。",[1513,1684,1686],{"id":1685},"路径-b认证失败后的强制关闭","路径 B：认证失败后的强制关闭",[10,1688,1689],{},"当 WebSocket 收到 1008 unauthorized 响应时的处理逻辑：",[28,1691,1693],{"className":1526,"code":1692,"language":1528,"meta":33,"style":33},"if (this._authRetryCount \u003C 2) {\n  this._authRetryCount++\n  this._refreshCredentialsAndReconnect()\n  return\n}\nthis._setConnected(false, 'auth_failed', `认证失败: ${e.reason}。请检查 Gateway Token 配置。`)\nthis._intentionalClose = true   \u002F\u002F ← 永久关闭标记\nthis._flushPending()\nreturn\n",[35,1694,1695,1714,1724,1735,1740,1744,1779,1796,1808],{"__ignoreMap":33},[38,1696,1697,1699,1701,1703,1706,1709,1712],{"class":40,"line":41},[38,1698,116],{"class":56},[38,1700,1561],{"class":119},[38,1702,1535],{"class":52},[38,1704,1705],{"class":119},"._authRetryCount ",[38,1707,1708],{"class":56},"\u003C",[38,1710,1711],{"class":52}," 2",[38,1713,129],{"class":119},[38,1715,1716,1718,1721],{"class":40,"line":132},[38,1717,1640],{"class":52},[38,1719,1720],{"class":119},"._authRetryCount",[38,1722,1723],{"class":56},"++\n",[38,1725,1726,1728,1730,1732],{"class":40,"line":156},[38,1727,1640],{"class":52},[38,1729,177],{"class":119},[38,1731,1579],{"class":44},[38,1733,1734],{"class":119},"()\n",[38,1736,1737],{"class":40,"line":189},[38,1738,1739],{"class":56},"  return\n",[38,1741,1742],{"class":40,"line":1595},[38,1743,192],{"class":119},[38,1745,1747,1749,1751,1753,1755,1757,1759,1762,1764,1767,1769,1771,1774,1777],{"class":40,"line":1746},6,[38,1748,1535],{"class":52},[38,1750,177],{"class":119},[38,1752,1645],{"class":44},[38,1754,1582],{"class":119},[38,1756,1650],{"class":52},[38,1758,1653],{"class":119},[38,1760,1761],{"class":48},"'auth_failed'",[38,1763,1653],{"class":119},[38,1765,1766],{"class":48},"`认证失败: ${",[38,1768,1663],{"class":119},[38,1770,177],{"class":48},[38,1772,1773],{"class":119},"reason",[38,1775,1776],{"class":48},"}。请检查 Gateway Token 配置。`",[38,1778,1587],{"class":119},[38,1780,1782,1784,1787,1789,1792],{"class":40,"line":1781},7,[38,1783,1535],{"class":52},[38,1785,1786],{"class":119},"._intentionalClose ",[38,1788,1541],{"class":56},[38,1790,1791],{"class":52}," true",[38,1793,1795],{"class":1794},"sJ8bj","   \u002F\u002F ← 永久关闭标记\n",[38,1797,1799,1801,1803,1806],{"class":40,"line":1798},8,[38,1800,1535],{"class":52},[38,1802,177],{"class":119},[38,1804,1805],{"class":44},"_flushPending",[38,1807,1734],{"class":119},[38,1809,1811],{"class":40,"line":1810},9,[38,1812,1813],{"class":56},"return\n",[10,1815,1816,1817,1820,1821,1824],{},"设置 ",[35,1818,1819],{},"_intentionalClose = true"," 之后，所有重连逻辑都被 ",[35,1822,1823],{},"if (!this._intentionalClose)"," 短路。即使只是 Gateway 重启窗口碰好出现 token 短暂不匹配，也会一次性把客户端打死，必须刷新页面才能恢复。",[1513,1826,1828],{"id":1827},"路径-c快速重连配额耗尽","路径 C：快速重连配额耗尽",[10,1830,1831,1832,104],{},"定义了常数 ",[35,1833,1834],{},"MAX_RECONNECT_ATTEMPTS = 60",[28,1836,1838],{"className":1526,"code":1837,"language":1528,"meta":33,"style":33},"if (this._reconnectAttempts >= MAX_RECONNECT_ATTEMPTS) {\n  this._setConnected(false, 'error', `连接失败，已停止重连。请手动刷新页面重试。`)\n  return\n}\n",[35,1839,1840,1859,1882,1886],{"__ignoreMap":33},[38,1841,1842,1844,1846,1848,1851,1854,1857],{"class":40,"line":41},[38,1843,116],{"class":56},[38,1845,1561],{"class":119},[38,1847,1535],{"class":52},[38,1849,1850],{"class":119},"._reconnectAttempts ",[38,1852,1853],{"class":56},">=",[38,1855,1856],{"class":52}," MAX_RECONNECT_ATTEMPTS",[38,1858,129],{"class":119},[38,1860,1861,1863,1865,1867,1869,1871,1873,1875,1877,1880],{"class":40,"line":132},[38,1862,1640],{"class":52},[38,1864,177],{"class":119},[38,1866,1645],{"class":44},[38,1868,1582],{"class":119},[38,1870,1650],{"class":52},[38,1872,1653],{"class":119},[38,1874,165],{"class":48},[38,1876,1653],{"class":119},[38,1878,1879],{"class":48},"`连接失败，已停止重连。请手动刷新页面重试。`",[38,1881,1587],{"class":119},[38,1883,1884],{"class":40,"line":156},[38,1885,1739],{"class":56},[38,1887,1888],{"class":40,"line":189},[38,1889,192],{"class":119},[10,1891,1892],{},"客户机晚上挂机，Gateway 因为便携磁盘 GC 或 Windows 休眠争抢资源短暂掉线，客户端尝试 60 次仍未成功重连，就永久停摆。早上用户回来面板已成死链。",[20,1894,1895],{"id":1895},"为什么三条缺陷同时存在",[10,1897,1898],{},"这是逐步累加的历史债：",[316,1900,1901,1908,1911],{},[220,1902,1903,1904,1907],{},"路径 B 是在修复\"避免无限自动配对循环\"时添加的保护，但用错了对象——",[35,1905,1906],{},"_intentionalClose=true"," 本来是用户主动断开的语义，不该用在被动失败上",[220,1909,1910],{},"路径 C 是早期\"避免无穷重试\"的防护，但 60 次后完全放弃而不留任何复活路径是绝对错误",[220,1912,1913],{},"路径 A 是在\"凭据 reload\"改动时把所有重连都改走 refreshCredentials，没留意 catch 分支已经成了终态",[10,1915,1916],{},"三条各自独立、都能单独打死客户端，组合在一起命中率非常高。",[20,1918,1919],{"id":1919},"修复方案",[10,1921,1922],{},"核心思想是永不彻底放弃。任何\"快速重连配额耗尽\"的分支都转入慢轮询，留出窗口让用户改配置或等待 Gateway 恢复后能自动复连。",[10,1924,1925,1926,1929],{},"新增辅助方法 ",[35,1927,1928],{},"_schedulePoll(delayMs, kind)"," 作为慢轮询的触发器：",[28,1931,1933],{"className":1526,"code":1932,"language":1528,"meta":33,"style":33},"const AUTH_RETRY_LIMIT = 2\nconst SLOW_POLL_DELAY_AUTH = 60_000      \u002F\u002F 认证持续失败：1 分钟探一次\nconst SLOW_POLL_DELAY_GENERAL = 300_000  \u002F\u002F 一般持续失败：5 分钟探一次\n\n_schedulePoll(delayMs, kind) {\n  this._clearReconnectTimer()\n  this._reconnectAttempts = 0\n  if (kind === 'auth') this._authRetryCount = 0\n  this._reconnectState = 'scheduled'\n  this._pendingReconnect = true\n  this._reconnectTimer = setTimeout(() => {\n    this._reconnectTimer = null\n    if (this._intentionalClose) return\n    this._reconnectState = 'attempting'\n    if (kind === 'auth') {\n      this._refreshCredentialsAndReconnect(0)\n    } else {\n      this._doConnect()\n    }\n  }, delayMs)\n}\n",[35,1934,1935,1949,1964,1979,1984,1992,2003,2014,2038,2050,2063,2080,2092,2107,2119,2132,2148,2159,2171,2177,2183],{"__ignoreMap":33},[38,1936,1937,1940,1943,1946],{"class":40,"line":41},[38,1938,1939],{"class":56},"const",[38,1941,1942],{"class":52}," AUTH_RETRY_LIMIT",[38,1944,1945],{"class":56}," =",[38,1947,1948],{"class":52}," 2\n",[38,1950,1951,1953,1956,1958,1961],{"class":40,"line":132},[38,1952,1939],{"class":56},[38,1954,1955],{"class":52}," SLOW_POLL_DELAY_AUTH",[38,1957,1945],{"class":56},[38,1959,1960],{"class":52}," 60_000",[38,1962,1963],{"class":1794},"      \u002F\u002F 认证持续失败：1 分钟探一次\n",[38,1965,1966,1968,1971,1973,1976],{"class":40,"line":156},[38,1967,1939],{"class":56},[38,1969,1970],{"class":52}," SLOW_POLL_DELAY_GENERAL",[38,1972,1945],{"class":56},[38,1974,1975],{"class":52}," 300_000",[38,1977,1978],{"class":1794},"  \u002F\u002F 一般持续失败：5 分钟探一次\n",[38,1980,1981],{"class":40,"line":189},[38,1982,1983],{"emptyLinePlaceholder":602},"\n",[38,1985,1986,1989],{"class":40,"line":1595},[38,1987,1988],{"class":44},"_schedulePoll",[38,1990,1991],{"class":119},"(delayMs, kind) {\n",[38,1993,1994,1996,1998,2001],{"class":40,"line":1746},[38,1995,1640],{"class":52},[38,1997,177],{"class":119},[38,1999,2000],{"class":44},"_clearReconnectTimer",[38,2002,1734],{"class":119},[38,2004,2005,2007,2009,2011],{"class":40,"line":1781},[38,2006,1640],{"class":52},[38,2008,1850],{"class":119},[38,2010,1541],{"class":56},[38,2012,2013],{"class":52}," 0\n",[38,2015,2016,2018,2021,2024,2027,2030,2032,2034,2036],{"class":40,"line":1798},[38,2017,1558],{"class":56},[38,2019,2020],{"class":119}," (kind ",[38,2022,2023],{"class":56},"===",[38,2025,2026],{"class":48}," 'auth'",[38,2028,2029],{"class":119},") ",[38,2031,1535],{"class":52},[38,2033,1705],{"class":119},[38,2035,1541],{"class":56},[38,2037,2013],{"class":52},[38,2039,2040,2042,2045,2047],{"class":40,"line":1810},[38,2041,1640],{"class":52},[38,2043,2044],{"class":119},"._reconnectState ",[38,2046,1541],{"class":56},[38,2048,2049],{"class":48}," 'scheduled'\n",[38,2051,2053,2055,2058,2060],{"class":40,"line":2052},10,[38,2054,1640],{"class":52},[38,2056,2057],{"class":119},"._pendingReconnect ",[38,2059,1541],{"class":56},[38,2061,2062],{"class":52}," true\n",[38,2064,2066,2068,2070,2072,2074,2076,2078],{"class":40,"line":2065},11,[38,2067,1640],{"class":52},[38,2069,1538],{"class":119},[38,2071,1541],{"class":56},[38,2073,1544],{"class":44},[38,2075,1547],{"class":119},[38,2077,1550],{"class":56},[38,2079,1553],{"class":119},[38,2081,2083,2085,2087,2089],{"class":40,"line":2082},12,[38,2084,1574],{"class":52},[38,2086,1538],{"class":119},[38,2088,1541],{"class":56},[38,2090,2091],{"class":52}," null\n",[38,2093,2095,2098,2100,2102,2105],{"class":40,"line":2094},13,[38,2096,2097],{"class":56},"    if",[38,2099,1561],{"class":119},[38,2101,1535],{"class":52},[38,2103,2104],{"class":119},"._intentionalClose) ",[38,2106,1813],{"class":56},[38,2108,2110,2112,2114,2116],{"class":40,"line":2109},14,[38,2111,1574],{"class":52},[38,2113,2044],{"class":119},[38,2115,1541],{"class":56},[38,2117,2118],{"class":48}," 'attempting'\n",[38,2120,2122,2124,2126,2128,2130],{"class":40,"line":2121},15,[38,2123,2097],{"class":56},[38,2125,2020],{"class":119},[38,2127,2023],{"class":56},[38,2129,2026],{"class":48},[38,2131,129],{"class":119},[38,2133,2135,2138,2140,2142,2144,2146],{"class":40,"line":2134},16,[38,2136,2137],{"class":52},"      this",[38,2139,177],{"class":119},[38,2141,1579],{"class":44},[38,2143,1582],{"class":119},[38,2145,411],{"class":52},[38,2147,1587],{"class":119},[38,2149,2151,2154,2157],{"class":40,"line":2150},17,[38,2152,2153],{"class":119},"    } ",[38,2155,2156],{"class":56},"else",[38,2158,1553],{"class":119},[38,2160,2162,2164,2166,2169],{"class":40,"line":2161},18,[38,2163,2137],{"class":52},[38,2165,177],{"class":119},[38,2167,2168],{"class":44},"_doConnect",[38,2170,1734],{"class":119},[38,2172,2174],{"class":40,"line":2173},19,[38,2175,2176],{"class":119},"    }\n",[38,2178,2180],{"class":40,"line":2179},20,[38,2181,2182],{"class":119},"  }, delayMs)\n",[38,2184,2186],{"class":40,"line":2185},21,[38,2187,192],{"class":119},[10,2189,2190,2191,2193,2194,2197],{},"慢轮询命中后失败会重新进入 ",[35,2192,1522],{}," 快速重连周期（因为 ",[35,2195,2196],{},"_reconnectAttempts"," 重置为 0），相当于\"快速 60 次 → 慢一次 → 快速 60 次 → 慢一次\"的循环，永不放弃。",[10,2199,2200],{},"三条修复并行：",[10,2202,2203,2206,2207,2210],{},[223,2204,2205],{},"Fix A","：普通重连直接走 ",[35,2208,2209],{},"_doConnect()","，凭据刷新隔离到认证失败分支，避免配置读取错误牵连普通重连。",[10,2212,2213,2216,2217,2220,2221,2223],{},[223,2214,2215],{},"Fix B","：认证失败耗尽后转入 ",[35,2218,2219],{},"_schedulePoll('auth')","，不再设置 ",[35,2222,1906],{},"，给认证恢复留出 60 秒的探测窗口。",[10,2225,2226,2229,2230,2233],{},[223,2227,2228],{},"Fix C","：重连次数超过 MAX_RECONNECT_ATTEMPTS 后转入 ",[35,2231,2232],{},"_schedulePoll('general')","，设置 UI 状态为\"连接持续失败，300 秒后重试\"而不是终态。",[10,2235,2236],{},"慢轮询失败后重新进入快速重连周期，形成\"快速 60 次 → 慢一次 → 快速 60 次\"的循环，永不放弃。",[20,2238,2239],{"id":2239},"防回归验证",[10,2241,2242,2243,2246],{},"新增 ",[35,2244,2245],{},"wakou-full\u002Fsrc\u002Flib\u002Fws-client.slow-poll.test.js","，覆盖 7 个 case：",[316,2248,2249,2257,2269,2272,2281,2288,2293],{},[220,2250,2251,2252,2254,2255],{},"Fix A：普通重连命中 ",[35,2253,2168],{},"，不调用 ",[35,2256,1579],{},[220,2258,2259,2260,2263,2264,2266,2267],{},"Fix C：MAX_RECONNECT_ATTEMPTS 后状态是 ",[35,2261,2262],{},"reconnecting","（非终态 ",[35,2265,1199],{},"），5 分钟后触发 ",[35,2268,2168],{},[220,2270,2271],{},"Fix C：第二轮慢轮询失败后还能再调度快速重连",[220,2273,2274,2275,2277,2278],{},"Fix B：",[35,2276,2219],{}," 等待 60 秒调用 ",[35,2279,2280],{},"_refreshCredentialsAndReconnect(0)",[220,2282,2274,2283,2285,2286],{},[35,2284,2232],{}," 等待后调用 ",[35,2287,2168],{},[220,2289,2290,2292],{},[35,2291,1906],{}," 时慢轮询应跳过",[220,2294,2295,2297],{},[35,2296,1579],{}," 抛错路径调度慢轮询",[10,2299,2300],{},"测试全部通过（7\u002F7）。",[20,2302,2303],{"id":2303},"外部因素的叠加",[10,2305,2306],{},"这个时期同步发生了另一个外部问题：CDN 对 WebSocket 空闲连接会在 4-9 分钟后强制切断。客户端收到连接中断后根据指数退避策略重连，如果恰好命中上述三条路径之一，就进入永久断开状态。这解释了为什么故障特别在长时间挂机后高发——空闲足够长，CDN 必定切断，而后续重连很容易落入某条缺陷路径。两个问题各自独立，但组合效果是\"挂机过夜必死\"。",[20,2308,2309],{"id":2309},"排查验证",[10,2311,2312],{},"故障修复后，可以用以下方式验证重连状态：",[28,2314,2316],{"className":1526,"code":2315,"language":1528,"meta":33,"style":33},"__clawpanelWsClient.getConnectionInfo()\n\u002F\u002F {\n\u002F\u002F   connected: false,\n\u002F\u002F   reconnectState: 'scheduled' | 'attempting',\n\u002F\u002F   reconnectAttempts: 0~60,\n\u002F\u002F   ...\n\u002F\u002F }\n",[35,2317,2318,2328,2333,2338,2343,2348,2353],{"__ignoreMap":33},[38,2319,2320,2323,2326],{"class":40,"line":41},[38,2321,2322],{"class":119},"__clawpanelWsClient.",[38,2324,2325],{"class":44},"getConnectionInfo",[38,2327,1734],{"class":119},[38,2329,2330],{"class":40,"line":132},[38,2331,2332],{"class":1794},"\u002F\u002F {\n",[38,2334,2335],{"class":40,"line":156},[38,2336,2337],{"class":1794},"\u002F\u002F   connected: false,\n",[38,2339,2340],{"class":40,"line":189},[38,2341,2342],{"class":1794},"\u002F\u002F   reconnectState: 'scheduled' | 'attempting',\n",[38,2344,2345],{"class":40,"line":1595},[38,2346,2347],{"class":1794},"\u002F\u002F   reconnectAttempts: 0~60,\n",[38,2349,2350],{"class":40,"line":1746},[38,2351,2352],{"class":1794},"\u002F\u002F   ...\n",[38,2354,2355],{"class":40,"line":1781},[38,2356,2357],{"class":1794},"\u002F\u002F }\n",[10,2359,2360,2363,2364,2367,2368,2363,2371,2374],{},[35,2361,2362],{},"reconnectState='scheduled'"," 且 ",[35,2365,2366],{},"reconnectAttempts=0"," 表示在慢轮询窗口正常工作；",[35,2369,2370],{},"reconnectState='idle'",[35,2372,2373],{},"connected=false"," 是老的永久断开路径。",[10,2376,2377],{},"重连机制里避免静默终态是最基本的要求。任何异常路径都必须有可见的状态提示和后续操作入口，否则用户端看到的就是死机。这次故障的根本教训是，不能让客户端在任何情况下进入\"不可恢复\"的状态而没有任何提示。慢轮询的引入给了所有失败情景一个\"最后的机会\"，即使前面的快速重连机制彻底耗尽了，用户等待足够长的时间后系统仍有自动恢复的可能。",[586,2379,2380],{},"html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}",{"title":33,"searchDepth":132,"depth":132,"links":2382},[2383,2388,2389,2390,2391,2392],{"id":1508,"depth":132,"text":1508,"children":2384},[2385,2386,2387],{"id":1515,"depth":156,"text":1516},{"id":1685,"depth":156,"text":1686},{"id":1827,"depth":156,"text":1828},{"id":1895,"depth":132,"text":1895},{"id":1919,"depth":132,"text":1919},{"id":2239,"depth":132,"text":2239},{"id":2303,"depth":132,"text":2303},{"id":2309,"depth":132,"text":2309},"2026-06-17","从 2026-05-11 开始，多台便携包客户反馈 WebSocket 断线后面板卡死。网关进程（openclaw_gateway.exe）明确在运行，\u002Fhealth 返回 200，但面板显示\"已停止重连，请手动刷新\"——用户必须手动重启才能恢复。",{},"\u002F2026-06-17-fa-016-ws",{"title":1479,"description":2394},"FA-016","2026-06-17-FA-016-WS永久断开三条路径","客户端重连逻辑中三处独立的\"静默终止\"缺陷导致 WS 永久断开，任何一条路径命中就不再调度重连计时器。",[611,2402,2403,2404,2405],"Node.js","客户端重连机制","异常处理","便携包","qaISKpAwjYnhiSdxEMYYnJtJ-A8QaOq-djOEJ0W9-bM",1785406912285]