[{"data":1,"prerenderedAt":2135},["ShallowReactive",2],{"\u002F2026-07-11-fa-017":3,"\u002F2026-07-11-fa-017-rel":479},{"id":4,"title":5,"body":6,"column":459,"date":460,"description":461,"extension":462,"hero_image":463,"meta":464,"navigation":465,"path":466,"seo":467,"series_id":468,"severity":463,"stem":469,"summary":470,"tags":471,"__hash__":478},"posts\u002F2026-07-11-FA-017-长任务异步化.md","三个超时机制，一起把长任务掐死了",{"type":7,"value":8,"toc":450},"minimark",[9,23,26,29,35,41,47,54,60,65,68,71,81,84,97,103,107,110,117,123,126,141,145,148,159,208,223,230,234,237,240,258,261,269,272,287,290,293,441,444,447],[10,11,12,13,17,18,22],"p",{},"数字人口播的「分析」功能从视频提取口播文稿、分镜脚本、结构卖点，需要下载视频、压缩、调 Gemini 视觉 API，耗时从几十秒到几分钟。最初同步实现：前端调 ",[14,15,16],"code",{},"POST \u002Fapi\u002Fworkflow\u002Fdub\u002Fanalyze-parsed","，后端直接跑完再返回。这个方案暴露的问题不是单点，而是",[19,20,21],"strong",{},"三个独立的缺陷叠加","。",[10,24,25],{},"网关超时、CDN 空闲切断、客户端重试——这三个没有一个是\"bug\"，每个都是合理的系统设计。但组合在一起就是连锁灾难：客户端看不到结果（超时或连接断），认为失败了重试，后端却已经扣过费了；或者分析其实已跑完，但因为响应发不出去，重试又跑了一遍。退款时无法幂等判断，某些用户最终被多扣了好几倍。",[10,27,28],{},"具体来说，三个问题各自是什么：",[10,30,31,34],{},[19,32,33],{},"第一个问题是网关超时","。云平台的入口网关（API Gateway）有一个固定的读超时，通常设为几十秒。当后端分析耗时超过这个阈值，网关主动断连接，客户端收到 504 或连接重置，但后端的分析进程并不知道客户端已经走了，继续跑完整个流程。如果分析结果的退款逻辑放在 try-catch 的 finally 里，此时 HTTP 响应已发不出去，客户就看到\"分析失败\"，但款已经扣了。",[10,36,37,40],{},[19,38,39],{},"第二个问题是 CDN 空闲切断","。不少 CDN 和负载均衡层有这样的策略：如果一条 HTTP 连接在某个时间段内没有数据往返，就认为它已死而主动关闭。这与 FA-013 那次踩到的问题一样——长连接因为没有心跳数据而被 CDN 中间件切断，导致请求被突然中断。这个机制在分析请求上也会造成麻烦：如果分析用时较长，连接就可能被切，响应发不出去。",[10,42,43,46],{},[19,44,45],{},"第三个问题是客户端重试导致重复扣费","。当客户端没有收到响应（网关超时或 CDN 切断），一般会重试这个请求。问题在于这次重试是一条完全独立的请求，后端会把它当作新任务处理，再次扣费。而且如果第一次的分析实际上已经跑完了，就会出现\"分析跑了两遍、扣费扣了两遍、用户只看到了一个结果\"的情况。如果第一次分析中途被 CDN 切了，第二次重试重新开始，那扣费可能是三倍。而退款的时候，因为业务流程上没有幂等设计，很难追溯哪个 operationId 应该被退，哪个不应该。",[10,48,49,50,53],{},"这三个问题是",[19,51,52],{},"叠加的","，不是独立的。它们共同指向一个根本的架构问题。",[10,55,56,57,22],{},"根本原因一个：",[19,58,59],{},"不该让耗时任务挂在 HTTP 连接上",[61,62,64],"h2",{"id":63},"异步任务是必需的不是可选的","异步任务是必需的，不是可选的",[10,66,67],{},"解决方案很简单：把分析改成异步任务模式。",[10,69,70],{},"前端提交分析请求直接返回任务 ID（HTTP 耗时极短：校验参数、建数据库记录、预扣费）。后端后台 worker 跑分析重逻辑，完成后持久化结果。前端轮询查询状态，得到结果后停止。",[72,73,78],"pre",{"className":74,"code":76,"language":77},[75],"language-text","客户端              后端\n  |                 |\n  +--提交---------->|\n  |             创建任务，返回 ID\n  |\u003C--返回 ID-----+\n  |                 |（后台运行分析）\n  |                 |\n  +--轮询--------->|\n  |             查询状态\n  |\u003C--返回状态---+\n  |                 |\n  |（重复轮询）\n  +--轮询--------->|\n  |             查询状态\n  |\u003C--返回结果---+\n  |                 |\n","text",[14,79,76],{"__ignoreMap":80},"",[10,82,83],{},"好处显而易见：",[85,86,87,91,94],"ul",{},[88,89,90],"li",{},"单次 HTTP 请求的耗时从分钟级降到秒级，不会触发网关或 CDN 的超时。",[88,92,93],{},"后端分析的运行生命周期与 HTTP 连接完全解耦。即使连接在中途被切，已经启动的分析会继续在后台跑，结果最终还是会被保存。",[88,95,96],{},"客户端重试不会产生新的分析任务。只要实现幂等判重，同一个请求在短时间内重复提交也只会产生一个任务。",[10,98,99,100,22],{},"但异步化不是万灵药，它把问题转移到了三个新的维度上：",[19,101,102],{},"幂等、持久化、失败语义",[61,104,106],{"id":105},"幂等同一请求只产生一个任务","幂等：同一请求只产生一个任务",[10,108,109],{},"假设客户端因为网络抖动，在短时间内连续发了两次\"开始分析\"请求，后端收到两条独立的 HTTP 请求。如果没有幂等设计，就会创建两个任务、扣两次费。",[10,111,112,113,116],{},"幂等的实现最直接的办法是在",[19,114,115],{},"提交阶段就判重","：为这个用户、这个资源设置一个\"只有一个进行中的分析任务\"的约束。素材中的设计是这样做的：",[72,118,121],{"className":119,"code":120,"language":77},[75],"if 该用户已有 status=running && kind=analyze_parsed 的任务:\n    return 409 Conflict（分析已在进行中）\n",[14,122,120],{"__ignoreMap":80},[10,124,125],{},"这样双击提交同一个视频，第一次返回 201 + taskId，第二次返回 409，客户端看到 409 就知道不要再建新任务，拿之前返回的 ID 去轮询。",[10,127,128,129,132,133,136,137,140],{},"另一个幂等层次是",[19,130,131],{},"计费操作的幂等","。预扣费的时候用一个全局唯一的 ",[14,134,135],{},"operationId","（比如 ",[14,138,139],{},"dub-analyze:{uuid}","），这个 ID 关联了这次预扣。如果扣费 API 被重复调用，因为 operationId 重复，系统只会扣一次。退款时也用同一个 operationId，保证退款与预扣能正确配对。",[61,142,144],{"id":143},"持久化进程重启后任务不丢","持久化：进程重启后任务不丢",[10,146,147],{},"后台分析 worker 可能因为发版、机器重启、容器被杀等原因在中途退出。如果任务的状态只保存在内存里，进程一死任务就丢了。",[10,149,150,151,154,155,158],{},"持久化的关键是",[19,152,153],{},"任务状态表","。素材中复用了既有的 ",[14,156,157],{},"SkyhumanTask"," 表，包含字段：",[85,160,161,167,173,178,184,190,196,202],{},[88,162,163,166],{},[14,164,165],{},"id",": 任务 ID",[88,168,169,172],{},[14,170,171],{},"status",": 进度（running \u002F completed \u002F failed）",[88,174,175,177],{},[14,176,135],{},": 幂等 ID",[88,179,180,183],{},[14,181,182],{},"chargedPoints",": 预扣费用",[88,185,186,189],{},[14,187,188],{},"resultPayload",": 分析结果的 JSON",[88,191,192,195],{},[14,193,194],{},"error",": 失败原因",[88,197,198,201],{},[14,199,200],{},"updatedAt",": 最后更新时间戳",[88,203,204,207],{},[14,205,206],{},"sourceObjectKey",": 源视频对象路径（用于幂等判重和重试）",[10,209,210,211,214,215,218,219,222],{},"任务创建时插一条 ",[14,212,213],{},"status=running"," 的记录。后台 worker 定期通过 heartbeat（每 30 秒做一次空 update，仅触碰 updatedAt）来证明自己还活着。分析完成时更新为 ",[14,216,217],{},"status=completed, resultPayload=...","；失败时更新为 ",[14,220,221],{},"status=failed, error=..."," 并触发退款。",[10,224,225,226,229],{},"这样即使 worker 进程突然死了，任务记录还在数据库里。新启起的 worker 可以扫描表里的 ",[14,227,228],{},"status=running && updatedAt 过期"," 的任务，判断它们已经没有 worker 在跑了，执行清理逻辑。",[61,231,233],{"id":232},"失败语义区分任务失败和查询失败","失败语义：区分\"任务失败\"和\"查询失败\"",[10,235,236],{},"异步设计引入了一个新的错误维度：客户端查询任务状态时得到的 404，可能是\"这个任务 ID 不存在\"（用户输错了），也可能是\"任务 ID 之前存在但已被清理\"（进程清理了过期数据）。这两种情况的含义很不一样。",[10,238,239],{},"更重要的是，前端轮询收到的错误需要区分是否应该重试：",[85,241,242,252],{},[88,243,244,247,248,251],{},[19,245,246],{},"任务本身失败","（",[14,249,250],{},"status=failed, error=\"Gemini API 返回错误\"","）：这种情况不该重试，因为重试也会失败。应该直接向用户展示失败信息，并根据 operationId 触发退款。",[88,253,254,257],{},[19,255,256],{},"查询接口失败","（500、网络超时）：这种情况应该重试查询，因为任务本身可能还在继续跑。",[10,259,260],{},"素材中的设计把这两类区分得很清楚：",[85,262,263,266],{},[88,264,265],{},"任务最终的状态（completed 或 failed）持久化到数据库，客户端查询会拿到明确的业务语义。",[88,267,268],{},"查询接口的 HTTP 错误（5xx）是技术层故障，客户端应该重试。",[10,270,271],{},"同时，后端的 reaper 机制（定期扫表清理超期任务）也遵循这个语义：",[85,273,274,280],{},[88,275,276,277,279],{},"如果一个 ",[14,278,213],{}," 的任务超过 90 秒没有 heartbeat 更新，说明 worker 已死，reaper 会强制置为 failed 并退款。",[88,281,282,283,286],{},"但这个强制置为 failed 不应该抛错，因为此时可能有其他进程也想更新这个任务。更新语句应该带条件：",[14,284,285],{},"UPDATE SkyhumanTask SET status=failed WHERE id=? AND status=running","，这样如果 reaper 和其他 worker 同时操作，只有一方会成功。",[61,288,289],{"id":289},"具体实现模式",[10,291,292],{},"数字人口播的分析异步化采用了这样的模式：",[294,295,296,341,388,411],"ol",{},[88,297,298,301,302],{},[19,299,300],{},"提交阶段","（同步）：",[85,303,304,307,310,316,322,329,335],{},[88,305,306],{},"校验用户和资源",[88,308,309],{},"判重：是否已有进行中的任务（409）",[88,311,312,315],{},[14,313,314],{},"getObject"," 拉视频，检测时长（若≤0 返 400 不扣费）",[88,317,318,321],{},[14,319,320],{},"chargeResource"," 预扣费用，获得 operationId",[88,323,324,325,328],{},"创建 ",[14,326,327],{},"SkyhumanTask{ status:running, operationId, sourceObjectKey }"," 记录",[88,330,331,334],{},[14,332,333],{},"scheduleTask"," 丢到后台队列",[88,336,337,338],{},"返回 ",[14,339,340],{},"{ taskId }",[88,342,343,346,347],{},[19,344,345],{},"后台运行阶段","：",[85,348,349,352,355,362,368,378,385],{},[88,350,351],{},"Worker 取出任务记录",[88,353,354],{},"开启 heartbeat 定时器（每 30s update 一次 updatedAt）",[88,356,357,358,361],{},"执行 ",[14,359,360],{},"analyzeDubVisionOnly","（下载→压缩→调用 Gemini→解析）",[88,363,364,365],{},"成功：",[14,366,367],{},"update( status:completed, resultPayload )",[88,369,370,371,374,375],{},"失败：",[14,372,373],{},"update( status:failed, error )"," + ",[14,376,377],{},"refundResource(operationId)",[88,379,380,381,384],{},"所有 update 都带条件 ",[14,382,383],{},"WHERE id=? AND status=running","（防被 reaper 覆盖）",[88,386,387],{},"Finally 清理 heartbeat 定时器",[88,389,390,393,394],{},[19,391,392],{},"查询阶段","（前端轮询）：",[85,395,396,405,408],{},[88,397,398,401,402],{},[14,399,400],{},"GET \u002Fapi\u002Fworkflow\u002Fdub\u002Ftasks\u002F:id"," 返回 ",[14,403,404],{},"{ status, resultPayload, error }",[88,406,407],{},"若 status=completed 或 failed，停止轮询",[88,409,410],{},"若查询接口返回 5xx，则后续重试",[88,412,413,416,417],{},[19,414,415],{},"清理阶段","（reaper）：",[85,418,419,422,429,432,438],{},[88,420,421],{},"后台定期扫表",[88,423,424,425,428],{},"对于 ",[14,426,427],{},"status=running && updatedAt > 90s"," 的任务，判定 worker 已死",[88,430,431],{},"若 providerTaskId（这里没有）存在，调用上游的 finalize API",[88,433,434,435,437],{},"若不存在，直接 ",[14,436,377],{}," + 置 failed",[88,439,440],{},"本设计无 providerTaskId，所以 reaper 自动走退款分支，无需改动 reaper 代码",[10,442,443],{},"这个模式的关键是三道防线：提交时判重（409）、后台心跳（定期 touch）、reaper 兜底（自动退款）。任何一个环节出问题，都有后续环节来补救。",[61,445,446],{"id":446},"防线缺一不可",[10,448,449],{},"长耗时任务不能挂在 HTTP 连接上不只是超时问题，而是连接脆弱性与重复处理的共谋。异步化表面是\"返回 ID、后台跑\"这一步，真正的难点在三个维度的同时保证：幂等（不重复扣费）、持久化（不丢数据）、失败语义（不破坏重试逻辑）。缺一就会在某个场景暴露。",{"title":80,"searchDepth":451,"depth":451,"links":452},2,[453,454,455,456,457,458],{"id":63,"depth":451,"text":64},{"id":105,"depth":451,"text":106},{"id":143,"depth":451,"text":144},{"id":232,"depth":451,"text":233},{"id":289,"depth":451,"text":289},{"id":446,"depth":451,"text":446},"故障档案","2026-07-11","数字人口播的「分析」功能从视频提取口播文稿、分镜脚本、结构卖点，需要下载视频、压缩、调 Gemini 视觉 API，耗时从几十秒到几分钟。最初同步实现：前端调 POST \u002Fapi\u002Fworkflow\u002Fdub\u002Fanalyze-parsed，后端直接跑完再返回。这个方案暴露的问题不是单点，而是三个独立的缺陷叠加。","md",null,{},true,"\u002F2026-07-11-fa-017",{"title":5,"description":461},"FA-017","2026-07-11-FA-017-长任务异步化","长耗时分析请求（几十秒到分钟级）用同步 HTTP 导致网关超时、CDN切断、客户端重试重复扣费。改异步任务需同时解决幂等、持久化、失败语义。",[472,473,474,475,476,477],"异步任务","HTTP超时","CDN","幂等","任务队列","微服务","ARFDvBMGX3tw-E3Rj_gw1RYEjrykplO6u1dxtdS7Uto",[480,899,1193],{"id":481,"title":482,"body":483,"column":459,"date":884,"description":487,"extension":462,"hero_image":463,"meta":885,"navigation":465,"path":886,"seo":887,"series_id":888,"severity":463,"stem":889,"summary":890,"tags":891,"__hash__":898},"posts\u002F2026-07-23-FA-018-共享存储的subPath陷阱.md","一个 PVC 装所有用户，当时看不出有什么问题",{"type":7,"value":484,"toc":875},[485,488,491,495,498,501,508,559,562,572,576,582,585,596,599,603,606,609,612,630,633,640,643,658,662,665,668,679,690,693,696,699,767,770,777,780,783,809,812,823,826,829,868,871],[10,486,487],{},"平台从 Docker Swarm 迁移到 Kubernetes，存储架构做了一个看似理想的调整：用共享 PVC + subPath 替代原来的\"每用户独占 PVC\"。方案优势显而易见——管理简单（1 个 PVC 对 574 个）、自动扩展、成本低。当时看不出有什么缺点。",[10,489,490],{},"但实施中遇到了四个隐蔽的问题。",[61,492,494],{"id":493},"权限错配bind-型数据的属主陷阱","权限错配：bind 型数据的属主陷阱",[10,496,497],{},"旧系统中，用户数据通过两种方式持久化：bind mount 和 Docker volume。迁移时需要从这两个来源完整导出数据。",[10,499,500],{},"对于 bind mount 型数据，文件属主通常是 10001（或其他固定 UID）。当我登上旧集群的宿主机，以普通用户身份尝试读取这些文件时，只能看到 13 个文件。权限拒绝。同一目录里其实有 4778 个文件，但大多数因为属主权限不匹配而不可见。",[10,502,503,504,507],{},"只有用 sudo 才能完整读到。所以迁移脚本必须用 ",[14,505,506],{},"sudo tar"," 来打包：",[72,509,513],{"className":510,"code":511,"language":512,"meta":80,"style":80},"language-bash shiki shiki-themes github-light github-dark","sudo tar -czf \u002Ftmp\u002Fmig\u002F${uid}\u002Fdata.tgz \\\n  -C \u002Fmnt\u002Fyun-claw\u002Fusers\u002F${uid} .\n","bash",[14,514,515,545],{"__ignoreMap":80},[516,517,520,524,528,532,535,539,542],"span",{"class":518,"line":519},"line",1,[516,521,523],{"class":522},"sScJk","sudo",[516,525,527],{"class":526},"sZZnC"," tar",[516,529,531],{"class":530},"sj4cs"," -czf",[516,533,534],{"class":526}," \u002Ftmp\u002Fmig\u002F",[516,536,538],{"class":537},"sVt8B","${uid}",[516,540,541],{"class":526},"\u002Fdata.tgz",[516,543,544],{"class":530}," \\\n",[516,546,547,550,553,556],{"class":518,"line":451},[516,548,549],{"class":530},"  -C",[516,551,552],{"class":526}," \u002Fmnt\u002Fyun-claw\u002Fusers\u002F",[516,554,555],{"class":537},"${uid} ",[516,557,558],{"class":526},".\n",[10,560,561],{},"这不是脚本设计的问题，而是 Kubernetes 环境下容器权限与宿主权限映射的一个陷阱。容器内运行的网关进程可能以容器用户身份运行，但宿主上的文件属主是不同的 UID。当两个权限体系接不上，访问就会失败。",[10,563,564,565,568,569,571],{},"新方案中，用户数据通过 subPath 挂到容器的 ",[14,566,567],{},"\u002Fdata"," 目录。如果 subPath 指向的目录是由 Kubernetes 在宿主机上创建的，属主可能是 root 或其他用户，容器内进程（以容器指定的 UID 运行）仍然会遭遇权限问题。这个风险需要在 Dockerfile 和容器启动时明确处理：容器内应该预先创建 ",[14,570,567],{}," 并设置正确的属主，或者使用 securityContext 的 fsGroup 或 runAsUser 来强制权限。",[61,573,575],{"id":574},"subpath-目录创建的时机与属主处理","subPath 目录创建的时机与属主处理",[10,577,578,579],{},"Kubernetes 对 subPath 的处理有个微妙之处：如果挂载的 subPath 目录在宿主机上不存在，kubelet 会自动创建它。但 ",[516,580,581],{},"待补：这个自动创建的目录的属主是什么？是 root 还是其他用户？容器的 securityContext 能否保证挂载后的权限符合预期？",[10,583,584],{},"设计文档中没有明确说明这一点。在实施前，需要验证：",[85,586,587,590,593],{},[88,588,589],{},"首次 subPath 目录不存在时，kubelet 创建它并将其属主设置为谁",[88,591,592],{},"容器的 securityContext（fsGroup、runAsUser）是否能在挂载时生效",[88,594,595],{},"是否应该预先在共享 PVC 上创建所有 subPath 目录并设置好属主，而不是依赖 Kubernetes 的隐式创建",[10,597,598],{},"没有明确的答案，就容易踩坑。一个保险做法是在 destroy 用户账户时清空 subPath 目录，同时在 provision 时让一个 init-container 验证或修复目录属主，确保容器进程有写权限。",[61,600,602],{"id":601},"单-pvc-中的节点级故障域","单 PVC 中的节点级故障域",[10,604,605],{},"这是最严重的问题，也是与 FA-015（节点假 Ready）的关联点。",[10,607,608],{},"当一个节点的 NFS 客户端或到 SFS 的网络链路发生 hang 时，所有依赖共享 PVC 的 Pod 都会受影响。这不是共享 PVC 本身的问题，而是故障隔离的问题。",[10,610,611],{},"具体现象：",[85,613,614,621,624,627],{},[88,615,616,617,620],{},"如果 NFS 挂载使用了硬 mount（",[14,618,619],{},"hard"," 参数是 NFS 的默认值），而网络故障或存储服务中断，NFS 客户端会无限重试",[88,622,623],{},"重试期间，所有试图访问 NFS 的进程都会被阻塞在 I\u002FO 上（D 状态，不可中断）",[88,625,626],{},"如果 kubelet 或 containerd 的某个操作（如创建容器的卷挂载步骤）陷入 I\u002FO 阻塞，整个节点就会冻死",[88,628,629],{},"该节点上的所有实例（无论是否在访问存储）都会因为无法创建或删除 Pod 而故障",[10,631,632],{},"这与独立 PVC 的效果截然不同。旧架构中，每个用户有独立 PVC，意味着如果某个 PVC 对应的存储有问题，只会影响这一个用户。其他用户的 PVC 可能分散在不同的存储卷甚至不同的节点上，相对独立。",[10,634,635,636,639],{},"新架构下，单个共享 PVC 承载所有用户，一旦这个 PVC 对应的网络链路或挂载出问题，",[19,637,638],{},"同节点上所有实例都会被殃及","。如果该节点碰巧堆积了 50 个实例，一次节点级的存储 hang 就会导致 50 个实例集体故障，且外表看起来是\"节点 Ready，但实例卡住\"——kubelet 的心跳正常，Pod 状态却卡在 ContainerCreating 或 Terminating。",[10,641,642],{},"这正是 FA-015 在 2026-06-05 观察到的现象：节点假 Ready，但大量实例网关无响应。防护措施包括：",[85,644,645,652,655],{},[88,646,647,648,651],{},"NFS 挂载参数改为软 mount（",[14,649,650],{},"soft,timeo=100,retrans=3","），让 I\u002FO 超时而不是无限等待",[88,653,654],{},"节点加健康检测，检查 ContainerCreating 堆积数和网关探测失败率，及早发现节点冻死",[88,656,657],{},"实例分散部署，用 topologySpreadConstraints 避免单节点堆积",[61,659,661],{"id":660},"无-per-subpath-硬配额","无 per-subPath 硬配额",[10,663,664],{},"共享存储的架构决定了无法在存储层面按 subPath 限制用户配额。SFS（及 NFS 一般）没有 per-directory 的硬配额功能，只能在整个卷级别限制。",[10,666,667],{},"这意味着：",[85,669,670,673,676],{},[88,671,672],{},"无法阻止某个用户的数据膨胀而挤占其他用户的空间",[88,674,675],{},"无法在存储层面实现\"超额用户的写入失败\"",[88,677,678],{},"只能在应用层进行监控、告警和逻辑控制",[10,680,681,682,685,686,689],{},"新系统的方案是应用层监控：定期 ",[14,683,684],{},"du -sb \u002Fdata"," 扫描每个用户的目录大小，落库到 Instance 表，",[14,687,688],{},"\u002Fstatus"," 接口返回超额标志位。前端据此提示用户\"存储已满\"。这是纯监控，不阻断。如果用户无视提示继续写入，直到共享卷真的满了，才会出现\"所有用户集体写入失败\"的惨淡局面。",[10,691,692],{},"这个限制是方案选择的代价。",[61,694,695],{"id":695},"为什么仍然选了这个方案",[10,697,698],{},"尽管有这些问题，平台仍然采用了共享 PVC + subPath 的方案。原因是对比了替代方案：",[700,701,702,721],"table",{},[703,704,705],"thead",{},[706,707,708,712,715,718],"tr",{},[709,710,711],"th",{},"方案",[709,713,714],{},"优点",[709,716,717],{},"缺点",[709,719,720],{},"故障隔离",[722,723,724,739,753],"tbody",{},[706,725,726,730,733,736],{},[727,728,729],"td",{},"独立 PVC（旧架构）",[727,731,732],{},"天然的每用户隔离；故障域清晰",[727,734,735],{},"管理复杂（574 个 PVC）；扩展性差；成本高",[727,737,738],{},"✓ 最好",[706,740,741,744,747,750],{},[727,742,743],{},"共享 PVC + subPath（新架构）",[727,745,746],{},"管理简单（1 个 PVC）；自动扩展；成本低",[727,748,749],{},"故障隔离性差；无硬配额；权限管理复杂",[727,751,752],{},"✗ 较差",[706,754,755,758,761,764],{},[727,756,757],{},"对象存储（S3\u002FOSS）",[727,759,760],{},"真正的多租户隔离；天然分布",[727,762,763],{},"延迟高；成本更高；应用改造大",[727,765,766],{},"✓ 最好，但代价大",[10,768,769],{},"独立 PVC 的管理开销是关键问题。574 个用户意味着 574 个 PVC 对象、574 条 PV 绑定、574 个存储卷。每次实例创建、删除或迁移都要涉及 PVC 的生命周期管理。扩展到 5000 用户时，这种开销会成为瓶颈。对象存储虽然隔离性最好，但需要应用层改造（兼容 S3 API、处理延迟、调整备份策略），而且成本更高。",[10,771,772,773,776],{},"共享 PVC 方案的核心优势是",[19,774,775],{},"运维简洁","：增删用户只需改 subPath（一行 YAML），不涉及存储层操作。代价是故障隔离性从\"用户级\"降到\"节点级\"。",[61,778,779],{"id":779},"适用边界",[10,781,782],{},"这个方案适合以下场景：",[85,784,785,791,797,803],{},[88,786,787,790],{},[19,788,789],{},"用户数量有上限","（几百到几千）。用户过多时，共享卷的单点压力会成问题。",[88,792,793,796],{},[19,794,795],{},"用户数据量可控","（单用户通常 GB 级）。如果单用户数据量达 TB，一次 du 扫描会拖累整体。",[88,798,799,802],{},[19,800,801],{},"可以接受节点级故障隔离","。只要网络和存储配置足够稳定（硬化 NFS 参数、冗余链路），节点 hang 的概率不会很高。",[88,804,805,808],{},[19,806,807],{},"能够实施应用层监控和告警","。无硬配额，就必须有实时监控。",[10,810,811],{},"不适合的场景包括：",[85,813,814,817,820],{},[88,815,816],{},"超大规模用户（万级以上）",[88,818,819],{},"用户数据特别不均衡（少数用户占大头）的场景",[88,821,822],{},"对故障隔离要求极高的系统（比如金融交易）",[61,824,825],{"id":825},"防护与调整",[10,827,828],{},"基于这四个问题，实施中做了以下调整：",[294,830,831,840,850,856,862],{},[88,832,833,836,837,839],{},[19,834,835],{},"权限处理","：容器 Dockerfile 中预先创建 ",[14,838,567],{}," 并设置正确的属主；启动脚本检查并修复权限。",[88,841,842,845,846,849],{},[19,843,844],{},"NFS 参数硬化","：StorageClass 的挂载选项改为 ",[14,847,848],{},"soft,timeo=100,retrans=3,intr","，避免硬 mount 导致节点冻。",[88,851,852,855],{},[19,853,854],{},"节点健康检测","：cron 每 60 秒检查各节点的 ContainerCreating 堆积和网关探测失败率，及早发现故障。",[88,857,858,861],{},[19,859,860],{},"实例分散","：StatefulSet 加 topologySpreadConstraints，避免单节点堆积 50+ 个实例。",[88,863,864,867],{},[19,865,866],{},"应用层配额","：\u002Fstatus 接口返回 diskUsedMi 和 overQuota 标志位，前端告警用户。",[10,869,870],{},"这些措施不能消除风险，但可以大幅降低故障概率和影响范围。",[872,873,874],"style",{},"html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":80,"searchDepth":451,"depth":451,"links":876},[877,878,879,880,881,882,883],{"id":493,"depth":451,"text":494},{"id":574,"depth":451,"text":575},{"id":601,"depth":451,"text":602},{"id":660,"depth":451,"text":661},{"id":695,"depth":451,"text":695},{"id":779,"depth":451,"text":779},{"id":825,"depth":451,"text":825},"2026-07-23",{},"\u002F2026-07-23-fa-018-subpath",{"title":482,"description":487},"FA-018","2026-07-23-FA-018-共享存储的subPath陷阱","K8s 共享 PVC + subPath 方案在实施中暴露的四个陷阱：权限不匹配、目录创建时机、节点级故障域、无 per-subPath 配额。",[892,893,894,895,896,897,720],"Kubernetes","存储","subPath","PVC","NFS","权限","4pJJyzbII4OOc9jJS74HUpUfyVc9CfxdE5ovso8ylSE",{"id":4,"title":5,"body":900,"column":459,"date":460,"description":461,"extension":462,"hero_image":463,"meta":1190,"navigation":465,"path":466,"seo":1191,"series_id":468,"severity":463,"stem":469,"summary":470,"tags":1192,"__hash__":478},{"type":7,"value":901,"toc":1182},[902,908,910,912,916,920,924,928,932,934,936,938,943,945,953,957,959,961,965,970,972,980,982,984,990,1024,1032,1036,1038,1040,1042,1054,1056,1062,1064,1074,1076,1078,1176,1178,1180],[10,903,12,904,17,906,22],{},[14,905,16],{},[19,907,21],{},[10,909,25],{},[10,911,28],{},[10,913,914,34],{},[19,915,33],{},[10,917,918,40],{},[19,919,39],{},[10,921,922,46],{},[19,923,45],{},[10,925,49,926,53],{},[19,927,52],{},[10,929,56,930,22],{},[19,931,59],{},[61,933,64],{"id":63},[10,935,67],{},[10,937,70],{},[72,939,941],{"className":940,"code":76,"language":77},[75],[14,942,76],{"__ignoreMap":80},[10,944,83],{},[85,946,947,949,951],{},[88,948,90],{},[88,950,93],{},[88,952,96],{},[10,954,99,955,22],{},[19,956,102],{},[61,958,106],{"id":105},[10,960,109],{},[10,962,112,963,116],{},[19,964,115],{},[72,966,968],{"className":967,"code":120,"language":77},[75],[14,969,120],{"__ignoreMap":80},[10,971,125],{},[10,973,128,974,132,976,136,978,140],{},[19,975,131],{},[14,977,135],{},[14,979,139],{},[61,981,144],{"id":143},[10,983,147],{},[10,985,150,986,154,988,158],{},[19,987,153],{},[14,989,157],{},[85,991,992,996,1000,1004,1008,1012,1016,1020],{},[88,993,994,166],{},[14,995,165],{},[88,997,998,172],{},[14,999,171],{},[88,1001,1002,177],{},[14,1003,135],{},[88,1005,1006,183],{},[14,1007,182],{},[88,1009,1010,189],{},[14,1011,188],{},[88,1013,1014,195],{},[14,1015,194],{},[88,1017,1018,201],{},[14,1019,200],{},[88,1021,1022,207],{},[14,1023,206],{},[10,1025,210,1026,214,1028,218,1030,222],{},[14,1027,213],{},[14,1029,217],{},[14,1031,221],{},[10,1033,225,1034,229],{},[14,1035,228],{},[61,1037,233],{"id":232},[10,1039,236],{},[10,1041,239],{},[85,1043,1044,1050],{},[88,1045,1046,247,1048,251],{},[19,1047,246],{},[14,1049,250],{},[88,1051,1052,257],{},[19,1053,256],{},[10,1055,260],{},[85,1057,1058,1060],{},[88,1059,265],{},[88,1061,268],{},[10,1063,271],{},[85,1065,1066,1070],{},[88,1067,276,1068,279],{},[14,1069,213],{},[88,1071,282,1072,286],{},[14,1073,285],{},[61,1075,289],{"id":289},[10,1077,292],{},[294,1079,1080,1110,1140,1156],{},[88,1081,1082,301,1084],{},[19,1083,300],{},[85,1085,1086,1088,1090,1094,1098,1102,1106],{},[88,1087,306],{},[88,1089,309],{},[88,1091,1092,315],{},[14,1093,314],{},[88,1095,1096,321],{},[14,1097,320],{},[88,1099,324,1100,328],{},[14,1101,327],{},[88,1103,1104,334],{},[14,1105,333],{},[88,1107,337,1108],{},[14,1109,340],{},[88,1111,1112,346,1114],{},[19,1113,345],{},[85,1115,1116,1118,1120,1124,1128,1134,1138],{},[88,1117,351],{},[88,1119,354],{},[88,1121,357,1122,361],{},[14,1123,360],{},[88,1125,364,1126],{},[14,1127,367],{},[88,1129,370,1130,374,1132],{},[14,1131,373],{},[14,1133,377],{},[88,1135,380,1136,384],{},[14,1137,383],{},[88,1139,387],{},[88,1141,1142,393,1144],{},[19,1143,392],{},[85,1145,1146,1152,1154],{},[88,1147,1148,401,1150],{},[14,1149,400],{},[14,1151,404],{},[88,1153,407],{},[88,1155,410],{},[88,1157,1158,416,1160],{},[19,1159,415],{},[85,1161,1162,1164,1168,1170,1174],{},[88,1163,421],{},[88,1165,424,1166,428],{},[14,1167,427],{},[88,1169,431],{},[88,1171,434,1172,437],{},[14,1173,377],{},[88,1175,440],{},[10,1177,443],{},[61,1179,446],{"id":446},[10,1181,449],{},{"title":80,"searchDepth":451,"depth":451,"links":1183},[1184,1185,1186,1187,1188,1189],{"id":63,"depth":451,"text":64},{"id":105,"depth":451,"text":106},{"id":143,"depth":451,"text":144},{"id":232,"depth":451,"text":233},{"id":289,"depth":451,"text":289},{"id":446,"depth":451,"text":446},{},{"title":5,"description":461},[472,473,474,475,476,477],{"id":1194,"title":1195,"body":1196,"column":459,"date":2120,"description":2121,"extension":462,"hero_image":463,"meta":2122,"navigation":465,"path":2123,"seo":2124,"series_id":2125,"severity":463,"stem":2126,"summary":2127,"tags":2128,"__hash__":2134},"posts\u002F2026-06-17-FA-016-WS永久断开三条路径.md","日志里没有重连记录——它根本没在重连",{"type":7,"value":1197,"toc":2108},[1198,1209,1216,1223,1226,1229,1234,1241,1321,1327,1397,1408,1412,1415,1541,1552,1556,1562,1617,1620,1623,1626,1641,1644,1647,1650,1657,1915,1925,1928,1938,1951,1961,1964,1967,1974,2025,2028,2031,2034,2037,2040,2085,2102,2105],[10,1199,1200,1201,1204,1205,1208],{},"从 2026-05-11 开始，多台便携包客户反馈 WebSocket 断线后面板卡死。网关进程（",[14,1202,1203],{},"openclaw_gateway.exe","）明确在运行，",[14,1206,1207],{},"\u002Fhealth"," 返回 200，但面板显示\"已停止重连，请手动刷新\"——用户必须手动重启才能恢复。",[10,1210,1211,1212,1215],{},"关键线索在控制台：看不到任何 ",[14,1213,1214],{},"[ws] 计划重连"," 的日志。",[10,1217,1218,1219,1222],{},"通常断线后客户端会频繁尝试重连，日志里应该是满屏的重连记录。没有日志意味着问题不在重连失败，而在",[19,1220,1221],{},"压根没启动重连机制","。说明客户端进入了某个永久休眠状态。",[61,1224,1225],{"id":1225},"三条独立的永久断开路径",[10,1227,1228],{},"代码审查发现了三条完全不同的、独立的永久断开路径。任意一条命中，就会停止调度任何重连计时器。",[1230,1231,1233],"h3",{"id":1232},"路径-a凭据刷新异常","路径 A：凭据刷新异常",[10,1235,1236,1237,1240],{},"在 ",[14,1238,1239],{},"_scheduleReconnect"," 方法末尾，每次普通重连都会调用：",[72,1242,1246],{"className":1243,"code":1244,"language":1245,"meta":80,"style":80},"language-javascript shiki shiki-themes github-light github-dark","this._reconnectTimer = setTimeout(() => {\n  if (!this._intentionalClose) {\n    this._refreshCredentialsAndReconnect(0)\n  }\n}, delay)\n","javascript",[14,1247,1248,1272,1288,1309,1315],{"__ignoreMap":80},[516,1249,1250,1253,1256,1260,1263,1266,1269],{"class":518,"line":519},[516,1251,1252],{"class":530},"this",[516,1254,1255],{"class":537},"._reconnectTimer ",[516,1257,1259],{"class":1258},"szBVR","=",[516,1261,1262],{"class":522}," setTimeout",[516,1264,1265],{"class":537},"(() ",[516,1267,1268],{"class":1258},"=>",[516,1270,1271],{"class":537}," {\n",[516,1273,1274,1277,1280,1283,1285],{"class":518,"line":451},[516,1275,1276],{"class":1258},"  if",[516,1278,1279],{"class":537}," (",[516,1281,1282],{"class":1258},"!",[516,1284,1252],{"class":530},[516,1286,1287],{"class":537},"._intentionalClose) {\n",[516,1289,1291,1294,1297,1300,1303,1306],{"class":518,"line":1290},3,[516,1292,1293],{"class":530},"    this",[516,1295,1296],{"class":537},".",[516,1298,1299],{"class":522},"_refreshCredentialsAndReconnect",[516,1301,1302],{"class":537},"(",[516,1304,1305],{"class":530},"0",[516,1307,1308],{"class":537},")\n",[516,1310,1312],{"class":518,"line":1311},4,[516,1313,1314],{"class":537},"  }\n",[516,1316,1318],{"class":518,"line":1317},5,[516,1319,1320],{"class":537},"}, delay)\n",[10,1322,1323,1324,1326],{},"而 ",[14,1325,1299],{}," 是异步方法，catch 分支没有任何后续处理：",[72,1328,1330],{"className":1243,"code":1329,"language":1245,"meta":80,"style":80},"} catch (e) {\n  console.error('[ws] 刷新凭据失败:', e)\n  this._setConnected(false, 'error', `凭据刷新失败: ${e}`)\n}\n",[14,1331,1332,1343,1358,1392],{"__ignoreMap":80},[516,1333,1334,1337,1340],{"class":518,"line":519},[516,1335,1336],{"class":537},"} ",[516,1338,1339],{"class":1258},"catch",[516,1341,1342],{"class":537}," (e) {\n",[516,1344,1345,1348,1350,1352,1355],{"class":518,"line":451},[516,1346,1347],{"class":537},"  console.",[516,1349,194],{"class":522},[516,1351,1302],{"class":537},[516,1353,1354],{"class":526},"'[ws] 刷新凭据失败:'",[516,1356,1357],{"class":537},", e)\n",[516,1359,1360,1363,1365,1368,1370,1373,1376,1379,1381,1384,1387,1390],{"class":518,"line":1290},[516,1361,1362],{"class":530},"  this",[516,1364,1296],{"class":537},[516,1366,1367],{"class":522},"_setConnected",[516,1369,1302],{"class":537},[516,1371,1372],{"class":530},"false",[516,1374,1375],{"class":537},", ",[516,1377,1378],{"class":526},"'error'",[516,1380,1375],{"class":537},[516,1382,1383],{"class":526},"`凭据刷新失败: ${",[516,1385,1386],{"class":537},"e",[516,1388,1389],{"class":526},"}`",[516,1391,1308],{"class":537},[516,1393,1394],{"class":518,"line":1311},[516,1395,1396],{"class":537},"}\n",[10,1398,1399,1400,1403,1404,1407],{},"如果 ",[14,1401,1402],{},"api.readOpenclawConfig()"," 或 ",[14,1405,1406],{},"api.autoPairDevice()"," 抛错——磁盘 IO 抖、Rust 端处理器忙、Tauri IPC 队列阻塞——这次重连就永久终止。便携磁盘的 IO 抖动 + 配置文件读写争抢时最容易发生。",[1230,1409,1411],{"id":1410},"路径-b认证失败后的强制关闭","路径 B：认证失败后的强制关闭",[10,1413,1414],{},"当 WebSocket 收到 1008 unauthorized 响应时的处理逻辑：",[72,1416,1418],{"className":1243,"code":1417,"language":1245,"meta":80,"style":80},"if (this._authRetryCount \u003C 2) {\n  this._authRetryCount++\n  this._refreshCredentialsAndReconnect()\n  return\n}\nthis._setConnected(false, 'auth_failed', `认证失败: ${e.reason}。请检查 Gateway Token 配置。`)\nthis._intentionalClose = true   \u002F\u002F ← 永久关闭标记\nthis._flushPending()\nreturn\n",[14,1419,1420,1441,1451,1462,1467,1471,1506,1523,1535],{"__ignoreMap":80},[516,1421,1422,1425,1427,1429,1432,1435,1438],{"class":518,"line":519},[516,1423,1424],{"class":1258},"if",[516,1426,1279],{"class":537},[516,1428,1252],{"class":530},[516,1430,1431],{"class":537},"._authRetryCount ",[516,1433,1434],{"class":1258},"\u003C",[516,1436,1437],{"class":530}," 2",[516,1439,1440],{"class":537},") {\n",[516,1442,1443,1445,1448],{"class":518,"line":451},[516,1444,1362],{"class":530},[516,1446,1447],{"class":537},"._authRetryCount",[516,1449,1450],{"class":1258},"++\n",[516,1452,1453,1455,1457,1459],{"class":518,"line":1290},[516,1454,1362],{"class":530},[516,1456,1296],{"class":537},[516,1458,1299],{"class":522},[516,1460,1461],{"class":537},"()\n",[516,1463,1464],{"class":518,"line":1311},[516,1465,1466],{"class":1258},"  return\n",[516,1468,1469],{"class":518,"line":1317},[516,1470,1396],{"class":537},[516,1472,1474,1476,1478,1480,1482,1484,1486,1489,1491,1494,1496,1498,1501,1504],{"class":518,"line":1473},6,[516,1475,1252],{"class":530},[516,1477,1296],{"class":537},[516,1479,1367],{"class":522},[516,1481,1302],{"class":537},[516,1483,1372],{"class":530},[516,1485,1375],{"class":537},[516,1487,1488],{"class":526},"'auth_failed'",[516,1490,1375],{"class":537},[516,1492,1493],{"class":526},"`认证失败: ${",[516,1495,1386],{"class":537},[516,1497,1296],{"class":526},[516,1499,1500],{"class":537},"reason",[516,1502,1503],{"class":526},"}。请检查 Gateway Token 配置。`",[516,1505,1308],{"class":537},[516,1507,1509,1511,1514,1516,1519],{"class":518,"line":1508},7,[516,1510,1252],{"class":530},[516,1512,1513],{"class":537},"._intentionalClose ",[516,1515,1259],{"class":1258},[516,1517,1518],{"class":530}," true",[516,1520,1522],{"class":1521},"sJ8bj","   \u002F\u002F ← 永久关闭标记\n",[516,1524,1526,1528,1530,1533],{"class":518,"line":1525},8,[516,1527,1252],{"class":530},[516,1529,1296],{"class":537},[516,1531,1532],{"class":522},"_flushPending",[516,1534,1461],{"class":537},[516,1536,1538],{"class":518,"line":1537},9,[516,1539,1540],{"class":1258},"return\n",[10,1542,1543,1544,1547,1548,1551],{},"设置 ",[14,1545,1546],{},"_intentionalClose = true"," 之后，所有重连逻辑都被 ",[14,1549,1550],{},"if (!this._intentionalClose)"," 短路。即使只是 Gateway 重启窗口碰好出现 token 短暂不匹配，也会一次性把客户端打死，必须刷新页面才能恢复。",[1230,1553,1555],{"id":1554},"路径-c快速重连配额耗尽","路径 C：快速重连配额耗尽",[10,1557,1558,1559,346],{},"定义了常数 ",[14,1560,1561],{},"MAX_RECONNECT_ATTEMPTS = 60",[72,1563,1565],{"className":1243,"code":1564,"language":1245,"meta":80,"style":80},"if (this._reconnectAttempts >= MAX_RECONNECT_ATTEMPTS) {\n  this._setConnected(false, 'error', `连接失败，已停止重连。请手动刷新页面重试。`)\n  return\n}\n",[14,1566,1567,1586,1609,1613],{"__ignoreMap":80},[516,1568,1569,1571,1573,1575,1578,1581,1584],{"class":518,"line":519},[516,1570,1424],{"class":1258},[516,1572,1279],{"class":537},[516,1574,1252],{"class":530},[516,1576,1577],{"class":537},"._reconnectAttempts ",[516,1579,1580],{"class":1258},">=",[516,1582,1583],{"class":530}," MAX_RECONNECT_ATTEMPTS",[516,1585,1440],{"class":537},[516,1587,1588,1590,1592,1594,1596,1598,1600,1602,1604,1607],{"class":518,"line":451},[516,1589,1362],{"class":530},[516,1591,1296],{"class":537},[516,1593,1367],{"class":522},[516,1595,1302],{"class":537},[516,1597,1372],{"class":530},[516,1599,1375],{"class":537},[516,1601,1378],{"class":526},[516,1603,1375],{"class":537},[516,1605,1606],{"class":526},"`连接失败，已停止重连。请手动刷新页面重试。`",[516,1608,1308],{"class":537},[516,1610,1611],{"class":518,"line":1290},[516,1612,1466],{"class":1258},[516,1614,1615],{"class":518,"line":1311},[516,1616,1396],{"class":537},[10,1618,1619],{},"客户机晚上挂机，Gateway 因为便携磁盘 GC 或 Windows 休眠争抢资源短暂掉线，客户端尝试 60 次仍未成功重连，就永久停摆。早上用户回来面板已成死链。",[61,1621,1622],{"id":1622},"为什么三条缺陷同时存在",[10,1624,1625],{},"这是逐步累加的历史债：",[85,1627,1628,1635,1638],{},[88,1629,1630,1631,1634],{},"路径 B 是在修复\"避免无限自动配对循环\"时添加的保护，但用错了对象——",[14,1632,1633],{},"_intentionalClose=true"," 本来是用户主动断开的语义，不该用在被动失败上",[88,1636,1637],{},"路径 C 是早期\"避免无穷重试\"的防护，但 60 次后完全放弃而不留任何复活路径是绝对错误",[88,1639,1640],{},"路径 A 是在\"凭据 reload\"改动时把所有重连都改走 refreshCredentials，没留意 catch 分支已经成了终态",[10,1642,1643],{},"三条各自独立、都能单独打死客户端，组合在一起命中率非常高。",[61,1645,1646],{"id":1646},"修复方案",[10,1648,1649],{},"核心思想是永不彻底放弃。任何\"快速重连配额耗尽\"的分支都转入慢轮询，留出窗口让用户改配置或等待 Gateway 恢复后能自动复连。",[10,1651,1652,1653,1656],{},"新增辅助方法 ",[14,1654,1655],{},"_schedulePoll(delayMs, kind)"," 作为慢轮询的触发器：",[72,1658,1660],{"className":1243,"code":1659,"language":1245,"meta":80,"style":80},"const AUTH_RETRY_LIMIT = 2\nconst SLOW_POLL_DELAY_AUTH = 60_000      \u002F\u002F 认证持续失败：1 分钟探一次\nconst SLOW_POLL_DELAY_GENERAL = 300_000  \u002F\u002F 一般持续失败：5 分钟探一次\n\n_schedulePoll(delayMs, kind) {\n  this._clearReconnectTimer()\n  this._reconnectAttempts = 0\n  if (kind === 'auth') this._authRetryCount = 0\n  this._reconnectState = 'scheduled'\n  this._pendingReconnect = true\n  this._reconnectTimer = setTimeout(() => {\n    this._reconnectTimer = null\n    if (this._intentionalClose) return\n    this._reconnectState = 'attempting'\n    if (kind === 'auth') {\n      this._refreshCredentialsAndReconnect(0)\n    } else {\n      this._doConnect()\n    }\n  }, delayMs)\n}\n",[14,1661,1662,1676,1691,1706,1711,1719,1730,1741,1765,1777,1790,1807,1819,1834,1846,1859,1875,1886,1898,1904,1910],{"__ignoreMap":80},[516,1663,1664,1667,1670,1673],{"class":518,"line":519},[516,1665,1666],{"class":1258},"const",[516,1668,1669],{"class":530}," AUTH_RETRY_LIMIT",[516,1671,1672],{"class":1258}," =",[516,1674,1675],{"class":530}," 2\n",[516,1677,1678,1680,1683,1685,1688],{"class":518,"line":451},[516,1679,1666],{"class":1258},[516,1681,1682],{"class":530}," SLOW_POLL_DELAY_AUTH",[516,1684,1672],{"class":1258},[516,1686,1687],{"class":530}," 60_000",[516,1689,1690],{"class":1521},"      \u002F\u002F 认证持续失败：1 分钟探一次\n",[516,1692,1693,1695,1698,1700,1703],{"class":518,"line":1290},[516,1694,1666],{"class":1258},[516,1696,1697],{"class":530}," SLOW_POLL_DELAY_GENERAL",[516,1699,1672],{"class":1258},[516,1701,1702],{"class":530}," 300_000",[516,1704,1705],{"class":1521},"  \u002F\u002F 一般持续失败：5 分钟探一次\n",[516,1707,1708],{"class":518,"line":1311},[516,1709,1710],{"emptyLinePlaceholder":465},"\n",[516,1712,1713,1716],{"class":518,"line":1317},[516,1714,1715],{"class":522},"_schedulePoll",[516,1717,1718],{"class":537},"(delayMs, kind) {\n",[516,1720,1721,1723,1725,1728],{"class":518,"line":1473},[516,1722,1362],{"class":530},[516,1724,1296],{"class":537},[516,1726,1727],{"class":522},"_clearReconnectTimer",[516,1729,1461],{"class":537},[516,1731,1732,1734,1736,1738],{"class":518,"line":1508},[516,1733,1362],{"class":530},[516,1735,1577],{"class":537},[516,1737,1259],{"class":1258},[516,1739,1740],{"class":530}," 0\n",[516,1742,1743,1745,1748,1751,1754,1757,1759,1761,1763],{"class":518,"line":1525},[516,1744,1276],{"class":1258},[516,1746,1747],{"class":537}," (kind ",[516,1749,1750],{"class":1258},"===",[516,1752,1753],{"class":526}," 'auth'",[516,1755,1756],{"class":537},") ",[516,1758,1252],{"class":530},[516,1760,1431],{"class":537},[516,1762,1259],{"class":1258},[516,1764,1740],{"class":530},[516,1766,1767,1769,1772,1774],{"class":518,"line":1537},[516,1768,1362],{"class":530},[516,1770,1771],{"class":537},"._reconnectState ",[516,1773,1259],{"class":1258},[516,1775,1776],{"class":526}," 'scheduled'\n",[516,1778,1780,1782,1785,1787],{"class":518,"line":1779},10,[516,1781,1362],{"class":530},[516,1783,1784],{"class":537},"._pendingReconnect ",[516,1786,1259],{"class":1258},[516,1788,1789],{"class":530}," true\n",[516,1791,1793,1795,1797,1799,1801,1803,1805],{"class":518,"line":1792},11,[516,1794,1362],{"class":530},[516,1796,1255],{"class":537},[516,1798,1259],{"class":1258},[516,1800,1262],{"class":522},[516,1802,1265],{"class":537},[516,1804,1268],{"class":1258},[516,1806,1271],{"class":537},[516,1808,1810,1812,1814,1816],{"class":518,"line":1809},12,[516,1811,1293],{"class":530},[516,1813,1255],{"class":537},[516,1815,1259],{"class":1258},[516,1817,1818],{"class":530}," null\n",[516,1820,1822,1825,1827,1829,1832],{"class":518,"line":1821},13,[516,1823,1824],{"class":1258},"    if",[516,1826,1279],{"class":537},[516,1828,1252],{"class":530},[516,1830,1831],{"class":537},"._intentionalClose) ",[516,1833,1540],{"class":1258},[516,1835,1837,1839,1841,1843],{"class":518,"line":1836},14,[516,1838,1293],{"class":530},[516,1840,1771],{"class":537},[516,1842,1259],{"class":1258},[516,1844,1845],{"class":526}," 'attempting'\n",[516,1847,1849,1851,1853,1855,1857],{"class":518,"line":1848},15,[516,1850,1824],{"class":1258},[516,1852,1747],{"class":537},[516,1854,1750],{"class":1258},[516,1856,1753],{"class":526},[516,1858,1440],{"class":537},[516,1860,1862,1865,1867,1869,1871,1873],{"class":518,"line":1861},16,[516,1863,1864],{"class":530},"      this",[516,1866,1296],{"class":537},[516,1868,1299],{"class":522},[516,1870,1302],{"class":537},[516,1872,1305],{"class":530},[516,1874,1308],{"class":537},[516,1876,1878,1881,1884],{"class":518,"line":1877},17,[516,1879,1880],{"class":537},"    } ",[516,1882,1883],{"class":1258},"else",[516,1885,1271],{"class":537},[516,1887,1889,1891,1893,1896],{"class":518,"line":1888},18,[516,1890,1864],{"class":530},[516,1892,1296],{"class":537},[516,1894,1895],{"class":522},"_doConnect",[516,1897,1461],{"class":537},[516,1899,1901],{"class":518,"line":1900},19,[516,1902,1903],{"class":537},"    }\n",[516,1905,1907],{"class":518,"line":1906},20,[516,1908,1909],{"class":537},"  }, delayMs)\n",[516,1911,1913],{"class":518,"line":1912},21,[516,1914,1396],{"class":537},[10,1916,1917,1918,1920,1921,1924],{},"慢轮询命中后失败会重新进入 ",[14,1919,1239],{}," 快速重连周期（因为 ",[14,1922,1923],{},"_reconnectAttempts"," 重置为 0），相当于\"快速 60 次 → 慢一次 → 快速 60 次 → 慢一次\"的循环，永不放弃。",[10,1926,1927],{},"三条修复并行：",[10,1929,1930,1933,1934,1937],{},[19,1931,1932],{},"Fix A","：普通重连直接走 ",[14,1935,1936],{},"_doConnect()","，凭据刷新隔离到认证失败分支，避免配置读取错误牵连普通重连。",[10,1939,1940,1943,1944,1947,1948,1950],{},[19,1941,1942],{},"Fix B","：认证失败耗尽后转入 ",[14,1945,1946],{},"_schedulePoll('auth')","，不再设置 ",[14,1949,1633],{},"，给认证恢复留出 60 秒的探测窗口。",[10,1952,1953,1956,1957,1960],{},[19,1954,1955],{},"Fix C","：重连次数超过 MAX_RECONNECT_ATTEMPTS 后转入 ",[14,1958,1959],{},"_schedulePoll('general')","，设置 UI 状态为\"连接持续失败，300 秒后重试\"而不是终态。",[10,1962,1963],{},"慢轮询失败后重新进入快速重连周期，形成\"快速 60 次 → 慢一次 → 快速 60 次\"的循环，永不放弃。",[61,1965,1966],{"id":1966},"防回归验证",[10,1968,1969,1970,1973],{},"新增 ",[14,1971,1972],{},"wakou-full\u002Fsrc\u002Flib\u002Fws-client.slow-poll.test.js","，覆盖 7 个 case：",[85,1975,1976,1984,1996,1999,2008,2015,2020],{},[88,1977,1978,1979,1981,1982],{},"Fix A：普通重连命中 ",[14,1980,1895],{},"，不调用 ",[14,1983,1299],{},[88,1985,1986,1987,1990,1991,1993,1994],{},"Fix C：MAX_RECONNECT_ATTEMPTS 后状态是 ",[14,1988,1989],{},"reconnecting","（非终态 ",[14,1992,194],{},"），5 分钟后触发 ",[14,1995,1895],{},[88,1997,1998],{},"Fix C：第二轮慢轮询失败后还能再调度快速重连",[88,2000,2001,2002,2004,2005],{},"Fix B：",[14,2003,1946],{}," 等待 60 秒调用 ",[14,2006,2007],{},"_refreshCredentialsAndReconnect(0)",[88,2009,2001,2010,2012,2013],{},[14,2011,1959],{}," 等待后调用 ",[14,2014,1895],{},[88,2016,2017,2019],{},[14,2018,1633],{}," 时慢轮询应跳过",[88,2021,2022,2024],{},[14,2023,1299],{}," 抛错路径调度慢轮询",[10,2026,2027],{},"测试全部通过（7\u002F7）。",[61,2029,2030],{"id":2030},"外部因素的叠加",[10,2032,2033],{},"这个时期同步发生了另一个外部问题：CDN 对 WebSocket 空闲连接会在 4-9 分钟后强制切断。客户端收到连接中断后根据指数退避策略重连，如果恰好命中上述三条路径之一，就进入永久断开状态。这解释了为什么故障特别在长时间挂机后高发——空闲足够长，CDN 必定切断，而后续重连很容易落入某条缺陷路径。两个问题各自独立，但组合效果是\"挂机过夜必死\"。",[61,2035,2036],{"id":2036},"排查验证",[10,2038,2039],{},"故障修复后，可以用以下方式验证重连状态：",[72,2041,2043],{"className":1243,"code":2042,"language":1245,"meta":80,"style":80},"__clawpanelWsClient.getConnectionInfo()\n\u002F\u002F {\n\u002F\u002F   connected: false,\n\u002F\u002F   reconnectState: 'scheduled' | 'attempting',\n\u002F\u002F   reconnectAttempts: 0~60,\n\u002F\u002F   ...\n\u002F\u002F }\n",[14,2044,2045,2055,2060,2065,2070,2075,2080],{"__ignoreMap":80},[516,2046,2047,2050,2053],{"class":518,"line":519},[516,2048,2049],{"class":537},"__clawpanelWsClient.",[516,2051,2052],{"class":522},"getConnectionInfo",[516,2054,1461],{"class":537},[516,2056,2057],{"class":518,"line":451},[516,2058,2059],{"class":1521},"\u002F\u002F {\n",[516,2061,2062],{"class":518,"line":1290},[516,2063,2064],{"class":1521},"\u002F\u002F   connected: false,\n",[516,2066,2067],{"class":518,"line":1311},[516,2068,2069],{"class":1521},"\u002F\u002F   reconnectState: 'scheduled' | 'attempting',\n",[516,2071,2072],{"class":518,"line":1317},[516,2073,2074],{"class":1521},"\u002F\u002F   reconnectAttempts: 0~60,\n",[516,2076,2077],{"class":518,"line":1473},[516,2078,2079],{"class":1521},"\u002F\u002F   ...\n",[516,2081,2082],{"class":518,"line":1508},[516,2083,2084],{"class":1521},"\u002F\u002F }\n",[10,2086,2087,2090,2091,2094,2095,2090,2098,2101],{},[14,2088,2089],{},"reconnectState='scheduled'"," 且 ",[14,2092,2093],{},"reconnectAttempts=0"," 表示在慢轮询窗口正常工作；",[14,2096,2097],{},"reconnectState='idle'",[14,2099,2100],{},"connected=false"," 是老的永久断开路径。",[10,2103,2104],{},"重连机制里避免静默终态是最基本的要求。任何异常路径都必须有可见的状态提示和后续操作入口，否则用户端看到的就是死机。这次故障的根本教训是，不能让客户端在任何情况下进入\"不可恢复\"的状态而没有任何提示。慢轮询的引入给了所有失败情景一个\"最后的机会\"，即使前面的快速重连机制彻底耗尽了，用户等待足够长的时间后系统仍有自动恢复的可能。",[872,2106,2107],{},"html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}",{"title":80,"searchDepth":451,"depth":451,"links":2109},[2110,2115,2116,2117,2118,2119],{"id":1225,"depth":451,"text":1225,"children":2111},[2112,2113,2114],{"id":1232,"depth":1290,"text":1233},{"id":1410,"depth":1290,"text":1411},{"id":1554,"depth":1290,"text":1555},{"id":1622,"depth":451,"text":1622},{"id":1646,"depth":451,"text":1646},{"id":1966,"depth":451,"text":1966},{"id":2030,"depth":451,"text":2030},{"id":2036,"depth":451,"text":2036},"2026-06-17","从 2026-05-11 开始，多台便携包客户反馈 WebSocket 断线后面板卡死。网关进程（openclaw_gateway.exe）明确在运行，\u002Fhealth 返回 200，但面板显示\"已停止重连，请手动刷新\"——用户必须手动重启才能恢复。",{},"\u002F2026-06-17-fa-016-ws",{"title":1195,"description":2121},"FA-016","2026-06-17-FA-016-WS永久断开三条路径","客户端重连逻辑中三处独立的\"静默终止\"缺陷导致 WS 永久断开，任何一条路径命中就不再调度重连计时器。",[2129,2130,2131,2132,2133],"WebSocket","Node.js","客户端重连机制","异常处理","便携包","qaISKpAwjYnhiSdxEMYYnJtJ-A8QaOq-djOEJ0W9-bM",1785406912030]