lib/mux: receiveWindow 数据竞争导致并发短连接场景下 NPC panic - #368
Open
everhopingandwaiting wants to merge 1 commit into
Open
lib/mux: receiveWindow 数据竞争导致并发短连接场景下 NPC panic#368everhopingandwaiting wants to merge 1 commit into
everhopingandwaiting wants to merge 1 commit into
Conversation
## Summary
`lib/mux` 的 receiveWindow 在数据转发读路径与连接关闭路径之间存在数据竞争。
并发短连接(连接→收发→关闭)下,NPC 工作池以 `slice bounds out of range` panic,
导致 mux 流损坏,直到客户端重连。
## Reproduction
nps + npc 同机、`secure_mode=false`、API 创建的 TCP 隧道指向本地 echo 服务。
8 轮 × 20 并发(共 160 连接)或 5 轮 × 50 并发(250 连接)短连接风暴后,npc.log 出现:
```
[ants]: worker exits from panic: runtime error: slice bounds out of range [253:62]
github.com/djylb/nps/lib/mux.(*receiveWindow).readFromQueue(...) conn_receive_window.go:225
```
触发率约 12%~24%(160 连/38 panics;250 连/31 panics)。
panic 后 mux 流损坏,该 NPC 下所有隧道报错(502 / SOCKS5 general failure / 空响应),
直到 NPC 按 5s 循环重连重建 mux。
## Root Cause 细节
`receiveWindow` 持有由单线程读路径独占的 `element`(含 `Buf []byte` / `L uint16`)。
两个 goroutine 可并发操作同一 element:
1. **读路径** `Read` → `readFromQueue`:读 `element.Buf`,耗尽后归还 `windowBuff` 池并弹出下一 element。
2. **关闭路径** `Conn.closeProcess` → `CloseWindow` → `release`:清空队列、归还 Buf/element、置 `element = nil`。
`release()` 与并发 `Read` 之间无任何同步。典型竞态:`Read` 正在拷贝
`Self.element.Buf` 时 `release()` 将该 Buf 归还池中,另一连接从池中取走并写入,
导致 `Buf[off:L]` 不一致(off > len(Buf))→ slice out of range。
并发 `Read`/`Read`(同一窗口被分发两次)同样可能破坏 `element`/`off`。
该 bug 存在于上游代码(master 至今相同,文件自 `f7ec854d` 起未改动)。
## Fix
`lib/mux/conn_receive_window.go` 最小改动(6 行):
```diff
type receiveWindow struct {
sizeMu sync.Mutex
+ readMu sync.Mutex
}
func (Self *receiveWindow) Read(p []byte, id int32) (n int, err error) {
+ Self.readMu.Lock()
+ defer Self.readMu.Unlock()
...
}
Self.element, err = Self.bufQueue.Pop()
Self.off = 0
if err != nil {
- Self.CloseWindow()
return
}
func (Self *receiveWindow) release() {
+ Self.readMu.Lock()
+ defer Self.readMu.Unlock()
...
}
```
- `Read` 与 `release()` 共用 `readMu`,串行化两条 element 持有路径。
- `readFromQueue` 出错时不再调用 `CloseWindow()`(会重入已持有的 `readMu` 造成死锁);资源清理交由必然执行的连接关闭路径。
- 无死锁:`CloseWindow` 先停止队列再让 `release()` 获取 `readMu`,阻塞在 `Pop()` 的 `Read` 会被 stop 信号唤醒并释放锁。
## Verification
- 修复后同一压力场景:**0 panic**(160 / 250 连接)。
- `go test -race ./lib/mux ./lib/goroutine` 通过。
- `go test ./lib/mux ./client` 通过。
- 端到端 API 套件(57 例)通过,0 panic。
## Impact
所有 NPC 承载并发短连接(HTTP 轮询、端口扫描、WebDAV、代理流量)的部署在高负载下都可能触发此 panic,
且 mux 损坏会影响该 NPC 的全部隧道直至重连。服务端 mux 代码相同,同样受影响。
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
lib/mux的 receiveWindow 在数据转发读路径与连接关闭路径之间存在数据竞争。 并发短连接(连接→收发→关闭)下,NPC 工作池以slice bounds out of rangepanic, 导致 mux 流损坏,直到客户端重连。Reproduction
nps + npc 同机、
secure_mode=false、API 创建的 TCP 隧道指向本地 echo 服务。 8 轮 × 20 并发(共 160 连接)或 5 轮 × 50 并发(250 连接)短连接风暴后,npc.log 出现:触发率约 12%~24%(160 连/38 panics;250 连/31 panics)。
panic 后 mux 流损坏,该 NPC 下所有隧道报错(502 / SOCKS5 general failure / 空响应), 直到 NPC 按 5s 循环重连重建 mux。
Root Cause 细节
receiveWindow持有由单线程读路径独占的element(含Buf []byte/L uint16)。 两个 goroutine 可并发操作同一 element:Read→readFromQueue:读element.Buf,耗尽后归还windowBuff池并弹出下一 element。Conn.closeProcess→CloseWindow→release:清空队列、归还 Buf/element、置element = nil。release()与并发Read之间无任何同步。典型竞态:Read正在拷贝Self.element.Buf时release()将该 Buf 归还池中,另一连接从池中取走并写入, 导致Buf[off:L]不一致(off > len(Buf))→ slice out of range。 并发Read/Read(同一窗口被分发两次)同样可能破坏element/off。该 bug 存在于上游代码(master 至今相同,文件自
f7ec854d起未改动)。Fix
lib/mux/conn_receive_window.go最小改动(6 行):type receiveWindow struct { sizeMu sync.Mutex + readMu sync.Mutex } func (Self *receiveWindow) Read(p []byte, id int32) (n int, err error) { + Self.readMu.Lock() + defer Self.readMu.Unlock() ... } Self.element, err = Self.bufQueue.Pop() Self.off = 0 if err != nil { - Self.CloseWindow() return } func (Self *receiveWindow) release() { + Self.readMu.Lock() + defer Self.readMu.Unlock() ... }Read与release()共用readMu,串行化两条 element 持有路径。readFromQueue出错时不再调用CloseWindow()(会重入已持有的readMu造成死锁);资源清理交由必然执行的连接关闭路径。CloseWindow先停止队列再让release()获取readMu,阻塞在Pop()的Read会被 stop 信号唤醒并释放锁。Verification
go test -race ./lib/mux ./lib/goroutine通过。go test ./lib/mux ./client通过。Impact
所有 NPC 承载并发短连接(HTTP 轮询、端口扫描、WebDAV、代理流量)的部署在高负载下都可能触发此 panic, 且 mux 损坏会影响该 NPC 的全部隧道直至重连。服务端 mux 代码相同,同样受影响。