Skip to content

fix(listen): stop reconnecting when a websocket pong is a little late - #473

Open
alexluong wants to merge 3 commits into
mainfrom
fix/websocket-keepalive
Open

alexluong wants to merge 3 commits into
mainfrom
fix/websocket-keepalive

Conversation

@alexluong

Copy link
Copy Markdown
Collaborator

hookdeck listen drops a healthy connection whenever the server's reply to a ping (pong) is more than about a second late. Users on slower or busier networks then see "Connection lost, reconnecting..." every few minutes.

Why: the client pings every 9s and drops the connection if no pong arrives within 10s of the previous one. That leaves about 1s of slack:

Before: ping every 9s, drop 10s after the last pong

  0s     pong received ──────────────── deadline 10s
  9s     ping sent
  10s    ✗ deadline hit, connection dropped   ("Connection lost, reconnecting...")
  10.5s  pong arrives, too late

After: ping every 10s, drop 30s after the last frame of any kind

  0s     pong received ─────────────────────────────────────── deadline 30s
  10s    ping sent
  10.5s  pong arrives ✓ deadline moves to 40.5s
Before After
Ping every 9s 10s
Drop after 10s without a pong 30s without any frame
Slack for a late pong ~1s ~20s
Incoming events keep it alive no yes
Old socket on reconnect left open until the server cleans it up closed

Changes (pkg/websocket/client.go):

  1. Pong wait 10s → 30s; ping every 10s.
  2. Any incoming frame renews the read deadline.
  3. Close the old socket before reconnecting.

Tests: a busy connection with late pongs stays up; the socket closes after a read timeout; new defaults. All three fail on the old code.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@leggetter leggetter left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Two small comments, neither blocking.


Generated by Claude Code

Comment thread pkg/websocket/client.go Outdated
// Close the old socket before reconnecting. Otherwise it stays open
// until the server's idle check notices it is dead.
c.stateMu.Lock()
_ = c.conn.Close()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add to this comment that it's deliberately not a 1000 close? On the server (handleSocketClose in core's websocketProxy.ts) a 1000 deletes the session and anything else keeps it for the reconnect. Would be easy for someone to "tidy" this into a ws.CloseMessage later and break reconnects.


Generated by Claude Code

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alexluong rather than a comment I've pushed 92f5223 to this branch. The close is now closeForReconnect(), next to Stop(), so the two exits read as a pair. TestReadTimeoutClosesSocket also asserts the server sees a 1006, so if someone swaps in a 1000 close the test fails rather than reconnects quietly breaking.


Generated by Claude Code

Comment thread pkg/websocket/client.go
defaultPongWait = 10 * time.Second
// defaultPongWait leaves room for a few late pongs (slow network, busy
// server) before the connection is treated as dead.
defaultPongWait = 30 * time.Second

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you consider 20s? With 10s pings that still covers one pong up to ~10s late.

The server never pings the CLI and only sweeps idle sockets every 30s, so when a connection dies silently it's the CLI's reconnect that stops events being sent to the dead socket. Those events end up as TIMEOUT. 30s means up to ~32s of that vs ~22s with 20s. Not a blocker if you think the extra tolerance is worth it.


Generated by Claude Code

Move the raw socket close used before reconnecting into closeForReconnect,
next to Stop, so the two exits read as a pair: Stop sends a 1000 close and
the server deletes the session; closeForReconnect sends no close frame, so
the server sees 1006 and keeps the session for the reconnect.

TestReadTimeoutClosesSocket now asserts the server sees an abnormal close,
so swapping in a clean close fails the test rather than breaking reconnects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FRc1axJoxMwiK5nm2q7YqP
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants