Skip to content

Stop resending metrics events the server has rejected - #76

Open
Tsyklop wants to merge 1 commit into
mainfrom
bug/ALTUX-3437-stop-resending-rejected-events
Open

Tsyklop wants to merge 1 commit into
mainfrom
bug/ALTUX-3437-stop-resending-rejected-events

Conversation

@Tsyklop

@Tsyklop Tsyklop commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

Fixes ALTUX-3437.

The metrics sender put a batch back into the queue after any failed send, including a 400 answer that can never succeed. The queue is saved to hyperskill.xml, so an install with invalid events kept resending them in a loop, ten requests of 1,000 events per round, and restarting the IDE didn't help.

Cause

Three problems fed each other:

  1. HyperskillMetricsScheduler.sendEvents requeued every failed batch, whatever the server answered.
  2. HyperskillMetricsService.allFrontendEvents took a snapshot by appending a placeholder HyperskillFrontendEvent() to the queue and reading up to it. When two reads overlapped (two sends, or a send and a state save), one took the other's placeholder as a real event. A placeholder has no action or route, so every batch containing it got a 400, went back to the queue and could be saved to disk. The same overlap also handed one event to several readers, so valid events were sent more than once.
  3. pluginLoaded started two more sending jobs for every plugin the IDE loaded dynamically, without checking which plugin it was. A long IDE session collects many jobs, which makes the overlap above frequent.

The size of the 400 answers fits this. 1,000 copies of {"action":["This field may not be null."],"route":["This field may not be null."]} with separators are exactly the 83,001 bytes seen in the logs, and a placeholder has valid client_time and context.

Fix

  • The connector returns a HyperskillEventsSendError instead of a plain string, so the scheduler can tell the cases apart:
    • Rejected (400, 413, 422): the batch is dropped and a warning is logged.
    • Failed (network errors, 401 after a failed token refresh, 403, 429, 5xx and other codes): the batch is kept for the next round. Token refresh works as before.
  • allFrontendEvents takes events with pollFirst() up to the queue size at the start, so nothing but real events is ever put into the queue.
  • loadState drops saved events without action or route, so installs that already have them recover after the update.
  • pluginLoaded starts sending only for this plugin, and HyperskillMetricsService allows the start only once.
  • Error text in the log is cut to 500 characters. A rejection used to write its whole answer, about 80 KB, to idea.log twice per request.
  • An exception in a sending job is logged instead of escaping. An escaping exception cancels all later runs of a periodic job, so events stopped being sent until the IDE restarted.

Behaviour changes

  • A batch answered with 400, 413 or 422 is lost, including any valid events in it. Invalid events are no longer created and are filtered out on load, so this should take a change in the server's validation.
  • Time-spent events follow the same rules as view events.

Testing

HyperskillMetricsTest has 7 new tests, and all 21 tests of the class pass on 263.

  • The overlap test reads the queue from four threads while a fifth adds 20,000 events. With the old snapshot code the readers took 61,268 events. Now each event is taken exactly once, and every event has an action and a route.
  • The class never logged in, so its sending tests returned early with "user logged out". They passed only when an earlier test in the same run had left a fake user logged in. The class now logs in in setUp.
  • I ran only this test class, and only on 263. I didn't compile the other platforms.
  • To build from main locally I had to apply the removal of the dead JetBrains Space repository (1c1390548, on another branch). Without it the Rust plugin doesn't resolve. That change isn't part of this PR.

A batch of metrics events went back to the queue after any failed send,
including a 400 that can never succeed, and the queue is saved to disk.
Two installs resent 10,000 invalid events in a loop.

- Drop a batch the server rejects (400, 413, 422). Keep retrying on
  network errors, 401, 403, 429 and 5xx. The connector now tells these
  cases apart instead of returning only an error string.
- Take events out of the queue with pollFirst instead of a placeholder
  marker. Overlapping readers took the marker as an event without action
  and route, which made every batch containing it fail, and handed the
  same events to several readers.
- Drop saved events without action or route when loading the state.
- Start the sender only once and only when this plugin is loaded:
  every plugin loaded dynamically used to add two more sending jobs.
- Log at most 500 characters of an error: a rejection lists one error
  per event, about 80 KB for a full batch.
- Keep a periodic job alive after an exception, which used to stop
  sending until the IDE restarted.

HyperskillMetricsTest now logs in a fake user. Before, its sending tests
passed only when an earlier test in the run had left a user logged in.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant