Repository navigation
Conversation
A batch of metrics events went back to the queue after any failed send, including a 400 that can never succeed, and the queue is saved to disk. Two installs resent 10,000 invalid events in a loop. - Drop a batch the server rejects (400, 413, 422). Keep retrying on network errors, 401, 403, 429 and 5xx. The connector now tells these cases apart instead of returning only an error string. - Take events out of the queue with pollFirst instead of a placeholder marker. Overlapping readers took the marker as an event without action and route, which made every batch containing it fail, and handed the same events to several readers. - Drop saved events without action or route when loading the state. - Start the sender only once and only when this plugin is loaded: every plugin loaded dynamically used to add two more sending jobs. - Log at most 500 characters of an error: a rejection lists one error per event, about 80 KB for a full batch. - Keep a periodic job alive after an exception, which used to stop sending until the IDE restarted. HyperskillMetricsTest now logs in a fake user. Before, its sending tests passed only when an earlier test in the run had left a user logged in.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes ALTUX-3437.
The metrics sender put a batch back into the queue after any failed send, including a 400 answer that can never succeed. The queue is saved to
hyperskill.xml, so an install with invalid events kept resending them in a loop, ten requests of 1,000 events per round, and restarting the IDE didn't help.Cause
Three problems fed each other:
HyperskillMetricsScheduler.sendEventsrequeued every failed batch, whatever the server answered.HyperskillMetricsService.allFrontendEventstook a snapshot by appending a placeholderHyperskillFrontendEvent()to the queue and reading up to it. When two reads overlapped (two sends, or a send and a state save), one took the other's placeholder as a real event. A placeholder has noactionorroute, so every batch containing it got a 400, went back to the queue and could be saved to disk. The same overlap also handed one event to several readers, so valid events were sent more than once.pluginLoadedstarted two more sending jobs for every plugin the IDE loaded dynamically, without checking which plugin it was. A long IDE session collects many jobs, which makes the overlap above frequent.The size of the 400 answers fits this. 1,000 copies of
{"action":["This field may not be null."],"route":["This field may not be null."]}with separators are exactly the 83,001 bytes seen in the logs, and a placeholder has validclient_timeandcontext.Fix
HyperskillEventsSendErrorinstead of a plain string, so the scheduler can tell the cases apart:Rejected(400, 413, 422): the batch is dropped and a warning is logged.Failed(network errors, 401 after a failed token refresh, 403, 429, 5xx and other codes): the batch is kept for the next round. Token refresh works as before.allFrontendEventstakes events withpollFirst()up to the queue size at the start, so nothing but real events is ever put into the queue.loadStatedrops saved events withoutactionorroute, so installs that already have them recover after the update.pluginLoadedstarts sending only for this plugin, andHyperskillMetricsServiceallows the start only once.idea.logtwice per request.Behaviour changes
Testing
HyperskillMetricsTesthas 7 new tests, and all 21 tests of the class pass on 263.setUp.mainlocally I had to apply the removal of the dead JetBrains Space repository (1c1390548, on another branch). Without it the Rust plugin doesn't resolve. That change isn't part of this PR.