Repository navigation
fix(procmon): retire audit events on EOE and keep receiving on overflow - #500
Closed
venkata-stepsecurity wants to merge 1 commit into
Closed
venkata-stepsecurity wants to merge 1 commit into
venkata-stepsecurity wants to merge 1 commit into
Conversation
Audit events were kept in ProcessMonitor.Events forever. send() on a connected socket is audited as sendto without an address, so those events never became ready and were never released. On a workload that streams over localhost TCP the agent grew by about 1.4 GiB per minute until the runner ran out of memory (harden-runner#709). Remove an event when its EOE record arrives. Events that never get one are dropped once 8192 records arrive without touching them, which also bounds the map. When the netlink socket overflows (ENOBUFS), count the loss and keep receiving instead of stopping audit monitoring for the rest of the job, and log if the receive loop does stop.
Collaborator
Author
|
Closing: this approach was replaced by go-libaudit's Reassembler in #503, which goes to int first and will be cherry-picked to main. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the agent memory growth reported in step-security/harden-runner#709.
Problem
ProcessMonitor.Eventsgets an entry for every audit sequence, and nothing ever removes one.The audit rules include
-S sendto -S sendmsgfor every process.send()on a connected socket is audited assendtowith noSOCKADDRrecord, so these events never become ready and stay in the map forever. On a workload where short-lived processes stream results over localhost TCP, 99% of the retained entries weresendtoevents with no address, and the agent grew by about 1.4 GiB per minute until the runner ran out of memory.A second problem showed up under the same load:
receivereturns on the first error. When the kernel's netlink socket overflows (ENOBUFS, "no buffer space available"), the loop exits and audit monitoring stops for the rest of the job, while the agent keeps running.MonitorProcessesdiscarded the error, so nothing was logged.Changes
handleRecord(the loop body, moved out so it can be tested) removes an event when itsEOErecord arrives. The kernel ends every multi-record event withEOE, after all of its records, so dispatch is unchanged.EOE, such as single-record events, are dropped once 8192 records arrive without touching them. The count is logged. This also bounds the map, since at most 8192 events can have been touched in the last 8192 records.PrepareEventrefreshes the event before sweeping and returns its own pointer, so the event being prepared is never dropped.ENOBUFS, the loss is counted, logged on 1, 2, 4, 8, ... occurrences, and receiving continues. Other errors still stop the loop, andMonitorProcessesnow logs why.Testing
sendtowithout an address is removed onEOE;connect, addressedsendto,execveand file-write events are each dispatched exactly once with their data; idle events are dropped, active events survive a sweep, and a late record during a sweep is handled; the receive loop keeps reading afterENOBUFS, using a fake netlink client. Each part of the fix was mutation-checked against the tests.go test -race ./...on Linux with Go 1.27.3: all pass exceptTest_addAuditRules, which needs root and also fails onmain.ubuntu-latest, same workload,mainand this branch built the same way:mainNot covered
Agent CPU stays near 100% on this workload, because every
send()is still audited and parsed. Reducing thesendto/sendmsgaudit volume is a separate change.