Skip to content

fix(procmon): retire audit events on EOE and keep receiving on overflow - #500

Closed
venkata-stepsecurity wants to merge 1 commit into
step-security:mainfrom
venkata-stepsecurity:venkat/fix-procmon-event-retention
Closed

venkata-stepsecurity wants to merge 1 commit into
step-security:mainfrom
venkata-stepsecurity:venkat/fix-procmon-event-retention

Conversation

@venkata-stepsecurity

Copy link
Copy Markdown
Collaborator

Fixes the agent memory growth reported in step-security/harden-runner#709.

Problem

ProcessMonitor.Events gets an entry for every audit sequence, and nothing ever removes one.

The audit rules include -S sendto -S sendmsg for every process. send() on a connected socket is audited as sendto with no SOCKADDR record, so these events never become ready and stay in the map forever. On a workload where short-lived processes stream results over localhost TCP, 99% of the retained entries were sendto events with no address, and the agent grew by about 1.4 GiB per minute until the runner ran out of memory.

A second problem showed up under the same load: receive returns on the first error. When the kernel's netlink socket overflows (ENOBUFS, "no buffer space available"), the loop exits and audit monitoring stops for the rest of the job, while the agent keeps running. MonitorProcesses discarded the error, so nothing was logged.

Changes

  • handleRecord (the loop body, moved out so it can be tested) removes an event when its EOE record arrives. The kernel ends every multi-record event with EOE, after all of its records, so dispatch is unchanged.
  • Events that never get an EOE, such as single-record events, are dropped once 8192 records arrive without touching them. The count is logged. This also bounds the map, since at most 8192 events can have been touched in the last 8192 records. PrepareEvent refreshes the event before sweeping and returns its own pointer, so the event being prepared is never dropped.
  • On ENOBUFS, the loss is counted, logged on 1, 2, 4, 8, ... occurrences, and receiving continues. Other errors still stop the loop, and MonitorProcesses now logs why.

Testing

  • New tests use real audit record text: sendto without an address is removed on EOE; connect, addressed sendto, execve and file-write events are each dispatched exactly once with their data; idle events are dropped, active events survive a sweep, and a late record during a sweep is handled; the receive loop keeps reading after ENOBUFS, using a fake netlink client. Each part of the fix was mutation-checked against the tests.
  • go test -race ./... on Linux with Go 1.27.3: all pass except Test_addAuditRules, which needs root and also fails on main.
  • End to end on GitHub-hosted ubuntu-latest, same workload, main and this branch built the same way:
main this branch
Agent RSS 47 MiB to 5.3 GiB, workload stopped at low memory 42 to 52 MiB, flat for 8 minutes
Probe detections (github.com, example.com at start, middle, end) 6 of 6 6 of 6
Source overwrite detections 2 2

Not covered

Agent CPU stays near 100% on this workload, because every send() is still audited and parsed. Reducing the sendto/sendmsg audit volume is a separate change.

Audit events were kept in ProcessMonitor.Events forever. send() on a
connected socket is audited as sendto without an address, so those
events never became ready and were never released. On a workload that
streams over localhost TCP the agent grew by about 1.4 GiB per minute
until the runner ran out of memory (harden-runner#709).

Remove an event when its EOE record arrives. Events that never get one
are dropped once 8192 records arrive without touching them, which also
bounds the map. When the netlink socket overflows (ENOBUFS), count the
loss and keep receiving instead of stopping audit monitoring for the
rest of the job, and log if the receive loop does stop.
@venkata-stepsecurity

Copy link
Copy Markdown
Collaborator Author

Closing: this approach was replaced by go-libaudit's Reassembler in #503, which goes to int first and will be cherry-picked to main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant