Skip to content

Add batch_observe methods to Histogram and Summary - #1228

Open
MahmoudAshraf97 wants to merge 1 commit into
prometheus:masterfrom
MahmoudAshraf97:batched_observe
Open

MahmoudAshraf97 wants to merge 1 commit into
prometheus:masterfrom
MahmoudAshraf97:batched_observe

Conversation

@MahmoudAshraf97

Copy link
Copy Markdown

This PR adds batch methods for observing a list of values that become available at once.

Motivation

The use case for this is in high-throughput ML inference systems where some metrics are calculated for the whole batch at once such as queue time spent per element, we usually looped over the elements and used Histogram.observe, but each call involved python loop overhead and acquired two locks per call

Speedup

Python 3.12.3, x86_64 (AMD EPYC 9334), default buckets, best of 5. Loop = n calls to observe().

Single process (MutexValue)

n Summary loop Summary batch speedup Histogram loop Histogram batch speedup
16 6.7 us 0.7 us 9.9x 11.1 us 3.5 us 3.2x
64 26.0 us 1.0 us 25.3x 45.4 us 7.6 us 6.0x
256 105.1 us 2.6 us 41.1x 182.6 us 23.0 us 8.0x
1024 416.8 us 8.5 us 49.1x 729.8 us 87.9 us 8.3x

Multiprocess mode (MultiProcessValue, mmap write per inc)

n Summary loop Summary batch speedup Histogram loop Histogram batch speedup
16 23.9 us 1.8 us 13.4x 29.6 us 8.1 us 3.7x
256 382.0 us 3.8 us 100.4x 477.6 us 28.7 us 16.6x

It might seem that despite the large speedup, it is still in order of microseconds, but in our use case, these microseconds block the asyncio event loop for hundreds of other tasks, and these batched methods increased the total system throughput by 2% just by improving observability tools, which is meaningful given the current GPU prices

I'm also open to extend the regular observe methods to accept lists or sequences, but this is left for the maintainers to decide

TIA

Signed-off-by: MahmoudAshraf97 <hassouna97.ma@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant