Skip to content

Bound the client cache and drop expired entries - #28

Closed
bfs2021 wants to merge 1 commit into
priompt:mainfrom
bfs2021:fix/bound-cache-expiry-eviction
Closed

bfs2021 wants to merge 1 commit into
priompt:mainfrom
bfs2021:fix/bound-cache-expiry-eviction

Conversation

@bfs2021

@bfs2021 bfs2021 commented Nov 14, 2026

Copy link
Copy Markdown

Problem

With cache_ttl set, entries in PromptClient._cache are only skipped once expired, never removed, and the cache has no size limit. A long-lived process that touches many URIs or list() prefixes (for example one URI per customer) therefore retains every response it has ever fetched. Fixes #26.

Root cause

get() and list() store (value, expiry) in a plain dict keyed by the URI / ("list", prefix). On a lookup after expiry the code falls through and overwrites the same key, and the only deletion anywhere is the pop for an accepted subscription reload in subscribe(). Distinct keys that are never queried again have no removal path at all.

Fix

  • Keep the cache in an OrderedDict and sweep expired entries whenever a new entry is stored, covering both get() and list() key shapes. Removing only on lookup was not enough: the reproduction inserts 10,000 distinct URIs, so no key is ever looked up twice.
  • Add cache_max_entries (default 1024) and evict the least recently used entry when the bound is exceeded; a hit moves the key to the end.
  • Guard cache operations with a threading.Lock: lookups are now read-modify-write (delete expired + move_to_end), and subscribe()'s handler pops the cache from its own thread.
  • Document the new parameter in the README.

Testing

  • pytest -q: 30 passed.
  • Added test_expired_entries_are_removed_for_get_and_list (fails on main: 101 entries retained, passes with the sweep) and test_cache_is_bounded_and_evicts_least_recently_used (LRU order plus a 10-insert run staying at the bound).
  • Existing cache, pinned-read, key-collision, subscription-reload and concurrent-read tests all still pass.

The cache only ever grew: expired entries were skipped on lookup but never
removed, and nothing capped the number of keys, so a long-lived process
touching many URIs or list() prefixes held every response for its life.

Prune expired entries on store (covering both get() and list() keys) and
keep the cache in an OrderedDict as a bounded LRU (cache_max_entries,
default 1024). Cache operations take a lock because lookups are now
read-modify-write and subscribe()'s handler touches the cache from its
own thread.

Signed-off-by: bfs2021 <101915083+bfs2021@users.noreply.github.com>
@sujal-maheshwari2004

Copy link
Copy Markdown
Member

Hi @bfs2021, thank you for taking the time to send this. We're closing it, and it's not about the quality of the change.

The issue it fixes is labelled good first issue. In this organisation those are reserved for people making their first open-source contributions: we're onboarding a group of students who will learn the workflow on exactly these issues. We've now written this down in our contribution guide.

If you'd like to keep contributing, we'd welcome help on our help wanted issues. That's where experienced and AI-assisted effort makes a real difference. Please claim one by commenting, wait to be assigned before opening a pull request, and keep to at most two open pull requests at a time.

Thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Client cache never evicts expired entries

2 participants