LLM-maintained and unaudited. This fork's changes and documentation are maintained with LLM agents and have not received an independent human audit.
This is a fork of perkeep/perkeep, a
personal storage system: a way to store, sync, share, import, model, and back
up content. The fork adds live query projections to the FUSE mount and the
search websocket, and makes bulk ingest of small blobs fast. It is a stack of
commits rebased onto upstream's master and is not coordinated with upstream.
- Perkeep Home and Docs
- BUILDING: how to compile it (
go run make.go) - CONTRIBUTING: upstream's development process
- Query nodes (
pkg/fs/query.go): a permanode withcamliNodeTypequerywhose content is a JSON spec, mounted bypk-mountas a read-only projection of its results, followed live. Specs can name entries by attributes and render each result as a directory of its own sub-query. A query node placed in a writable directory bycamliPathshows there as its results. Seego doc perkeep.org/cmd/pk-mount. pk watch <query>: one JSON line per change to a query's results.- Incremental search subscriptions over the search websocket, following
claims along relation edges (
pkg/search/incremental.go). pk whoami signerprints the public key blobref claims are signed with.pk-getrestores directory and symlink times, and owners when run as root.
- Edges (
edgesto, "what points here") come from every attribute whose value is a blobref, not onlycamliMember. A store indexed before this needsperkeepd -reindexonce to find the older ones. - Localhost auth finds callers on dual-stack sockets, such as a JVM's, whose
IPv4 connections Linux lists in
/proc/net/tcp6. - Deleted permanodes stay out of unsorted node-type queries and followed queries.
- A delete claim received before what it deletes, as
pk syncinto an empty server delivers them, takes effect once its target arrives rather than at the next restart. - A blob indexed before the blob it depends on reaches storage, which the index and storage being written concurrently allows, is indexed once storage has it instead of waiting forever. An index holding such waiters from before indexes them at the next start.
pk-putcarries on without its have or stat cache when another process holds the cache's lock, instead of exiting.- An image too large for the index's
ImageInfois left unsized instead of failing its receive. - A long list of absolute ignored directories is looked up by ancestor, so
pk-putstays cheap with many of them. - diskpacked receives that arrive together share one fsync (group commit) rather than syncing a blob at a time; each is still on disk before it is indexed or acknowledged. A failed fsync stops all further writes, since a retry could report success for pages the failure dropped.
- A multipart upload receives its parts concurrently, so a batch shares a few fsyncs, and answers once all are stored. Its parts may be stored in any order, and on a failure any subset of them; a client that needs an order sends its batches in that order. It logs a line per upload, not per blob.
- The indexer waits for its in-flight blobs to drain rather than sleeping 100ms, and skips the scan for missing-dependency rows of a blob never noted as missing one. leveldb tables are larger, kept open, and their index and filter blocks cached, so random lookups in a store of tens of millions of blobs stay off the disk. Together these take bulk ingest of small blobs from under a thousand a second to about twenty thousand.
- The in-memory corpus keeps only schema blobs' metadata; the index answers for the rest, and a query over all blobs reads them from the index. A store of 16M small non-schema blobs starts in 6s at 0.6 GB rather than 20s at 3 GB.
Reports about this fork's changes belong with the fork; establish whether an issue also occurs upstream before reporting it there.
The project retains upstream's Apache 2.0 license, copyright notices and author attribution. The LLM-maintenance notice applies to this fork's changes, not to upstream.