Skip to content
typhdotcomPublic
forked from perkeep/perkeep

About

Perkeep (née Camlistore) is your personal storage system for life: a way of storing, syncing, sharing, modelling and backing up content.

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

 
 

Latest commit

 

History

7,568 Commits

Folders and files

Repository files navigation

Perkeep: fork

LLM-maintained and unaudited. This fork's changes and documentation are maintained with LLM agents and have not received an independent human audit.

This is a fork of perkeep/perkeep, a personal storage system: a way to store, sync, share, import, model, and back up content. The fork adds live query projections to the FUSE mount and the search websocket, and makes bulk ingest of small blobs fast. It is a stack of commits rebased onto upstream's master and is not coordinated with upstream.

Additions

  • Query nodes (pkg/fs/query.go): a permanode with camliNodeType query whose content is a JSON spec, mounted by pk-mount as a read-only projection of its results, followed live. Specs can name entries by attributes and render each result as a directory of its own sub-query. A query node placed in a writable directory by camliPath shows there as its results. See go doc perkeep.org/cmd/pk-mount.
  • pk watch <query>: one JSON line per change to a query's results.
  • Incremental search subscriptions over the search websocket, following claims along relation edges (pkg/search/incremental.go).
  • pk whoami signer prints the public key blobref claims are signed with.
  • pk-get restores directory and symlink times, and owners when run as root.

Fixes

  • Edges (edgesto, "what points here") come from every attribute whose value is a blobref, not only camliMember. A store indexed before this needs perkeepd -reindex once to find the older ones.
  • Localhost auth finds callers on dual-stack sockets, such as a JVM's, whose IPv4 connections Linux lists in /proc/net/tcp6.
  • Deleted permanodes stay out of unsorted node-type queries and followed queries.
  • A delete claim received before what it deletes, as pk sync into an empty server delivers them, takes effect once its target arrives rather than at the next restart.
  • A blob indexed before the blob it depends on reaches storage, which the index and storage being written concurrently allows, is indexed once storage has it instead of waiting forever. An index holding such waiters from before indexes them at the next start.
  • pk-put carries on without its have or stat cache when another process holds the cache's lock, instead of exiting.
  • An image too large for the index's ImageInfo is left unsized instead of failing its receive.
  • A long list of absolute ignored directories is looked up by ancestor, so pk-put stays cheap with many of them.
  • diskpacked receives that arrive together share one fsync (group commit) rather than syncing a blob at a time; each is still on disk before it is indexed or acknowledged. A failed fsync stops all further writes, since a retry could report success for pages the failure dropped.
  • A multipart upload receives its parts concurrently, so a batch shares a few fsyncs, and answers once all are stored. Its parts may be stored in any order, and on a failure any subset of them; a client that needs an order sends its batches in that order. It logs a line per upload, not per blob.
  • The indexer waits for its in-flight blobs to drain rather than sleeping 100ms, and skips the scan for missing-dependency rows of a blob never noted as missing one. leveldb tables are larger, kept open, and their index and filter blocks cached, so random lookups in a store of tens of millions of blobs stay off the disk. Together these take bulk ingest of small blobs from under a thousand a second to about twenty thousand.
  • The in-memory corpus keeps only schema blobs' metadata; the index answers for the rest, and a query over all blobs reads them from the index. A store of 16M small non-schema blobs starts in 6s at 0.6 GB rather than 20s at 3 GB.

Reporting and license

Reports about this fork's changes belong with the fork; establish whether an issue also occurs upstream before reporting it there.

The project retains upstream's Apache 2.0 license, copyright notices and author attribution. The LLM-maintenance notice applies to this fork's changes, not to upstream.

About

Perkeep (née Camlistore) is your personal storage system for life: a way of storing, syncing, sharing, modelling and backing up content.

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages