external-sort performs bounded external sorting of fixed-size opaque records
while encrypting every temporary record with AES-256-GCM. It is intended for
large reconciliation and migration datasets that cannot safely be retained in
memory or written to plaintext temporary files.
go get github.com/faustbrian/go-external-sort@v1See the compiler-checked package example for a complete runnable setup that creates a store, adds fixed-width records, consumes them in sorted order, and closes all owned resources.
The parent directory must already exist, must not be a symlink, and must have no group or other permission bits. Existing ancestor links are resolved when the factory is created. Opening a store binds the resolved directory to a rooted handle and revalidates its recorded identity and permissions; subsequent storage and cleanup remain relative to that handle. The key must contain exactly 32 bytes.
This repository contains one Go module,
github.com/faustbrian/go-external-sort, with the sole public root package
externalsort. It has no subpackages or optional adapter modules.
- fixed record size and explicit total-record limit;
- bounded contiguous in-memory chunks;
- at most 64 files in one merge;
- lexicographic byte ordering with duplicates preserved;
- a random-seeded process-unique nonce domain, a retry-safe record counter, and AES-256-GCM authentication for every temporary record;
- authentication of store identity, format version, chunk, ordinal, and record size;
- exact owner-only temporary modes (
0700directories and0600files), independent of process umask; - descriptor-relative storage and cleanup that cannot be redirected by a renamed or replaced parent pathname ancestor; and
- complete temporary-directory removal after a successful
Close.
Stores permit one active lifecycle operation. While open, overlapping or
reentrant calls fail with ErrConcurrentUse; callers can retry after the active
operation returns. Once closing begins, input and iteration return ErrClosed.
A record passed to the iteration callback is valid only until that callback
returns. Copy it when retention is required.
Use this module when the data is fixed-width, sorting must be bounded, and plaintext spill files are unacceptable. Prefer an in-memory sort for small, public datasets. This implementation deliberately rejects configurations that need more than 64 chunks instead of hiding an unbounded or multi-pass merge. Increase the chunk size within the declared byte ceiling or partition the dataset at a higher semantic layer.
Close removes process-owned artifacts, but no process can guarantee cleanup
after abrupt termination or host loss. Operators should place the parent on an
encrypted ephemeral filesystem and apply the descriptor-relative ownership and
age checks in the operations guide before removing stale
directories.
For shared package selection, ownership, and lifecycle guidance, see the versioned Golib ecosystem index and its Integration and data movement family.
- API and lifecycle
- Architecture and file format
- Adoption and migration
- Compatibility
- Threat model
- Performance
- Operations and Kubernetes
- FAQ
- Support
- Security policy
- Release notes
Use Go 1.27.2 for development; the public minimum remains Go 1.27.0.
Ordinary go test ./... works in a standalone checkout. make check and
make ci use bounded local checks with the public pinned verifier and a focused
race selection for lifecycle exclusion, cancellation and reader cleanup.
make check-full retains online specification checks and full configured
assurance. See CONTRIBUTING.md for focused
two-second fuzzing, optional root-package mutation and selected release checks.
MIT. See LICENSE.
Complete documentation map · Current source package map
Use the versioned ecosystem entry for package selection. The frozen compatibility cohort is separate from current package releases and unpublished source changes. Adopting them does not silently update that cohort.