Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -695,7 +695,7 @@ subenum/
│ ├── assets/ # Brand, CSS, JS, TUI screenshot, demo clip
│ ├── _data/, _includes/, _layouts/
│ ├── ARCHITECTURE.md, CONTRIBUTING.md, DEVELOPER_GUIDE.md, ROADMAP.md
│ ├── start.md, cli.md, pipelines.md, library.md, labs.md, monitoring.md, docker.md
│ ├── start.md, cli.md, pipelines.md, library.md, labs.md, monitoring.md, docker.md, mcp.md
│ └── index.html # Landing page
├── examples/
│ ├── labs/ # Lab scenario zones and their wordlist
Expand Down
21 changes: 11 additions & 10 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ main_permute.go - -permute / -seeds: the permutation pass
main_resume.go - -resume / -state: save and replay an interrupted run
main_results.go - resultLog: a target's results, kept for -permute seeds and -diff
main_summary.go - assemble the run-quality report (-stats, JSONL summary line)
main_wordlist.go - readWordlist: the -w file, or the bundled list when -w is not given
main_wordlist.go - prepareWordlist: the -w file, or the bundled list when -w is not given
internal/scan/options.go - Options: user-facing settings shared by CLI and TUI (Validate, Config)
internal/scan/runner.go - Scan engine: Config, Event/EventKind/NoticeKind, Stats, Run
internal/scan/dispatcher.go - dispatcher: lazy candidate generation in chunks, budgets, resume watermark
Expand All @@ -74,6 +74,7 @@ internal/scan/verdict.go - Run-quality verdicts (Stats.Verdict, VerdictRan
internal/scan/verify.go - Verify: re-check names against the trusted resolver (used by -diff)
internal/dns/resolver.go - NewResolver, Lookup (ResolveTypes, Resolve, CheckWildcard,
FingerprintWildcard), Classify, ParseTypes
internal/dns/cname.go - lookupCNAME: a single CNAME query, without implicit A/AAAA lookups
internal/dns/transport.go - UDP (TCP fallback), DNS over TLS and DNS over HTTPS dialers
internal/dns/axfr.go - Transfer: an AXFR zone transfer over TCP (-axfr)
internal/dns/nsec.go - ProbeDenial: NSEC, NSEC3 or no DNSSEC proof of absence (-axfr)
Expand Down Expand Up @@ -121,7 +122,7 @@ tools/wordlist-gen.go - Standalone wordlist generator
* **Implementation**: `run()` only calls `runWith(runArgs)`, which takes the arguments, stdin, stdout and stderr as injected values so tests can drive the whole CLI in-process. Per-run state lives in a `session` (`main_session.go`) whose methods are the run's phases: `parse`, `configure`, `loadInputs`, `openOutputs`, `scanAll` (which calls `scanOneTarget` per target) and `finish`. `cliFlags` holds only parsed flag values; what the run derives from them (attempts, record types, excludes, the resolver pool, the `-max-duration` deadline) is in `runtimeInputs`, and each pass over a target is a `scanPass`. `runWith` first handles the `completion` and `man` subcommands and `-resume`, then builds a `flag.FlagSet` with `parseFlagsWith`, which binds every flag into a `cliFlags` struct and applies config-file and environment defaults (see section 5). The standard `flag` package stops at the first non-flag argument, so `cliargs.Parse` (`internal/cliargs`, shared with `tools/wordlist-gen.go`) re-parses around each positional; flags written after the domain are honored, and everything after a `--` terminator is positional. `runWith` then:
* handles `-tui` (launches `tui.Start` with the form pre-filled from the domain and the flags it has fields for, warning about the rest), `-print-config` and `-version` first;
* parses `-format` (`output.ParseFormat`), `-type` (`dns.ParseTypes`), and merges `-attempts` with the deprecated `-retries` (`effectiveAttempts`);
* calls `validateFlags`, which checks that `-w` and a target are present, that a positional domain and `-dL` are not both given, and that `-w` and `-dL` do not both read stdin (`-w` may repeat, but only one entry may be `-`). Numeric ranges and the resolver address are checked by `scan.Options.Validate`, the same validator the TUI uses. The positional domain is canonicalized with `validate.NormalizeDomain`.
* calls `validateFlags`, which checks that a target is present, that a positional domain and `-dL` are not both given, and that `-w` and `-dL` do not both read stdin (`-w` may repeat, but only one entry may be `-`). Numeric ranges and the resolver address are checked by `scan.Options.Validate`, the same validator the TUI uses. The positional domain is canonicalized with `validate.NormalizeDomain`.
* **Domain normalization**: `validate.NormalizeDomain` trims and lowercases the input, strips a URL scheme and path, a `:port`, a leading `*.` and a trailing dot (returning a note for each change), converts IDNs to punycode, and checks the result with `validate.Domain`. `validate.DNSServer` checks the resolver: `ip:port`, `tls://host[:port]` or `https://host/path`. Both are shared with the TUI.
* **Target list (`-dL`)**: `loadTargets` returns either the single positional domain or the entries of the `-dL` file (`-` for stdin). Blank lines, `#` comments and duplicates are skipped; invalid domains are reported and skipped. An empty result is an error.
* **Interactions**: `scanOptions` maps the parsed flags onto a `scan.Options`; its `Config` method produces the `scan.Config` passed to the scan engine.
Expand Down Expand Up @@ -155,7 +156,7 @@ tools/wordlist-gen.go - Standalone wordlist generator
* **Flagging**: every contradiction counts against the member; three take it out of rotation for the rest of the run. If every member is flagged, the trusted resolver answers alone.

Per-member counts appear in the run-quality report.
* **Lookups**: a `dns.Lookup` value names the resolver, per-type timeout, attempt count, record types and verbose logger, and the lookup helpers are its methods, so a swapped argument cannot compile. `Lookup.ResolveTypes` performs one lookup per requested record type (A and AAAA via `LookupIP`, CNAME via `LookupCNAME`, NS, MX and TXT via `LookupNS`, `LookupMX` and `LookupTXT`, an MX value being `<preference> <host>`; default `dns.DefaultTypes`, A and AAAA) and returns typed `dns.Record{Type, Value}` results, the time spent, and the most severe error. Each type gets its own timeout, so a slow A answer cannot starve AAAA or CNAME. Names are queried as absolute (trailing dot) so resolv.conf search suffixes never multiply queries. `Lookup.Logf` receives verbose per-lookup lines (the package never writes to stderr itself; `dns.StderrLogf` is the unsynchronized fallback).
* **Lookups**: a `dns.Lookup` value names the resolver, per-type timeout, attempt count, record types and verbose logger, and the lookup helpers are its methods, so a swapped argument cannot compile. `Lookup.ResolveTypes` looks up the requested record types concurrently, with one query per type (A and AAAA via `LookupIP`, CNAME via the package's own `lookupCNAME`, NS, MX and TXT via `LookupNS`, `LookupMX` and `LookupTXT`, an MX value being `<preference> <host>`; default `dns.DefaultTypes`, A and AAAA) and returns typed `dns.Record{Type, Value}` results, the time spent, and the most severe error. Each type gets its own timeout, so a slow A answer cannot starve AAAA or CNAME. Names are queried as absolute (trailing dot) so resolv.conf search suffixes never multiply queries. `Lookup.Logf` receives verbose per-lookup lines (the package never writes to stderr itself; `dns.StderrLogf` is the unsynchronized fallback).
* **Retries and classification**: `Lookup.Resolve` looks the name up `Lookup.Attempts` times at most (at least once) with a short linear backoff, and returns the records plus a `dns.Outcome`: `OutcomeFound`, `OutcomeNXDomain`, `OutcomeTimeout`, `OutcomeRefused`, `OutcomeOther`, or `OutcomeCanceled`. `dns.Classify` maps a resolver error onto an outcome. NXDOMAIN (including NODATA, a name with no records of the requested types) is definitive and not retried; any infrastructure failure on one type outranks a not-found answer on another, so a SERVFAIL is never mistaken for NXDOMAIN.
* **Rate limiting**: `dns.RateLimiter` hands out evenly spaced reservations; `dns.WithLimiter` attaches it to a context. `Lookup.ResolveTypes` reserves one slot per query a lookup is expected to send (one per type: CNAME is sent as its own query rather than through Go's `LookupCNAME`, which would add A and AAAA queries, #249) before that lookup's timeout starts. A per-lookup `wireBudget` travels on the context into the `Dial` hook: Go's resolver dials once per query, so any dial beyond the reserved slots (Go's own internal retries) waits for another slot. `-rate` therefore bounds queries on the wire, including retries, wildcard probes and the preflight. The same hook notices REFUSED responses, which Go reports only as "server misbehaving", and names the server actually dialed in errors.
* **Wildcard detection**: probes resolve random 32-character hex labels (`dns.RandomLabel`) under a domain for the scan's record types, each with the scan's `-attempts`, on the trusted resolver. If any probe resolves, the domain is a wildcard and the union of the answers is its fingerprint.
Expand Down Expand Up @@ -191,10 +192,10 @@ tools/wordlist-gen.go - Standalone wordlist generator
* **Purpose**: Thread-safe output that keeps stdout pipe-clean. Resolved subdomains go to stdout; everything else (progress, verbose diagnostics, notices, errors) goes to stderr.
* **Implementation**:
* `output.Writer` with mutex-protected methods, created with `output.New` (stdout plus an optional file) or `output.NewFile` (file only; used by the TUI so the alt-screen viewport keeps the terminal):
* `Emit` - writes one `output.Result` (the engine's `scan.Result` plus the CLI's `Change`, `PreviousRecords` and `Permutation`; one result type flows from engine to writers). In `text` format it prints `Found: <domain>` on a terminal, or the bare name when stdout is piped (`SetPlain`), and the bare name to the output file; `-show-records` (`SetShowRecords`) appends `TYPE=value` pairs. In `json` format results are buffered and written as a single array by `Finish`; `jsonl` streams one object per line; `csv` streams `subdomain,type,value` rows with a header, plus a `simulated` column under `-simulate`, a `takeover_candidate` column when `-type` includes `CNAME` or `NS`, and a `change` column with `-diff`. The format is selected with `-format text|json|jsonl|csv` (`output.ParseFormat`, default `text`).
* `Emit` - writes one `output.Result` (the engine's `scan.Result` plus the CLI's `Change`, `PreviousRecords` and `Permutation`; one result type flows from engine to writers). In `text` format it prints `Found: <domain>` on a terminal, or the bare name when stdout is piped (`SetPlain`), and the bare name to the output file; `-show-records` (`SetShowRecords`) appends `TYPE=value` pairs. In `json` format results are streamed as array elements as they arrive, and `Finish` closes the array; `jsonl` streams one object per line; `csv` streams `subdomain,type,value` rows with a header, plus a `simulated` column under `-simulate`, a `takeover_candidate` column when `-type` includes `CNAME` or `NS`, and a `change` column with `-diff`. The format is selected with `-format text|json|jsonl|csv` (`output.ParseFormat`, default `text`).
* `Progress` / `ProgressDone` - a carriage-return progress line on stderr, or whole lines when `SetProgressLines` is on.
* `Info` / `Error` - informational and error lines on stderr, cleared around the progress line.
* `Finish` - writes the buffered JSON array, flushes CSV, and returns the first error writing the output file.
* `Finish` - closes the streamed JSON array, flushes CSV, and returns the first error writing the output file.
* **Simulated output marking**: under `-simulate` every result is marked so invented names never pass for real recon output: text stdout prints `Found (SIMULATED): ...`, a text results file starts with a `# SIMULATED` header naming the seed (`SetSeed`), JSON and JSONL objects carry `"simulated": true` (`scan.Result.Simulated`, embedded in `output.Result`), and CSV gets a `simulated` column.
* **Atomic results file**: `output.CreateFile` returns an `output.File` that writes to a temporary file next to the target. `Close(true)` renames it over the target; `Close(false)`, or a failed flush or close, discards it. The CLI commits only if at least one scan finished, so a run that fails before scanning (preflight, wildcard abort, Ctrl+C) leaves an existing results file unchanged. Targets that are not regular files (such as `/dev/stdout`) are written directly.
* **Verbose Output** (`-v`): configuration summary, per-lookup resolution lines (through `Info`, serialized with the progress line), and final statistics.
Expand Down Expand Up @@ -247,10 +248,10 @@ Visually, this can be seen as:

### 4.1. User Input Errors

* **Missing Required Arguments**: When `-w` or a target (domain or `-dL`) is missing, the tool says what is missing, prints the usage message and flag list, and exits with status 2.
* **Missing Required Arguments**: When a target (domain or `-dL`) is missing, the tool says what is missing, prints the usage message and flag list, and exits with status 2.
* **Validation** (status 2 on failure): the tool validates:
* Concurrency, timeout, attempts and depth must be at least 1; `-rate`, `-rate-per-resolver`, `-max-queries` and `-max-duration` must be 0 (unlimited) or positive. These checks live in `scan.Options.Validate`.
* The DNS server must be a valid `ip:port` with an IP host and a port in 1-65535 (`validate.DNSServer`; not checked in simulation mode).
* The DNS server must be `ip:port` with an IP host and a port in 1-65535, `tls://host[:port]` (port 853 by default), or `https://host/path`. Encrypted transports also accept hostname hosts (`validate.DNSServer`; not checked in simulation mode).
* The target domain must conform to DNS naming rules after normalization (`validate.NormalizeDomain`, `validate.Domain`). In a `-dL` list, invalid domains are skipped with a message instead.
* Hit rate (simulation mode) must be 1-100.
* `-format` and `-type` must name supported values; `-attempts` and `-retries` cannot both be set.
Expand All @@ -265,7 +266,7 @@ Visually, this can be seen as:

* **Lookup Failure**: When a DNS lookup fails, the error is classified (`NXDomain`, `Timeout`, `Refused`, `Other`, `Canceled`) and counted. NXDOMAIN is a definitive negative and is not treated as an infrastructure failure; this bucket also covers NODATA (the name exists but has no records of the requested types, such as `-type CNAME` on a name with only A records). Lookups cut short by cancellation (`Canceled`) are not counted at all, so an interrupt cannot inflate the failure rate. Timeouts, refusals, and other errors feed the reliability guard. The CLI always prints the per-outcome breakdown after `EventDone`.
* **Timeout Handling**: The user-specified timeout (`-timeout` flag) limits each per-type lookup, starting only after its rate-limit slots are granted, so queueing behind `-rate` never turns into a timeout. A lookup that exceeds it is classified as a timeout and retried if attempts remain.
* **Resolver failures before scanning**: a failed preflight or an inconclusive wildcard check (without `-force`) emits `EventError` and ends the scan without `EventDone`; the target counts as failed.
* **Resolver failures before scanning**: a failed preflight or an inconclusive wildcard check (without `-force`) emits `EventError`, followed by exactly one `EventDone` with `Stopped` set; the target counts as failed.

### 4.4. Concurrency-Related Issues

Expand Down Expand Up @@ -328,7 +329,7 @@ With `-permute` (or `-seeds`, which implies it), each target that finished gets

### 5.5. Certificate Transparency Seeding

With `-ct`, `ct.Client.Names` makes one HTTPS request per target to crt.sh (no API key) and returns the names its certificates cover, relative to the target. `withCTEntries` appends those not already in the wordlist after the wordlist entries, so wordlist resume positions stay put. They are only candidates and are resolved like any other; those that resolve also seed `-permute`. A failed lookup is a warning, not a failed scan. A resumed run reuses the saved CT names so its candidate list is the same.
With `-ct`, `ct.Client.Names` makes one HTTPS request per target to crt.sh (no API key) and returns the names its certificates cover, relative to the target. `passiveNames` gathers the names, and `passive.Merge` appends those not already in the wordlist after the wordlist entries, so wordlist resume positions stay put. They are only candidates and are resolved like any other; those that resolve also seed `-permute`. A failed lookup is a warning, not a failed scan. A resumed run reuses the saved CT names so its candidate list is the same.

### 5.6. Lab Mode

Expand All @@ -340,7 +341,7 @@ With `-ct`, `ct.Client.Names` makes one HTTPS request per target to crt.sh (no A

### 5.8. Configuration Precedence

`main_config.go` fills flag defaults from a JSON config file (`$SUBENUM_CONFIG`, else `<user config dir>/subenum/config.json`) and from `SUBENUM_*` environment variables named after the flag (`-dns-server` is `SUBENUM_DNS_SERVER`). Precedence, highest first: command-line flag > environment > config file > built-in default. Mode and action flags (`-tui`, `-version`, `-print-config`, `-resume`, `-h`) are never taken from defaults. `-print-config` prints every setting's effective value and its source. An invalid config value is a usage error, except under `-version`, `-print-config` and `-tui`, which only warn so the value can be found and fixed. The TUI pre-fills its form from the same environment and config values.
`main_config.go` fills flag defaults from a JSON config file (`$SUBENUM_CONFIG`, else `<user config dir>/subenum/config.json`) and from `SUBENUM_*` environment variables named after the flag (`-dns-server` is `SUBENUM_DNS_SERVER`). Precedence, highest first: command-line flag > environment > config file > built-in default. Mode and action flags (`-tui`, `-version`, `-print-config`, `-dry-run`, `-resume`, `-h`) are never taken from defaults. `-print-config` prints every setting's effective value and its source. An invalid config value is a usage error, except under `-version`, `-print-config` and `-tui`, which only warn so the value can be found and fixed. The TUI pre-fills its form from the same environment and config values.

### 5.9. Go API (`pkg/subenum`)

Expand Down
Loading
Loading