From 01032b4898df5057c5df28528d02d4b39b5c00e5 Mon Sep 17 00:00:00 2001 From: Asahi Arataki Date: Sun, 15 Nov 2026 14:29:10 +0800 Subject: [PATCH] docs: align architecture and examples with current behavior --- README.md | 2 +- docs/ARCHITECTURE.md | 21 +++++++++++---------- docs/DEVELOPER_GUIDE.md | 6 +++--- docs/index.html | 4 ++-- docs/labs.md | 6 +++--- tools/README.md | 2 +- 6 files changed, 21 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index c0bedd8..f07b4f2 100644 --- a/README.md +++ b/README.md @@ -695,7 +695,7 @@ subenum/ │ ├── assets/ # Brand, CSS, JS, TUI screenshot, demo clip │ ├── _data/, _includes/, _layouts/ │ ├── ARCHITECTURE.md, CONTRIBUTING.md, DEVELOPER_GUIDE.md, ROADMAP.md -│ ├── start.md, cli.md, pipelines.md, library.md, labs.md, monitoring.md, docker.md +│ ├── start.md, cli.md, pipelines.md, library.md, labs.md, monitoring.md, docker.md, mcp.md │ └── index.html # Landing page ├── examples/ │ ├── labs/ # Lab scenario zones and their wordlist diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 79ff3a0..7134f71 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -63,7 +63,7 @@ main_permute.go - -permute / -seeds: the permutation pass main_resume.go - -resume / -state: save and replay an interrupted run main_results.go - resultLog: a target's results, kept for -permute seeds and -diff main_summary.go - assemble the run-quality report (-stats, JSONL summary line) -main_wordlist.go - readWordlist: the -w file, or the bundled list when -w is not given +main_wordlist.go - prepareWordlist: the -w file, or the bundled list when -w is not given internal/scan/options.go - Options: user-facing settings shared by CLI and TUI (Validate, Config) internal/scan/runner.go - Scan engine: Config, Event/EventKind/NoticeKind, Stats, Run internal/scan/dispatcher.go - dispatcher: lazy candidate generation in chunks, budgets, resume watermark @@ -74,6 +74,7 @@ internal/scan/verdict.go - Run-quality verdicts (Stats.Verdict, VerdictRan internal/scan/verify.go - Verify: re-check names against the trusted resolver (used by -diff) internal/dns/resolver.go - NewResolver, Lookup (ResolveTypes, Resolve, CheckWildcard, FingerprintWildcard), Classify, ParseTypes +internal/dns/cname.go - lookupCNAME: a single CNAME query, without implicit A/AAAA lookups internal/dns/transport.go - UDP (TCP fallback), DNS over TLS and DNS over HTTPS dialers internal/dns/axfr.go - Transfer: an AXFR zone transfer over TCP (-axfr) internal/dns/nsec.go - ProbeDenial: NSEC, NSEC3 or no DNSSEC proof of absence (-axfr) @@ -121,7 +122,7 @@ tools/wordlist-gen.go - Standalone wordlist generator * **Implementation**: `run()` only calls `runWith(runArgs)`, which takes the arguments, stdin, stdout and stderr as injected values so tests can drive the whole CLI in-process. Per-run state lives in a `session` (`main_session.go`) whose methods are the run's phases: `parse`, `configure`, `loadInputs`, `openOutputs`, `scanAll` (which calls `scanOneTarget` per target) and `finish`. `cliFlags` holds only parsed flag values; what the run derives from them (attempts, record types, excludes, the resolver pool, the `-max-duration` deadline) is in `runtimeInputs`, and each pass over a target is a `scanPass`. `runWith` first handles the `completion` and `man` subcommands and `-resume`, then builds a `flag.FlagSet` with `parseFlagsWith`, which binds every flag into a `cliFlags` struct and applies config-file and environment defaults (see section 5). The standard `flag` package stops at the first non-flag argument, so `cliargs.Parse` (`internal/cliargs`, shared with `tools/wordlist-gen.go`) re-parses around each positional; flags written after the domain are honored, and everything after a `--` terminator is positional. `runWith` then: * handles `-tui` (launches `tui.Start` with the form pre-filled from the domain and the flags it has fields for, warning about the rest), `-print-config` and `-version` first; * parses `-format` (`output.ParseFormat`), `-type` (`dns.ParseTypes`), and merges `-attempts` with the deprecated `-retries` (`effectiveAttempts`); - * calls `validateFlags`, which checks that `-w` and a target are present, that a positional domain and `-dL` are not both given, and that `-w` and `-dL` do not both read stdin (`-w` may repeat, but only one entry may be `-`). Numeric ranges and the resolver address are checked by `scan.Options.Validate`, the same validator the TUI uses. The positional domain is canonicalized with `validate.NormalizeDomain`. + * calls `validateFlags`, which checks that a target is present, that a positional domain and `-dL` are not both given, and that `-w` and `-dL` do not both read stdin (`-w` may repeat, but only one entry may be `-`). Numeric ranges and the resolver address are checked by `scan.Options.Validate`, the same validator the TUI uses. The positional domain is canonicalized with `validate.NormalizeDomain`. * **Domain normalization**: `validate.NormalizeDomain` trims and lowercases the input, strips a URL scheme and path, a `:port`, a leading `*.` and a trailing dot (returning a note for each change), converts IDNs to punycode, and checks the result with `validate.Domain`. `validate.DNSServer` checks the resolver: `ip:port`, `tls://host[:port]` or `https://host/path`. Both are shared with the TUI. * **Target list (`-dL`)**: `loadTargets` returns either the single positional domain or the entries of the `-dL` file (`-` for stdin). Blank lines, `#` comments and duplicates are skipped; invalid domains are reported and skipped. An empty result is an error. * **Interactions**: `scanOptions` maps the parsed flags onto a `scan.Options`; its `Config` method produces the `scan.Config` passed to the scan engine. @@ -155,7 +156,7 @@ tools/wordlist-gen.go - Standalone wordlist generator * **Flagging**: every contradiction counts against the member; three take it out of rotation for the rest of the run. If every member is flagged, the trusted resolver answers alone. Per-member counts appear in the run-quality report. - * **Lookups**: a `dns.Lookup` value names the resolver, per-type timeout, attempt count, record types and verbose logger, and the lookup helpers are its methods, so a swapped argument cannot compile. `Lookup.ResolveTypes` performs one lookup per requested record type (A and AAAA via `LookupIP`, CNAME via `LookupCNAME`, NS, MX and TXT via `LookupNS`, `LookupMX` and `LookupTXT`, an MX value being ` `; default `dns.DefaultTypes`, A and AAAA) and returns typed `dns.Record{Type, Value}` results, the time spent, and the most severe error. Each type gets its own timeout, so a slow A answer cannot starve AAAA or CNAME. Names are queried as absolute (trailing dot) so resolv.conf search suffixes never multiply queries. `Lookup.Logf` receives verbose per-lookup lines (the package never writes to stderr itself; `dns.StderrLogf` is the unsynchronized fallback). + * **Lookups**: a `dns.Lookup` value names the resolver, per-type timeout, attempt count, record types and verbose logger, and the lookup helpers are its methods, so a swapped argument cannot compile. `Lookup.ResolveTypes` looks up the requested record types concurrently, with one query per type (A and AAAA via `LookupIP`, CNAME via the package's own `lookupCNAME`, NS, MX and TXT via `LookupNS`, `LookupMX` and `LookupTXT`, an MX value being ` `; default `dns.DefaultTypes`, A and AAAA) and returns typed `dns.Record{Type, Value}` results, the time spent, and the most severe error. Each type gets its own timeout, so a slow A answer cannot starve AAAA or CNAME. Names are queried as absolute (trailing dot) so resolv.conf search suffixes never multiply queries. `Lookup.Logf` receives verbose per-lookup lines (the package never writes to stderr itself; `dns.StderrLogf` is the unsynchronized fallback). * **Retries and classification**: `Lookup.Resolve` looks the name up `Lookup.Attempts` times at most (at least once) with a short linear backoff, and returns the records plus a `dns.Outcome`: `OutcomeFound`, `OutcomeNXDomain`, `OutcomeTimeout`, `OutcomeRefused`, `OutcomeOther`, or `OutcomeCanceled`. `dns.Classify` maps a resolver error onto an outcome. NXDOMAIN (including NODATA, a name with no records of the requested types) is definitive and not retried; any infrastructure failure on one type outranks a not-found answer on another, so a SERVFAIL is never mistaken for NXDOMAIN. * **Rate limiting**: `dns.RateLimiter` hands out evenly spaced reservations; `dns.WithLimiter` attaches it to a context. `Lookup.ResolveTypes` reserves one slot per query a lookup is expected to send (one per type: CNAME is sent as its own query rather than through Go's `LookupCNAME`, which would add A and AAAA queries, #249) before that lookup's timeout starts. A per-lookup `wireBudget` travels on the context into the `Dial` hook: Go's resolver dials once per query, so any dial beyond the reserved slots (Go's own internal retries) waits for another slot. `-rate` therefore bounds queries on the wire, including retries, wildcard probes and the preflight. The same hook notices REFUSED responses, which Go reports only as "server misbehaving", and names the server actually dialed in errors. * **Wildcard detection**: probes resolve random 32-character hex labels (`dns.RandomLabel`) under a domain for the scan's record types, each with the scan's `-attempts`, on the trusted resolver. If any probe resolves, the domain is a wildcard and the union of the answers is its fingerprint. @@ -191,10 +192,10 @@ tools/wordlist-gen.go - Standalone wordlist generator * **Purpose**: Thread-safe output that keeps stdout pipe-clean. Resolved subdomains go to stdout; everything else (progress, verbose diagnostics, notices, errors) goes to stderr. * **Implementation**: * `output.Writer` with mutex-protected methods, created with `output.New` (stdout plus an optional file) or `output.NewFile` (file only; used by the TUI so the alt-screen viewport keeps the terminal): - * `Emit` - writes one `output.Result` (the engine's `scan.Result` plus the CLI's `Change`, `PreviousRecords` and `Permutation`; one result type flows from engine to writers). In `text` format it prints `Found: ` on a terminal, or the bare name when stdout is piped (`SetPlain`), and the bare name to the output file; `-show-records` (`SetShowRecords`) appends `TYPE=value` pairs. In `json` format results are buffered and written as a single array by `Finish`; `jsonl` streams one object per line; `csv` streams `subdomain,type,value` rows with a header, plus a `simulated` column under `-simulate`, a `takeover_candidate` column when `-type` includes `CNAME` or `NS`, and a `change` column with `-diff`. The format is selected with `-format text|json|jsonl|csv` (`output.ParseFormat`, default `text`). + * `Emit` - writes one `output.Result` (the engine's `scan.Result` plus the CLI's `Change`, `PreviousRecords` and `Permutation`; one result type flows from engine to writers). In `text` format it prints `Found: ` on a terminal, or the bare name when stdout is piped (`SetPlain`), and the bare name to the output file; `-show-records` (`SetShowRecords`) appends `TYPE=value` pairs. In `json` format results are streamed as array elements as they arrive, and `Finish` closes the array; `jsonl` streams one object per line; `csv` streams `subdomain,type,value` rows with a header, plus a `simulated` column under `-simulate`, a `takeover_candidate` column when `-type` includes `CNAME` or `NS`, and a `change` column with `-diff`. The format is selected with `-format text|json|jsonl|csv` (`output.ParseFormat`, default `text`). * `Progress` / `ProgressDone` - a carriage-return progress line on stderr, or whole lines when `SetProgressLines` is on. * `Info` / `Error` - informational and error lines on stderr, cleared around the progress line. - * `Finish` - writes the buffered JSON array, flushes CSV, and returns the first error writing the output file. + * `Finish` - closes the streamed JSON array, flushes CSV, and returns the first error writing the output file. * **Simulated output marking**: under `-simulate` every result is marked so invented names never pass for real recon output: text stdout prints `Found (SIMULATED): ...`, a text results file starts with a `# SIMULATED` header naming the seed (`SetSeed`), JSON and JSONL objects carry `"simulated": true` (`scan.Result.Simulated`, embedded in `output.Result`), and CSV gets a `simulated` column. * **Atomic results file**: `output.CreateFile` returns an `output.File` that writes to a temporary file next to the target. `Close(true)` renames it over the target; `Close(false)`, or a failed flush or close, discards it. The CLI commits only if at least one scan finished, so a run that fails before scanning (preflight, wildcard abort, Ctrl+C) leaves an existing results file unchanged. Targets that are not regular files (such as `/dev/stdout`) are written directly. * **Verbose Output** (`-v`): configuration summary, per-lookup resolution lines (through `Info`, serialized with the progress line), and final statistics. @@ -247,10 +248,10 @@ Visually, this can be seen as: ### 4.1. User Input Errors -* **Missing Required Arguments**: When `-w` or a target (domain or `-dL`) is missing, the tool says what is missing, prints the usage message and flag list, and exits with status 2. +* **Missing Required Arguments**: When a target (domain or `-dL`) is missing, the tool says what is missing, prints the usage message and flag list, and exits with status 2. * **Validation** (status 2 on failure): the tool validates: * Concurrency, timeout, attempts and depth must be at least 1; `-rate`, `-rate-per-resolver`, `-max-queries` and `-max-duration` must be 0 (unlimited) or positive. These checks live in `scan.Options.Validate`. - * The DNS server must be a valid `ip:port` with an IP host and a port in 1-65535 (`validate.DNSServer`; not checked in simulation mode). + * The DNS server must be `ip:port` with an IP host and a port in 1-65535, `tls://host[:port]` (port 853 by default), or `https://host/path`. Encrypted transports also accept hostname hosts (`validate.DNSServer`; not checked in simulation mode). * The target domain must conform to DNS naming rules after normalization (`validate.NormalizeDomain`, `validate.Domain`). In a `-dL` list, invalid domains are skipped with a message instead. * Hit rate (simulation mode) must be 1-100. * `-format` and `-type` must name supported values; `-attempts` and `-retries` cannot both be set. @@ -265,7 +266,7 @@ Visually, this can be seen as: * **Lookup Failure**: When a DNS lookup fails, the error is classified (`NXDomain`, `Timeout`, `Refused`, `Other`, `Canceled`) and counted. NXDOMAIN is a definitive negative and is not treated as an infrastructure failure; this bucket also covers NODATA (the name exists but has no records of the requested types, such as `-type CNAME` on a name with only A records). Lookups cut short by cancellation (`Canceled`) are not counted at all, so an interrupt cannot inflate the failure rate. Timeouts, refusals, and other errors feed the reliability guard. The CLI always prints the per-outcome breakdown after `EventDone`. * **Timeout Handling**: The user-specified timeout (`-timeout` flag) limits each per-type lookup, starting only after its rate-limit slots are granted, so queueing behind `-rate` never turns into a timeout. A lookup that exceeds it is classified as a timeout and retried if attempts remain. -* **Resolver failures before scanning**: a failed preflight or an inconclusive wildcard check (without `-force`) emits `EventError` and ends the scan without `EventDone`; the target counts as failed. +* **Resolver failures before scanning**: a failed preflight or an inconclusive wildcard check (without `-force`) emits `EventError`, followed by exactly one `EventDone` with `Stopped` set; the target counts as failed. ### 4.4. Concurrency-Related Issues @@ -328,7 +329,7 @@ With `-permute` (or `-seeds`, which implies it), each target that finished gets ### 5.5. Certificate Transparency Seeding -With `-ct`, `ct.Client.Names` makes one HTTPS request per target to crt.sh (no API key) and returns the names its certificates cover, relative to the target. `withCTEntries` appends those not already in the wordlist after the wordlist entries, so wordlist resume positions stay put. They are only candidates and are resolved like any other; those that resolve also seed `-permute`. A failed lookup is a warning, not a failed scan. A resumed run reuses the saved CT names so its candidate list is the same. +With `-ct`, `ct.Client.Names` makes one HTTPS request per target to crt.sh (no API key) and returns the names its certificates cover, relative to the target. `passiveNames` gathers the names, and `passive.Merge` appends those not already in the wordlist after the wordlist entries, so wordlist resume positions stay put. They are only candidates and are resolved like any other; those that resolve also seed `-permute`. A failed lookup is a warning, not a failed scan. A resumed run reuses the saved CT names so its candidate list is the same. ### 5.6. Lab Mode @@ -340,7 +341,7 @@ With `-ct`, `ct.Client.Names` makes one HTTPS request per target to crt.sh (no A ### 5.8. Configuration Precedence -`main_config.go` fills flag defaults from a JSON config file (`$SUBENUM_CONFIG`, else `/subenum/config.json`) and from `SUBENUM_*` environment variables named after the flag (`-dns-server` is `SUBENUM_DNS_SERVER`). Precedence, highest first: command-line flag > environment > config file > built-in default. Mode and action flags (`-tui`, `-version`, `-print-config`, `-resume`, `-h`) are never taken from defaults. `-print-config` prints every setting's effective value and its source. An invalid config value is a usage error, except under `-version`, `-print-config` and `-tui`, which only warn so the value can be found and fixed. The TUI pre-fills its form from the same environment and config values. +`main_config.go` fills flag defaults from a JSON config file (`$SUBENUM_CONFIG`, else `/subenum/config.json`) and from `SUBENUM_*` environment variables named after the flag (`-dns-server` is `SUBENUM_DNS_SERVER`). Precedence, highest first: command-line flag > environment > config file > built-in default. Mode and action flags (`-tui`, `-version`, `-print-config`, `-dry-run`, `-resume`, `-h`) are never taken from defaults. `-print-config` prints every setting's effective value and its source. An invalid config value is a usage error, except under `-version`, `-print-config` and `-tui`, which only warn so the value can be found and fixed. The TUI pre-fills its form from the same environment and config values. ### 5.9. Go API (`pkg/subenum`) diff --git a/docs/DEVELOPER_GUIDE.md b/docs/DEVELOPER_GUIDE.md index d9e0333..f6842b3 100644 --- a/docs/DEVELOPER_GUIDE.md +++ b/docs/DEVELOPER_GUIDE.md @@ -31,7 +31,7 @@ To work with `subenum`, you'll need: To build the project, run: ```bash - # Standard build (version fallback is the `Version` var in main.go) + # Standard build (reports dev without an injected or module version) go build -buildvcs=false # Inject the git tag into the binary @@ -88,7 +88,7 @@ subenum/ │ ├── DOCUMENTATION_STRUCTURE.md │ ├── ROADMAP.md # Planned work │ ├── docker.md # Container setup and volume mounting -│ ├── start.md, cli.md, pipelines.md, library.md, labs.md, monitoring.md # User guides +│ ├── start.md, cli.md, pipelines.md, library.md, labs.md, monitoring.md, mcp.md # User guides │ ├── _config.yml # Jekyll config for GitHub Pages │ ├── _data/nav.yml # Site navigation: sidebar, footer, search, reading order │ ├── _includes/, _layouts/, assets/ # Jekyll site templates, CSS, JS, images @@ -387,6 +387,6 @@ The following capabilities are implemented and available today: See `docs/ROADMAP.md` for the next-pass list. Areas still open: -* **Additional record types**: `Lookup.ResolveTypes` covers A, AAAA, CNAME, NS, MX and TXT; a new type is a case in `lookupType`, an entry in `supportedTypes` and `queriesPerType`, and (to test it) an answer builder in `internal/dnsserver` and a record type in `internal/labzone`. +* **Additional record types**: `Lookup.ResolveTypes` covers A, AAAA, CNAME, NS, MX and TXT; a new type is a case in `lookupType`, an entry in `typeOrder` and `queriesPerType`, and (to test it) an answer builder in `internal/dnsserver` and a record type in `internal/labzone`. When working on new features, please update the documentation accordingly and add tests to cover the new functionality. \ No newline at end of file diff --git a/docs/index.html b/docs/index.html index 36512b3..96575a8 100644 --- a/docs/index.html +++ b/docs/index.html @@ -76,7 +76,7 @@

What you can rely on.

- + @@ -99,7 +99,7 @@

What you can rely on.

See it on a lab zone.

Two of the labs, recorded as they run. Lab 1 lists each host with its records and flags a CNAME that points at a name nobody owns:

-
old.example.com CNAME=old-app.herokuapp.com TAKEOVER?=dangling:heroku
+
blog.lab.example CNAME=ghost-lab.example.net TAKEOVER?=dangling

In lab 2 every name resolves, so subenum reports the wildcard instead of thirty false hits; with -force it filters the wildcard's answers and keeps the three real hosts.

The answers come from a DNS server on your own machine, so you can try the same commands with no network traffic.

diff --git a/docs/labs.md b/docs/labs.md index f7e784b..d1f6770 100644 --- a/docs/labs.md +++ b/docs/labs.md @@ -201,9 +201,9 @@ documentation ranges (`192.0.2.0/24`, `198.51.100.0/24`, `203.0.113.0/24`, `2001:db8::/32`) and reserved names such as `lab.example`, so nothing in a lab points at a real host. -Scenarios work with every other flag, such as `-dL` (each listed domain gets +Scenarios work with the other flags, such as `-dL` (each listed domain gets the same zone), `-permute`, `-diff`, `-format json` and `-stats`, which records the scenario file under `zone`. `-simulate-zone` cannot be combined -with `-simulate`, `-dns-server`, `-r` or `-ct`, whether they come from the command -line, a `SUBENUM_*` variable or the config file: a lab run refuses to start +with `-simulate`, `-dns-server`, `-r`, `-ct`, `-sources` or `-axfr`, whether they +come from the command line, a `SUBENUM_*` variable or the config file: a lab run refuses to start rather than send a query off the machine. diff --git a/tools/README.md b/tools/README.md index 196cda8..6cd4d7d 100644 --- a/tools/README.md +++ b/tools/README.md @@ -29,7 +29,7 @@ go build -buildvcs=false -o wordlist-gen wordlist-gen.go ./wordlist-gen -domain company-name.com -o company-wordlist.txt # Add custom combinations -./wordlist-gen -domain company-name.com -combine "dev,test,staging" -o company-wordlist.txt +./wordlist-gen -domain company-name.com -combine "dev,test,staging" -f -o company-wordlist.txt ``` ### Parameters
ClaimWhat it meansCheck it
Resolver pools that can't lie to youSpread queries across many resolvers. Failing ones are benched, every hit is re-checked against your trusted resolver before it's printed, and canary checks catch a resolver that hides names that exist.-r resolvers.txt -dns-server 9.9.9.9
Resolver pools that can't lie to youSpread queries across many resolvers. Failing ones are benched, every hit is re-checked against your trusted resolver before it's printed, and canary checks catch a resolver that hides names that exist.-r resolvers.txt -dns-server 9.9.9.9:53
Wildcards, including rotating onessubenum fingerprints wildcard answers before the scan and learns CDN pools that rotate. An inconclusive check is reported as inconclusive, never as “no wildcard”."wildcard_filtered": 112
A rate limit you can quote-rate caps packets on the wire, counting retries and every record type, so the number in your rules of engagement is the number you send."achieved_qps": 198.6
Scope that holdsExcluded names are never queried, not filtered afterwards. Exact names and *.parent patterns, inline or from a file.-exclude '*.corp.example.com'