diff --git a/CLAUDE.md b/CLAUDE.md index f4425d4..769a9cf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -15,10 +15,27 @@ committed spec is the full transcript: `docs/design/specs/v2-l1-emission.md`. - **`application.id`** = `can://` (app outermost, so a polyglot app shares one prefix), where `` is the **`--app-name`** value, defaulting to the input directory's base name (per the CLI contract; the SDK's Neo4j backend must use the same anchor). Module ids are - `can:///go/`; externals are language-neutral `can:///@external//`. + `can:///go/`; externals are language-neutral `can:///@external///` (see § Id member segments). L3/L4 build ids on this unchanged. -- **Callable signature** = existing `signatureOf()` output as the last `can://` path - segment; receiver folded in for methods. One canonicalizer, unchanged from v1. +- **Id member segments (spec `docs/design/specs/v2-go-ids.md`, issue #46)**: a type is + `/`; a method is `//`; a package-level function is + `/` (a method declared in another file uses that file as ``, see + § Cross-file method ids below); a closure is `/closure_N`. The package import path is dropped: + the member segment is the signature's part after its last dot (Go names contain no dots). + Follows `codeanalyzer-typescript` `idFromSig` and `codeanalyzer-python` `child_id` + [user, 2026-11-20]. Externals: `/@external///` (import path + whole) [user, 2026-11-20]. +- **Callable and type `signature`** = the full `signatureOf()` output (import-qualified), a + JSON field on both (types gained it, additive, for the Neo4j `:GoType.signature`) [user, + 2026-11-20]. The id carries only the bare member name, so this field is where the package + path lives. One canonicalizer, unchanged from v1. +- **Cross-file method ids** [user, 2026-11-20]: a method declared in another file than its receiver + type gets `//`, e.g. `…/handlers.go/frontendServer/logoutHandler`. + The node stays in the receiver type's `callables{}`. `source_file` is kept for text recovery. + Neo4j adds `GO_DECLARES` from the declaring `:GoModule` to the method, beside `GO_HAS_METHOD`. + Spec: `docs/design/specs/v2-go-ids.md` § Amendment. +- **Generic types**: `signatureOf()` carries no type parameters, so a generic type or method + ids as `//`, with no `[T]` in the id. - **`span.bytes`** = **UTF-8 byte offsets** into `module.source` (Go strings are natively UTF-8; avoids the UTF-16/rune slicing mismatch TS hit in cants#179). @@ -64,8 +81,9 @@ committed spec is the full transcript: `docs/design/specs/v2-l1-emission.md`. `(*strings.Builder).WriteString`) is no longer dropped: the call node's `callee` and the `call_graph` edge `dst` are the callee's `@external` id. Builtins, type conversions (no longer minted as `.__new__`) and calls through function values stay `null`. -- **Id** = `can:///@external//` where `` is the callee signature - minus the `.` prefix (`Println`, `Builder.WriteString`); it may also name a +- **Id** = `can:///@external///` where `` is the callee signature + minus the `.` prefix; its dots become segments (`Println` → `…/fmt/Println`, + `Builder.WriteString` → `…/strings/Builder/WriteString`); it may also name a library *interface* method (`@external/io/Writer.Write`). Methods on generic library types record the declared (origin) signature, not the instance. `@external` is language-neutral (not `go/`), so a polyglot app shares one external namespace. @@ -129,6 +147,17 @@ Reuses Group A ids/spans and the L3 vocabulary unchanged; every L4 fact is an ad - A Go-specific `select` cfg kind was not added (`select` is a `switch` with `switch_case` edges). +### Repository-artifact decisions (spec `docs/design/specs/v2-artifacts.md`, branch `feat/artifacts`) +Made with the user in the 2026-11-19 design loop, anchored on `codeanalyzer-python`'s `PyArtifact` / `PyDependency`. +- **Placement:** `application.artifacts{: artifact}` (map); `application.dependencies[]` (list). +- **Scope:** all non-`.go` project files [user, 2026-11-19]. Binaries are metadata only (`source:""`) [user]. +- **Artifact id:** `can:///artifact/`, app-scoped (Python parity) [user]. The Go projector's `:Artifact` flips to `CanNode=true`. +- **Package id:** version-less `pkg:golang/` [user]; version lives on the dependency edge. +- **Source text:** only with `--artifact-text`, default off [user]. `config_keys` deferred [user]. +- **Dependencies:** every `go.mod` require; `direct=false` on `// indirect`; `kind:"runtime"`; `go.sum` supplies `locked_version` only, `go.sum`-only entries excluded [user]. +- **`provides_imports`:** longest-prefix match against `go.mod` module paths [user]. +- **Level:** L1, every level, not gated [user]. + ## Neo4j-projection decisions The graph projection (`--emit neo4j`) is a **second projection of the same v2 envelope**, diff --git a/README.md b/README.md index 0d61c82..f063d6d 100644 --- a/README.md +++ b/README.md @@ -173,6 +173,7 @@ Flags: -f, --format string Output format: json|msgpack (default "json") --graph-field-depth int Field-path depth bound for L3 dataflow nodes (k limit; applies at -a 3 and -a 4) (default 3) -h, --help help for cango + --artifact-text Capture each text artifact's whole source in analysis.json (default: metadata only) -i, --input string Project root to analyze (required) -j, --jobs int Worker parallelism (default: CPU cores) --library-models string JSON file of library stand-in flows overriding the all-to-all default (-a 4 and --emit neo4j only) diff --git a/cmd/codeanalyzer/main.go b/cmd/codeanalyzer/main.go index d2b5379..3cceee3 100644 --- a/cmd/codeanalyzer/main.go +++ b/cmd/codeanalyzer/main.go @@ -46,6 +46,7 @@ func rootCmd() *cobra.Command { libraryModels string targetFiles []string skipTests bool + artifactText bool eager bool cacheDir string jobs int @@ -172,6 +173,7 @@ via CLDK(language="go").analysis(project_path=...).`, AnalyzerVersion: version, TargetFiles: targetFiles, SkipTests: skipTests, + ArtifactText: artifactText, Eager: eager, CacheDir: cacheDir, Jobs: jobs, @@ -238,6 +240,8 @@ via CLDK(language="go").analysis(project_path=...).`, f.StringSliceVarP(&targetFiles, "target-files", "t", nil, "Restrict analysis to specific files (incremental mode)") f.BoolVar(&skipTests, "skip-tests", true, "Skip *_test.go files") + f.BoolVar(&artifactText, "artifact-text", false, + "Capture each text artifact's whole source in analysis.json (default: metadata only)") f.BoolVar(&eager, "eager", false, "Force clean rebuild (ignore cache)") f.StringVarP(&cacheDir, "cache-dir", "c", "", "Cache directory (default: ~/.cldk/go-cache)") f.IntVarP(&jobs, "jobs", "j", 0, "Worker parallelism (default: CPU cores)") diff --git a/docs/design/specs/v2-artifacts.md b/docs/design/specs/v2-artifacts.md new file mode 100644 index 0000000..6d8ed76 --- /dev/null +++ b/docs/design/specs/v2-artifacts.md @@ -0,0 +1,155 @@ +# Spec: repository artifacts and dependencies (codeanalyzer-go) + +> **Cross-repo spec.** Canonical home is `codellm-devkit/.github → docs/design/specs/`, +> beside the coordinating parent issue. Staged locally on `feat/artifacts` until that repo +> is available (same precedent as `v2-l1-emission.md`). This is the design transcript of the +> datamodel loop, linked from the tracking record, not pasted into it. + +## Contract-Impact Triage + +**Does this change schema v2 output?** Yes. It adds two application-scoped fields +(`artifacts`, `dependencies`), new id shapes for artifact and package nodes, one new CLI flag +(`--artifact-text`), and populates the already-declared Neo4j neutral layer (`:Artifact`, +`:Package`, `HAS_ARTIFACT`, `DECLARES_DEPENDENCY`, `LOCKS`). + +| Change type | Analyzers | SDKs | Docs | +| --- | --- | --- | --- | +| Schema v2 evolution (additive) | **codeanalyzer-go** (emit + projector); codeanalyzer-python: none (parity adopted, no id change) | **python-sdk** (Go backend reconstruction of `:Artifact` / `:Package`) | `CLAUDE.md` § Schema decisions, this spec | + +Anchor: `codeanalyzer-python` already emits this layer (`PyArtifact`, `PyDependency`, +`PyConfigKey`, `application.artifacts`, `application.dependencies`, +`codeanalyzer/schema/py_schema.py`), and `python-sdk/cldk/analysis/commons/artifacts.py` +rebuilds it from Neo4j, with the neutral labels shared across analyzers. + +## Problem + +The v2 schema and projector declare the neutral repository-artifact layer, but the Go +analyzer never populates it. `go.mod` is read only for `go mod download`. Against the Go +analysis of `microservices-demo/src/frontend`, `MATCH (n:Artifact)` returns 0 rows. + +## Design-loop decisions (the transcript) + +Each decision was made with the user, one question per decision, in the order below. + +### Application placement +- **`application.artifacts`** is a **map keyed by repo-relative path**, as Python does. Stays + reachable from `application` by containment (keystone rule). +- **`application.dependencies`** is a **list** (Python parity). + +### Artifact scope and fields +- **Scope: all non-`.go` project files** (user decision, broader than the recommended manifests-only + scope). `.go` files are modules, not artifacts. Directories `.git/` and `vendor/` are excluded + (implementation default, see Open items). +- **Artifact fields** (Python `PyArtifact` parity): + | field | type | notes | + | --- | --- | --- | + | `id` | string | `can:///artifact/` | + | `kind` | string | `"artifact"` | + | `path` | string | repo-relative POSIX path (also the map key) | + | `format` | string | `toml\|yaml\|json\|ini\|properties\|requirements\|dockerfile\|text\|binary`, plus Go's `gomod` / `gosum` (Open items) | + | `roles` | string[] | `dependency-manifest` for `go.mod`, `lockfile` for `go.sum`; others only when recognized | + | `size_bytes` | int | always | + | `sha256` | string | full file hash, always | + | `source` | string | whole file text **only** with `--artifact-text`; otherwise `""` | + | `extraction` | string | `none` \| `partial` \| `full`; `full` when `source` is set, else `none` | +- **Binary files**: metadata only. `format:"binary"`, `size_bytes` and `sha256` set, + `source:""` (user decision). +- **`--artifact-text`** (default **off**): when on, `source` carries the whole file text + (user decision). Off keeps default output to metadata. +- **`config_keys`** is **deferred** (user decision): not emitted in this change. No + `:ConfigKey` rows are written. + +### Artifact id +- **`can:///artifact/`** (Python parity, user decision). App-scoped, so the + projector's `can:////` prefix scoping covers artifact nodes. This changes the + Go projector's `:Artifact` flag to `CanNode=true`; the current Go schema comment ("neutral, + not a can:// id") is superseded. + +### Package and dependencies +- **Package id: version-less `pkg:golang/`** (user decision), for example + `pkg:golang/github.com/gorilla/mux`. Mirrors Python's `pkg:pypi/`. The version lives + on the edge and the record, so every dependent app converges on one `:Package`. +- **Dependency records come from every `go.mod` `require`** (user decision). Direct and + indirect entries alike. + | field | source | + | --- | --- | + | `name` | module path | + | `ecosystem` | `"golang"` | + | `spec` | version in `go.mod` (for example `v1.8.0`) | + | `kind` | `"runtime"` for every entry (`go.mod` has no dev/optional split) | + | `direct` | `false` when the line carries `// indirect`, else `true` | + | `locked_version` | version pinned in `go.sum` for that module, when present | + | `declared_in` | the `go.mod` artifact id | + | `provides_imports` | import paths in the project that resolve to this module (see below) | + | `prov` | `["declared"]`, plus `"lockfile"` when `go.sum` supplies `locked_version` | +- **`go.sum`-only entries are not dependencies** (user decision, the non-recommended option + would have included them; the recommended default is not to, because `go.sum` retains + entries for pruned modules). `go.sum` contributes only `locked_version` and `prov`. +- **`provides_imports`**: populated by longest-prefix match of each project import path + against the `go.mod` module paths (user decision, Python parity). Import paths that match + no requirement are **not** recorded in this change (see Open items). + +### Analysis level +- **L1, every level** (user decision). Parse-only facts, identical at `-a 1` through `-a 4`. + Not gated. + +### Neo4j projection (the neutral layer, unchanged labels) +- `:Artifact {id, path, format}` (keyed by `id`, `CanNode=true`), under `HAS_ARTIFACT` + from `:GoApplication`. +- `:Package {id, ecosystem, name}` keyed by `id`. +- `DECLARES_DEPENDENCY` `(:Artifact)→(:Package)`, `_k = kind` (`runtime`), carrying `spec`, + `direct`, `prov` as properties. +- `LOCKS` `(:Artifact)→(:Package)` from `go.sum`, carrying `locked_version`. +- No `:ConfigKey` rows and no `DEFINES_CONFIG` rows in this change. + +## Open items (flagged for review, not asked) + +These are defaults I chose while writing the spec. Confirm them during review: + +1. **`format` values for go.mod and go.sum.** I used `gomod` and `gosum`, because the Python + enum has no Go case. The alternative is `text`. +2. **Excluded directories.** `.git/` and `vendor/` are skipped. `vendor/` is third-party + source, not project configuration. +3. **Imports with no matching requirement.** Not recorded. They are `unresolved_imports` in + Python. That field is deferred with the repository-provenance field. +4. **`locked_version` when `go.sum` has several lines for one module** (the `h1:` and + `/go.mod` hashes). Take the version common to both lines; error if they disagree. + +## Resolution (implementation, branch `feat/artifacts`) + +The open items were resolved with the defaults below. Two are deviations from the wording +above, marked **(deviation)**: + +1. **`format`**: `gomod` for `go.mod`, `gosum` for `go.sum`, adopted. +2. **Excluded directories**: `.git/` and `vendor/` are skipped at any depth. The tool's own + outputs are also skipped when they sit in the project: `analysis.json` and `graph.cypher` + in the `-o` directory, and the `-c` cache subtree. Without this, a run that writes into the + project would capture the previous run's output. +3. **Unrecorded imports**: an import matching no `go.mod` requirement is not recorded. +4. **Locked version (deviation)**: a requirement is locked only when go.sum has **both** the + `h1:` and the `/go.mod` line for the exact required version. A version with only one line + is left unlocked (`locked_version` absent, `prov` without `lockfile`), with no error. A + disagreement cannot arise, since the lookup is keyed by the required version. +5. **Artifact id root (deviation)**: the code's grammar is `can:////…`, the + same root every other node uses, so an artifact id is + `can:////artifact/`. The `can:///artifact/` form in the + decisions above is shorthand for that root. + +Also decided at implementation: `source` is omitted from JSON when empty (`omitempty`), not +written as `""`; only the root `go.mod` contributes dependencies (nested modules are artifacts +only); `dependencies` is absent when there is no root `go.mod`. + +## Consumers + +- **codeanalyzer-go**: emits the fields; the projector writes the rows above. Tests: a fixture + with `go.mod` and `go.sum`, a projector test for the new rows, and a `--artifact-text` + on/off test. +- **python-sdk**: the Go backend reads `:Artifact` / `:Package` through the shared + `cldk/analysis/commons/artifacts.py` reconstructors. The Go backend needs its artifact getters + added. Scope of that rung is decided at decomposition. + +## Out of scope (deferred) + +- Config keys and `DEFINES_CONFIG` (user decision). +- `unresolved_imports`, repository provenance (`repository`), `config_uses`. +- Codeanalyzer-python: no change. diff --git a/docs/design/specs/v2-go-ids.md b/docs/design/specs/v2-go-ids.md new file mode 100644 index 0000000..b35ba2d --- /dev/null +++ b/docs/design/specs/v2-go-ids.md @@ -0,0 +1,158 @@ +# Spec: schema v2 id format for Go callables and types (codeanalyzer-go) + +> **Cross-repo spec.** The python-sdk Go backend keys types and fields by the last id segment, +> so the canonical home is `codellm-devkit/.github → docs/design/specs/`, beside the +> coordinating record. Staged locally on `feat/v2-go-ids` until that repo is available (same +> precedent as `v2-l1-emission.md` and `v2-artifacts.md`). This is the design transcript of the +> datamodel loop, linked from the tracking issue, not pasted into it. + +## Contract-Impact Triage + +**Does this change schema v2 output?** Yes. It changes the id shape of `type`, method, and +function nodes, and of `external_symbols` keys. It does not change the module, field, +artifact, or ordinal-id shapes, and it adds no field. + +| Change type | Analyzers | SDKs | Docs | +| --- | --- | --- | --- | +| Schema v2 id-format change | **codeanalyzer-go** (`internal/schema/v2/ids.go`, `v2emit`, Neo4j projector) | **python-sdk** Go backend (`cldk/analysis/go/neo4j/neo4j_backend.py` keys types and fields by last id segment; verify methods at the SDK rung) | `CLAUDE.md` § Schema decisions, `canonical-schema.md` references, this spec | + +Convention anchor: `codeanalyzer-typescript/src/schema/ids.ts` (`idFromSig`: strip the module +prefix from the dotted signature, split the tail on `.`) and `codeanalyzer-python/codeanalyzer/schema/ids.py` +(`child_id`, `ordinal_id`, `external_id`, `artifact_id`). + +## Problem + +Go ids embed the full import-qualified signature as one segment. For a method this repeats the +package path twice: + +``` +can://boutique/frontend/go/middleware.go/github.com/GoogleCloudPlatform/microservices-demo/src/frontend.logHandler/github.com/GoogleCloudPlatform/microservices-demo/src/frontend.logHandler.ServeHTTP/github.com/GoogleCloudPlatform/microservices-demo/src/frontend.logHandler.ServeHTTP.closure_1@82:3/2/actual_out:2 +``` + +The segment is not a path segment. A parser that splits on `/` (the Neo4j projector's +`lastSegment`) cannot recover a Go callable's name from its id, and the id is long and +unreadable. + +## Design-loop decisions (the transcript) + +Each decision was made with the user, one per question, in spine order. + +### Module (unchanged) +`can:////go/`. Already the convention. No question was needed. + +### Type +- **Tail = ``** (user decision, recommended). The Go package path is stripped from the + signature, then the type name is appended to the module: `/logHandler`. Precedent: + TypeScript `idFromSig`, Python `child_id`. The package path is not repeated, since the module + already names the scope. +- The package path that is stripped is the `pkgPath` of the symbol's signature, which equals + the package's import path. It is the same for every symbol in one file. + +### Callable +- **Function: `/`**, e.g. `middleware.go/ensureSessionID`. No receiver. +- **Method: `//`** (user decision, recommended). A method nests under its + receiver type, matching containment (`type.callables`). Precedent: TypeScript split-on-dots, + Python `child_id`. Example: `middleware.go/logHandler/ServeHTTP`. +- **Closure: `/closure_N`** (user decision, recommended). N is the source-order index + within the enclosing callable, as the analyzer numbers them today. Example: + `middleware.go/logHandler/ServeHTTP/closure_1`. Position is not used, because the body + ordinal key already carries it. +- **Ordinal and synthetic keys are unchanged**: `@`, `@entry`, `@exit`, + `@formal_in:`, `@actual_in:` / `@actual_out`, and the `/N` disambiguator for nested + calls at one position. Nothing in those changes. + +### Field +`/`, already in this shape. No change. + +### External symbols (decided earlier with the user) +`/@external///`, the TypeScript `/` form. The +import path stays whole, since it contains dots and slashes. A package-level function is +`/@external//`. + +### Call body node and call-graph edge +Unchanged in shape. A call node's `callee` and a call-graph edge's endpoints take the new +callable and external ids. No new field. + +### Projector: signature property (user decision) +The Neo4j `signature` property on `:GoCallable` and `:GoType` is copied from the node's JSON +`signature` field, not derived from the id. Today the projector takes the last `/` or `@` +segment of the id (`lastSegment`), which yields `Type.Method` and loses the import-qualified +name the JSON carries. The id no longer needs to carry it. + +## Consequences + +- **Parsing**: a Go id splits cleanly on `/`: module, then type, then method. The `@` ordinal + suffix is unchanged. +- **Uniqueness**: ids are unique within a file. Go forbids duplicate package-scope identifiers, + and forbids a duplicate method name on one type. Two files in one package have distinct + module ids, so the stripped package path never causes a collision. +- **Cross-file methods** (amended 2026-11-20, see Amendment below): a method declared in another + file gets an id under the file that declares it, `//`. The + method node stays in its receiver type's `callables{}`, and `source_file` names the declaring + file for text recovery. +- **python-sdk**: `cldk/analysis/go/neo4j/neo4j_backend.py` keys types by `ty.id.split("/")[-1]`, + which still returns the type name, and fields by `fld.id.split("/")[-1]`, which is unchanged. + Callables keyed by a dotted sig will change meaning: the last segment is now the bare method + name. The SDK rung must verify this. + +## Open items (flagged for review) + +1. **Name collision inside a file**: a type `X` and a function `X` cannot share a file's package + scope, so the ids cannot collide. Confirm no other Go construct gives two nodes the same + `//` (a type's method set is unique by name). Expected: none. +2. **Generic types**: a generic type's signature includes type parameters. The `[...]` part is + not part of the segment name. Decide at implementation: strip the type parameters (default) + or keep them. +3. **Dot-free function names in `external_symbols` keys**: keys become `//`. + Confirm the key is still unique when two packages share a last path element. + +## Tracking + +One issue in `codeanalyzer-go`, rungs as checklist items (user decision). Base branch +`feat/artifacts`, since the artifact code is the base for this work (user decision). Analyzer +ships first; the python-sdk Go backend follows once the id shape is released. + +## Implementation notes (2026-11-20) + +- **Type signature field (amended)**: the spec said this change adds no field. The Neo4j + `:GoType.signature` needs the full name, which the type id no longer carries, so `Type` + gained a `signature` JSON field (the full `signatureOf()`), mirroring `Callable`. The + projector copies it. Decided with the user. +- **Member name rule**: implemented as "the part after the last dot of the signature". This + is the same as stripping the import path, since Go identifiers contain no dots. +- **Open item 2 (generics) resolved**: `signatureOf()` has no type parameters, so a generic + type `Set[T]` ids as `set/set.go/Set` and its method as `set/set.go/Set/Add`. +- **Open item 1 (collisions) resolved**: the generics fixture and the chi/init fixtures + produce distinct ids. `init` functions are distinct per file by module. + +## Amendment (2026-11-20): cross-file method ids + +Decided with the user in the design loop, after the id change landed. + +**Problem.** A method is nested under its receiver type's module, so its id names the file +where the receiver lives, not the file where the method is declared. Example: `frontendServer.logoutHandler` +is declared in `handlers.go` but its id was `…/main.go/frontendServer/logoutHandler`. 22 methods in the +frontend app are affected (those with `source_file` set). + +**Decisions.** + +1. **Id.** A callable's id contains the file where it is declared: + `//`, e.g. `…/handlers.go/frontendServer/logoutHandler`. The type + segment stays. Rejected: `/`, because a method and a package-level + function of the same name in one file would collide. +2. **JSON containment.** The node stays in its receiver type's `callables{}` (containment by + receiver is unchanged). Only its id changes. Moving it under the declaring module was rejected + because it changes containment for every consumer and hides which type the method belongs to. +3. **Text recovery.** `source_file` is kept. The id's module now equals `source_file`, but the + node still nests under the receiver's module, so its span still needs `source_file` to resolve. +4. **Neo4j.** `GO_HAS_METHOD` from the receiver `:GoType` is kept. A `GO_DECLARES` edge is added from + the declaring `:GoModule` to the method, so "which file declares this method" is one query. No + new relationship kind. + +**Release.** Same PR and release as the id format (feat/v2-go-ids). Tracked in #46. + +## Out of scope + +- Module, field, artifact, and ordinal-id shapes. +- Any new node kind, edge kind, or field. +- Renaming the `signature` field (kept as-is). diff --git a/internal/core/analyzer.go b/internal/core/analyzer.go index a82ac2c..b9dbf34 100644 --- a/internal/core/analyzer.go +++ b/internal/core/analyzer.go @@ -32,6 +32,7 @@ import ( "github.com/codellm-devkit/codeanalyzer-go/internal/analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" "github.com/codellm-devkit/codeanalyzer-go/internal/options" + "github.com/codellm-devkit/codeanalyzer-go/internal/repository" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" "github.com/codellm-devkit/codeanalyzer-go/internal/sdg" "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" @@ -93,6 +94,25 @@ func New(opts options.AnalysisOptions) *Analyzer { // Analyze runs the full analysis pipeline and returns a GoApplication. func (a *Analyzer) Analyze() (*schema.GoApplication, error) { + app, err := a.analyze() + if err != nil { + return nil, err + } + // The repository layer reads files outside the Go source, so it is computed + // on every run, after the (possibly cached) application is ready. + facts, err := repository.Facts(a.opts.InputPath, app, repository.Options{ + CaptureText: a.opts.ArtifactText, + OutputDir: a.opts.OutputDir, + CacheDir: a.opts.CacheDir, + }) + if err != nil { + return nil, fmt.Errorf("repository layer: %w", err) + } + app.Repository = facts + return app, nil +} + +func (a *Analyzer) analyze() (*schema.GoApplication, error) { if _, err := resolveIdentity(a.opts); err != nil { return nil, err } diff --git a/internal/core/crossfile_method_test.go b/internal/core/crossfile_method_test.go new file mode 100644 index 0000000..aa3060a --- /dev/null +++ b/internal/core/crossfile_method_test.go @@ -0,0 +1,161 @@ +package core_test + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/core" + "github.com/codellm-devkit/codeanalyzer-go/internal/options" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis/v2emit" +) + +// crossFileProject: T is declared in a.go, its method M in b.go, a package-level +// function M in b.go too (same name, different id), and Use in c.go calls both. +func crossFileProject(t *testing.T) string { + t.Helper() + dir := t.TempDir() + files := map[string]string{ + "go.mod": "module example.com/xf\n\ngo 1.25.0\n", + "a.go": "package xf\n\ntype T struct{ n int }\n", + "b.go": "package xf\n\nfunc (t T) M() int { return t.n }\n\nfunc M() int { return 2 }\n", + "c.go": "package xf\n\nfunc Use() int {\n\tvar t T\n\treturn t.M() + M()\n}\n", + } + for rel, content := range files { + if err := os.WriteFile(filepath.Join(dir, rel), []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + return dir +} + +// analyzeL2 runs the real pipeline at -a 2 and returns the analysis.json tree as +// generic maps (the v2 Go structs are not needed for these checks). +func analyzeL2(t *testing.T, dir string) map[string]any { + t.Helper() + opts := options.AnalysisOptions{ + InputPath: dir, AppName: "xf", Level: options.LevelCallGraph, + SkipTests: true, CacheDir: t.TempDir(), OutputDir: t.TempDir(), + } + app, err := core.New(opts).Analyze() + if err != nil { + t.Fatal(err) + } + identity, err := v2.ResolveIdentity("xf", nil) + if err != nil { + t.Fatal(err) + } + payload := v2emit.Emit(app, identity, dir, int(opts.Level), "test", 0, nil, nil) + raw, err := json.Marshal(payload) + if err != nil { + t.Fatal(err) + } + var doc map[string]any + if err := json.Unmarshal(raw, &doc); err != nil { + t.Fatal(err) + } + return doc +} + +func asMap(v any) map[string]any { m, _ := v.(map[string]any); return m } + +// methodNode finds the method whose signature ends in suffix, under any type. +func methodNode(st map[string]any, suffix string) map[string]any { + for _, mod := range st { + for _, ty := range asMap(asMap(mod)["types"]) { + for _, c := range asMap(asMap(ty)["callables"]) { + if strings.HasSuffix(asMap(c)["signature"].(string), suffix) { + return asMap(c) + } + } + } + } + return nil +} + +func TestCrossFileMethod_IDNamesDeclaringFile(t *testing.T) { + doc := analyzeL2(t, crossFileProject(t)) + st := asMap(asMap(doc["application"])["symbol_table"]) + + m := methodNode(st, ".T.M") + if m == nil { + t.Fatal("method T.M not found") + } + const wantID = "can://xf/xf/go/b.go/T/M" + if got := m["id"]; got != wantID { + t.Errorf("method id = %v, want %s (declaring file b.go, not the receiver's a.go)", got, wantID) + } + if got := m["source_file"]; got != "b.go" { + t.Errorf("source_file = %v, want b.go", got) + } +} + +func TestCrossFileMethod_SameNameFunctionDistinct(t *testing.T) { + doc := analyzeL2(t, crossFileProject(t)) + st := asMap(asMap(doc["application"])["symbol_table"]) + fn := asMap(asMap(asMap(st["b.go"])["functions"])) + var fnID string + for _, f := range fn { + if asMap(f)["signature"] == "example.com/xf.M" { + fnID = asMap(f)["id"].(string) + } + } + const wantFn = "can://xf/xf/go/b.go/M" + if fnID != wantFn { + t.Errorf("function M id = %q, want %s", fnID, wantFn) + } + if m := methodNode(st, ".T.M"); m != nil && m["id"] == fnID { + t.Errorf("method and function named M collide at %s", fnID) + } +} + +// The call in Use (c.go) to the cross-file method resolves to the declaring-file +// id: the call-graph edge and the body call node both name b.go. +func TestCrossFileMethod_CallsResolveToDeclaringFile(t *testing.T) { + doc := analyzeL2(t, crossFileProject(t)) + app := asMap(doc["application"]) + const method = "can://xf/xf/go/b.go/T/M" + + var caller string + for _, mod := range asMap(app["symbol_table"]) { + for _, f := range asMap(asMap(mod)["functions"]) { + if asMap(f)["signature"] == "example.com/xf.Use" { + caller = asMap(f)["id"].(string) + } + } + } + if caller == "" { + t.Fatal("Use not found") + } + + edge := false + for _, e := range app["call_graph"].([]any) { + em := asMap(e) + if em["src"] == caller && em["dst"] == method { + edge = true + } + } + if !edge { + t.Errorf("no call_graph edge %s -> %s", caller, method) + } + + resolved := false + for _, mod := range asMap(app["symbol_table"]) { + for _, f := range asMap(asMap(mod)["functions"]) { + if asMap(f)["id"] != caller { + continue + } + for _, bn := range asMap(asMap(f)["body"]) { + if c, ok := asMap(bn)["callee"].(string); ok && c == method { + resolved = true + } + } + } + } + if !resolved { + t.Errorf("no body call node in Use has callee %s", method) + } +} diff --git a/internal/core/repository_test.go b/internal/core/repository_test.go new file mode 100644 index 0000000..1f72580 --- /dev/null +++ b/internal/core/repository_test.go @@ -0,0 +1,109 @@ +package core_test + +import ( + "encoding/json" + "os" + "path/filepath" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/core" + "github.com/codellm-devkit/codeanalyzer-go/internal/options" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis/v2emit" +) + +// repoProject is a Go module with no requires (so no network materialization), +// one .go file, and two non-.go files. +func repoProject(t *testing.T) string { + t.Helper() + dir := t.TempDir() + files := map[string]string{ + "go.mod": "module example.com/repoapp\n\ngo 1.25.0\n", + "go.sum": "", + "main.go": "package main\n\nfunc main() {}\n", + "README.md": "# repo app\n", + } + for rel, content := range files { + if err := os.WriteFile(filepath.Join(dir, rel), []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + return dir +} + +// emitJSON runs Analyze at level 1 and returns the v2 analysis.json bytes. +func emitJSON(t *testing.T, dir string, artifactText bool) []byte { + t.Helper() + opts := options.AnalysisOptions{ + InputPath: dir, AppName: "repoapp", Level: options.LevelSymbolTable, + SkipTests: true, CacheDir: t.TempDir(), OutputDir: t.TempDir(), ArtifactText: artifactText, + } + app, err := core.New(opts).Analyze() + if err != nil { + t.Fatal(err) + } + identity, err := v2.ResolveIdentity("repoapp", nil) + if err != nil { + t.Fatal(err) + } + payload := v2emit.Emit(app, identity, dir, int(opts.Level), "test", 0, nil, nil) + out, err := json.Marshal(payload) + if err != nil { + t.Fatal(err) + } + return out +} + +func TestAnalyze_ArtifactsDefaultMetadataOnly(t *testing.T) { + dir := repoProject(t) + var doc struct { + Application struct { + Artifacts map[string]v2.Artifact `json:"artifacts"` + } `json:"application"` + } + if err := json.Unmarshal(emitJSON(t, dir, false), &doc); err != nil { + t.Fatal(err) + } + art, ok := doc.Application.Artifacts["README.md"] + if !ok { + t.Fatalf("README.md missing from artifacts: %v", doc.Application.Artifacts) + } + if art.ID != "can://repoapp/repoapp/artifact/README.md" { + t.Errorf("artifact id = %q", art.ID) + } + if art.Format != "text" || art.Extraction != "none" || art.Source != "" { + t.Errorf("default artifact should be metadata only: %+v", art) + } + if _, ok := doc.Application.Artifacts["main.go"]; ok { + t.Errorf("a .go file became an artifact") + } +} + +func TestAnalyze_ArtifactTextCapturesSource(t *testing.T) { + dir := repoProject(t) + var doc struct { + Application struct { + Artifacts map[string]v2.Artifact `json:"artifacts"` + } `json:"application"` + } + if err := json.Unmarshal(emitJSON(t, dir, true), &doc); err != nil { + t.Fatal(err) + } + art := doc.Application.Artifacts["README.md"] + if art.Source != "# repo app\n" || art.Extraction != "full" { + t.Errorf("--artifact-text: source/extraction = %q/%q", art.Source, art.Extraction) + } + mod := doc.Application.Artifacts["go.mod"] + if mod.Format != "gomod" || len(mod.Roles) != 1 || mod.Roles[0] != "dependency-manifest" { + t.Errorf("go.mod artifact = %+v", mod) + } +} + +func TestAnalyze_ArtifactsReproducibleAcrossRuns(t *testing.T) { + dir := repoProject(t) + a := emitJSON(t, dir, false) + b := emitJSON(t, dir, false) + if string(a) != string(b) { + t.Errorf("two runs over the same project differ") + } +} diff --git a/internal/neo4j/bolt.go b/internal/neo4j/bolt.go index da5b001..8a96fa4 100644 --- a/internal/neo4j/bolt.go +++ b/internal/neo4j/bolt.go @@ -129,6 +129,20 @@ func (w *boltWriter) run() error { } } + // Purge every changed module before upserting any of them. A module's purge is + // by id prefix, and a node a module OWNS may carry another module's prefix (a + // method declared in rpc.go but owned by main.go's receiver). Purging rpc.go + // after main.go had upserted that method would delete it, so all purges run first. + if w.cfg.Eager && scoped { + for _, m := range sortedModuleKeys(byModule) { + if !changed[m] { + continue + } + if err := w.purgeModule(m); err != nil { + return err + } + } + } // Upsert shared nodes (MERGE-only, every run) then changed-module nodes. if err := w.upsertNodes(shared); err != nil { return err @@ -137,11 +151,6 @@ func (w *boltWriter) run() error { if !changed[m] { continue } - if w.cfg.Eager && scoped { - if err := w.purgeModule(m); err != nil { - return err - } - } if err := w.upsertNodes(byModule[m]); err != nil { return err } diff --git a/internal/neo4j/crossfile_test.go b/internal/neo4j/crossfile_test.go new file mode 100644 index 0000000..1c7132d --- /dev/null +++ b/internal/neo4j/crossfile_test.go @@ -0,0 +1,54 @@ +package neo4j + +import ( + "testing" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// A method declared in b.go but whose receiver type lives in a.go: GO_HAS_METHOD +// from the type, and GO_DECLARES from the declaring module b.go. +func TestProject_CrossFileMethodDeclaredByItsModule(t *testing.T) { + const ( + app = "can://xf/xf" + typeMod = app + "/go/a.go" + declMod = app + "/go/b.go" + typeID = typeMod + "/T" + methodID = declMod + "/T/M" + ) + a := &v2.Analysis{ + IdentityVersion: 2, SchemaVersion: v2.SchemaVersion, Language: v2.Language, MaxLevel: 1, + Application: v2.Application{ + IdentityVersion: 2, ID: app, Kind: "application", + SymbolTable: map[string]v2.Module{ + "a.go": {ID: typeMod, Kind: "module", Types: map[string]v2.Type{ + "T": {ID: typeID, Kind: "struct", Signature: "example.com/xf.T", + Callables: map[string]v2.Callable{ + "example.com/xf.T.M": {ID: methodID, Kind: "method", Signature: "example.com/xf.T.M", SourceFile: "b.go"}, + }}, + }}, + "b.go": {ID: declMod, Kind: "module"}, + }, + }, + } + rows := Project(a) + + hasMethod, declares := false, false + for _, e := range rows.Edges { + if e.To.Value != methodID { + continue + } + switch e.Type { + case RelHasMethod: + hasMethod = e.From.Value == typeID + case RelDeclares: + declares = e.From.Label == LabelModule && e.From.Value == declMod + } + } + if !hasMethod { + t.Errorf("no GO_HAS_METHOD from receiver type %s to %s", typeID, methodID) + } + if !declares { + t.Errorf("no GO_DECLARES from declaring module %s to %s", declMod, methodID) + } +} diff --git a/internal/neo4j/project.go b/internal/neo4j/project.go index 78fde41..8790153 100644 --- a/internal/neo4j/project.go +++ b/internal/neo4j/project.go @@ -2,6 +2,7 @@ package neo4j import ( "encoding/json" + "sort" "strings" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" @@ -33,6 +34,7 @@ func Project(a *v2.Analysis) GraphRows { declared: collectDeclaredIDs(a), } p.projectApplication() + p.projectRepository() for _, mod := range a.Application.SymbolTable { p.projectModule(mod) } @@ -84,6 +86,48 @@ func (p *projector) appRef() NodeRef { return NodeRef{Label: LabelApplication, KeyProp: "id", Value: p.analysis.Application.ID} } +// ─── repository layer (spec v2-artifacts) ──────────────────────────────────── + +// projectRepository writes the artifact and dependency rows. An artifact is an +// app-scoped node (its id is a can:// id) hung off the application; a dependency +// is a neutral :Package keyed by its version-less purl, declared by the go.mod +// artifact and, when go.sum locks it, locked by the go.sum artifact. +func (p *projector) projectRepository() { + arts := p.analysis.Application.Artifacts + paths := make([]string, 0, len(arts)) + for path := range arts { + paths = append(paths, path) + } + sort.Strings(paths) + for _, path := range paths { + art := arts[path] + ref := p.b.Node( + []string{LabelArtifact, MarkerLabel}, + "id", art.ID, + Props{"path": art.Path, "format": art.Format}, + "", // an artifact belongs to no single module + ) + p.b.Edge(RelHasArtifact, p.appRef(), ref, nil, "") + } + + goSum, hasGoSum := arts["go.sum"] + for _, dep := range p.analysis.Application.Dependencies { + pkgRef := p.b.Node( + []string{LabelPackage}, + "id", "pkg:"+dep.Ecosystem+"/"+dep.Name, + Props{"ecosystem": dep.Ecosystem, "name": dep.Name}, + "", + ) + declaredBy := NodeRef{Label: LabelArtifact, KeyProp: "id", Value: dep.DeclaredIn} + p.b.Edge(RelDeclaresDependency, declaredBy, pkgRef, + Props{"spec": dep.Spec, "direct": dep.Direct, "prov": dep.Prov}, dep.Kind) + if dep.LockedVersion != "" && hasGoSum { + p.b.Edge(RelLocks, NodeRef{Label: LabelArtifact, KeyProp: "id", Value: goSum.ID}, pkgRef, + Props{"locked_version": dep.LockedVersion}, "") + } + } +} + // ─── module ────────────────────────────────────────────────────────────────── func (p *projector) projectModule(m v2.Module) { @@ -118,7 +162,7 @@ func (p *projector) projectType(modRef NodeRef, module string, t v2.Type) { "id", t.ID, Props{ "kind": t.Kind, // struct|interface|alias|defined (decision T1) - "signature": lastSegment(t.ID), + "signature": t.Signature, "span_json": spanJSON(t.Span), }, module, @@ -151,6 +195,12 @@ func (p *projector) projectType(modRef NodeRef, module string, t v2.Type) { for _, m := range t.Callables { mRef := p.projectCallable(tRef, module, m) p.b.Edge(RelHasMethod, tRef, mRef, nil, "") + // A method declared in another file is also declared by that file's module, + // so "which file declares this method" is one query (spec v2-go-ids § Amendment). + if m.SourceFile != "" { + declaring := NodeRef{Label: LabelModule, KeyProp: "id", Value: v2.ModuleID(p.analysis.Application.ID, m.SourceFile)} + p.b.Edge(RelDeclares, declaring, mRef, nil, "") + } } } diff --git a/internal/neo4j/repository_test.go b/internal/neo4j/repository_test.go new file mode 100644 index 0000000..b2554f6 --- /dev/null +++ b/internal/neo4j/repository_test.go @@ -0,0 +1,97 @@ +package neo4j + +import ( + "testing" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// repoAnalysis is a minimal L1 analysis carrying only the repository layer: +// go.mod and go.sum artifacts, and two dependencies (one locked, one not). +func repoAnalysis() *v2.Analysis { + const app = "can://frontend/frontend" + goMod := app + "/artifact/go.mod" + goSum := app + "/artifact/go.sum" + return &v2.Analysis{ + IdentityVersion: 2, SchemaVersion: v2.SchemaVersion, Language: v2.Language, MaxLevel: 1, + Application: v2.Application{ + IdentityVersion: 2, ID: app, Kind: "application", + Artifacts: map[string]v2.Artifact{ + "go.mod": {ID: goMod, Kind: "artifact", Path: "go.mod", Format: "gomod", Roles: []string{"dependency-manifest"}}, + "go.sum": {ID: goSum, Kind: "artifact", Path: "go.sum", Format: "gosum", Roles: []string{"lockfile"}}, + }, + Dependencies: []v2.Dependency{ + {Name: "github.com/gorilla/mux", Ecosystem: "golang", Spec: "v1.8.0", Kind: "runtime", Direct: true, + LockedVersion: "v1.8.0", DeclaredIn: goMod, Prov: []string{"declared", "lockfile"}}, + {Name: "golang.org/x/text", Ecosystem: "golang", Spec: "v0.14.0", Kind: "runtime", Direct: false, + DeclaredIn: goMod, Prov: []string{"declared"}}, + }, + }, + } +} + +func TestProject_RepositoryLayerRows(t *testing.T) { + rows := Project(repoAnalysis()) + + nodes := map[string]NodeRow{} + for _, n := range rows.Nodes { + nodes[n.Value] = n + } + + goMod := nodes["can://frontend/frontend/artifact/go.mod"] + if !hasLabel(goMod, LabelArtifact) || !hasLabel(goMod, MarkerLabel) { + t.Errorf("artifact node labels = %v, want Artifact + %s (app-scoped)", goMod.Labels, MarkerLabel) + } + mux := nodes["pkg:golang/github.com/gorilla/mux"] + if !hasLabel(mux, LabelPackage) || hasLabel(mux, MarkerLabel) { + t.Errorf("package node labels = %v, want Package only (neutral, no marker)", mux.Labels) + } + + edges := map[string][]EdgeRow{} + for _, e := range rows.Edges { + edges[e.Type] = append(edges[e.Type], e) + } + if len(edges[RelHasArtifact]) != 2 { + t.Errorf("HAS_ARTIFACT edges = %d, want 2 (go.mod, go.sum)", len(edges[RelHasArtifact])) + } + + decl := edges[RelDeclaresDependency] + if len(decl) != 2 { + t.Fatalf("DECLARES_DEPENDENCY edges = %d, want 2", len(decl)) + } + for _, e := range decl { + if e.From.Value != "can://frontend/frontend/artifact/go.mod" || e.To.Label != LabelPackage { + t.Errorf("DECLARES_DEPENDENCY wired %s -> %s:%s, want go.mod -> Package", e.From.Value, e.To.Label, e.To.Value) + } + if e.Key != "runtime" { + t.Errorf("DECLARES_DEPENDENCY _k = %q, want runtime", e.Key) + } + } + + locks := edges[RelLocks] + if len(locks) != 1 { + t.Fatalf("LOCKS edges = %d, want 1 (only the locked dependency)", len(locks)) + } + if locks[0].From.Value != "can://frontend/frontend/artifact/go.sum" || + locks[0].To.Value != "pkg:golang/github.com/gorilla/mux" || + locks[0].Props["locked_version"] != "v1.8.0" { + t.Errorf("LOCKS = %+v, want go.sum -> mux with locked_version v1.8.0", locks[0]) + } +} + +func TestProject_NoRepositoryLayerEmitsNoRows(t *testing.T) { + a := repoAnalysis() + a.Application.Artifacts, a.Application.Dependencies = nil, nil + rows := Project(a) + for _, n := range rows.Nodes { + if hasLabel(n, LabelArtifact) || hasLabel(n, LabelPackage) { + t.Errorf("repository node %v emitted without facts", n.Labels) + } + } + for _, e := range rows.Edges { + switch e.Type { + case RelHasArtifact, RelDeclaresDependency, RelLocks: + t.Errorf("repository edge %s emitted without facts", e.Type) + } + } +} diff --git a/internal/neo4j/schema.go b/internal/neo4j/schema.go index 18853b4..274d39d 100644 --- a/internal/neo4j/schema.go +++ b/internal/neo4j/schema.go @@ -227,12 +227,15 @@ var Nodes = []NodeSpec{ Props: []string{"name", "path", "signature", "module"}, }, - // Neutral layer. Keyed by own id (not a can:// id), so NO marker label and - // outside prefix scoping — a sibling analyzer converges on the same node. + // Repository layer. An artifact id is can:////artifact/ + // (spec v2-artifacts, Python parity), so it is app-scoped: it carries the + // :GoCanNode marker and falls under the prefix scope on wipe. { - Label: LabelArtifact, MergeLabel: LabelArtifact, Key: "id", CanNode: false, + Label: LabelArtifact, MergeLabel: LabelArtifact, Key: "id", CanNode: true, Props: []string{"path", "format"}, }, + // Neutral layer. Keyed by its pkg: purl (not a can:// id), so NO marker label + // and outside prefix scoping — a sibling analyzer converges on the same node. { Label: LabelPackage, MergeLabel: LabelPackage, Key: "id", CanNode: false, Props: []string{"ecosystem", "name"}, @@ -265,8 +268,8 @@ var Rels = []RelSpec{ // Repository layer. {Type: RelHasArtifact, From: []string{LabelApplication}, To: []string{LabelArtifact}}, - {Type: RelDeclaresDependency, From: []string{LabelArtifact}, To: []string{LabelPackage}, Discriminant: "kind"}, - {Type: RelLocks, From: []string{LabelArtifact}, To: []string{LabelPackage}}, + {Type: RelDeclaresDependency, From: []string{LabelArtifact}, To: []string{LabelPackage}, Props: []string{"spec", "direct", "prov"}, Discriminant: "kind"}, + {Type: RelLocks, From: []string{LabelArtifact}, To: []string{LabelPackage}, Props: []string{"locked_version"}}, {Type: RelDefinesConfig, From: []string{LabelArtifact}, To: []string{LabelConfigKey}}, {Type: RelProvides, From: []string{LabelPackage}, To: []string{LabelExternal}}, {Type: RelUnresolvedImport, From: []string{LabelApplication}, To: []string{LabelExternal}}, diff --git a/internal/neo4j/signature_test.go b/internal/neo4j/signature_test.go new file mode 100644 index 0000000..1df9ffc --- /dev/null +++ b/internal/neo4j/signature_test.go @@ -0,0 +1,75 @@ +package neo4j + +import ( + "testing" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// The :GoCallable signature property is the JSON signature field, the full +// import-qualified name, not a segment parsed out of the id (the id now carries +// only the bare member name). +func TestProject_CallableSignatureFromJSONField(t *testing.T) { + const ( + sig = "github.com/acme/app/worker.Worker.Run" + mod = "can://ns/app/go/worker.go" + ) + a := &v2.Analysis{ + IdentityVersion: 2, SchemaVersion: v2.SchemaVersion, Language: v2.Language, MaxLevel: 1, + Application: v2.Application{ + IdentityVersion: 2, ID: "can://ns/app", Kind: "application", + SymbolTable: map[string]v2.Module{ + "worker.go": { + ID: mod, Kind: "module", + Types: map[string]v2.Type{ + "Worker": { + ID: mod + "/Worker", Kind: "struct", Signature: "github.com/acme/app/worker.Worker", + Callables: map[string]v2.Callable{ + "Run": {ID: mod + "/Worker/Run", Kind: "method", Signature: sig}, + }, + }, + }, + }, + }, + }, + } + rows := Project(a) + for _, n := range rows.Nodes { + if hasLabel(n, LabelCallable) && n.Value == mod+"/Worker/Run" { + if got := n.Props["signature"]; got != sig { + t.Errorf("signature property = %v, want %q", got, sig) + } + return + } + } + t.Fatalf("no :GoCallable row for %s", mod+"/Worker/Run") +} + +// The :GoType signature is the type JSON's signature field (full name), not the +// bare type name the id ends with. +func TestProject_TypeSignatureFromJSONField(t *testing.T) { + const ( + sig = "github.com/acme/app/worker.Worker" + mod = "can://ns/app/go/worker.go" + ) + a := &v2.Analysis{ + IdentityVersion: 2, SchemaVersion: v2.SchemaVersion, Language: v2.Language, MaxLevel: 1, + Application: v2.Application{ + IdentityVersion: 2, ID: "can://ns/app", Kind: "application", + SymbolTable: map[string]v2.Module{ + "worker.go": {ID: mod, Kind: "module", Types: map[string]v2.Type{ + "Worker": {ID: mod + "/Worker", Kind: "struct", Signature: sig}, + }}, + }, + }, + } + for _, n := range Project(a).Nodes { + if hasLabel(n, LabelType) && n.Value == mod+"/Worker" { + if got := n.Props["signature"]; got != sig { + t.Errorf("type signature property = %v, want %q", got, sig) + } + return + } + } + t.Fatalf("no :GoType row for %s", mod+"/Worker") +} diff --git a/internal/options/options.go b/internal/options/options.go index 13f0af9..ebeb300 100644 --- a/internal/options/options.go +++ b/internal/options/options.go @@ -57,6 +57,9 @@ type AnalysisOptions struct { TargetFiles []string // SkipTests skips test files (files ending in _test.go). SkipTests bool + // ArtifactText captures each text artifact's whole source (--artifact-text). + // Off by default: artifacts carry metadata only. + ArtifactText bool // Eager forces a clean rebuild ignoring any cache. Eager bool // CacheDir is where per-file caches and intermediate data are stored. diff --git a/internal/repository/repository.go b/internal/repository/repository.go new file mode 100644 index 0000000..45906d1 --- /dev/null +++ b/internal/repository/repository.go @@ -0,0 +1,279 @@ +// Package repository collects the repository-artifact layer (spec +// docs/design/specs/v2-artifacts.md): every non-.go project file as an artifact, +// and the go.mod requires as dependencies, with go.sum supplying locked versions. +// +// It reads files outside the Go source the symbol table sees, so it runs on every +// analysis (never from the cache) and is independent of the analysis level. +package repository + +import ( + "bytes" + "crypto/sha256" + "encoding/hex" + "fmt" + "io/fs" + "os" + "path/filepath" + "sort" + "strings" + "unicode/utf8" + + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" + "golang.org/x/mod/modfile" +) + +// Ecosystem is the purl type used for Go modules (pkg:golang/). +const Ecosystem = "golang" + +// Options controls discovery. +type Options struct { + // CaptureText puts each text artifact's whole source into Source (--artifact-text). + CaptureText bool + // OutputDir and CacheDir are the tool's own output locations. When either lies + // inside the project, its files are not artifacts: a run that writes analysis.json + // into the project must not capture the previous run's output. + OutputDir string + CacheDir string +} + +// skipDirNames are directories never walked: VCS metadata and vendored third-party code. +var skipDirNames = map[string]bool{".git": true, "vendor": true} + +// outputFileNames are the files this tool writes into OutputDir. +var outputFileNames = map[string]bool{"analysis.json": true, "graph.cypher": true} + +// Discover returns every non-.go regular file under root as an artifact, keyed by +// its repo-relative POSIX path. Files are visited in lexical order. +func Discover(root string, opts Options) (map[string]schema.GoArtifact, error) { + absRoot, err := filepath.Abs(root) + if err != nil { + return nil, err + } + outDir := absOrEmpty(opts.OutputDir) + cacheDir := absOrEmpty(opts.CacheDir) + out := map[string]schema.GoArtifact{} + err = filepath.WalkDir(absRoot, func(p string, d fs.DirEntry, werr error) error { + if werr != nil { + return werr + } + if d.IsDir() { + if p != absRoot && (skipDirNames[d.Name()] || (cacheDir != "" && p == cacheDir)) { + return filepath.SkipDir + } + return nil + } + if !d.Type().IsRegular() || strings.HasSuffix(d.Name(), ".go") { + return nil + } + if outDir != "" && filepath.Dir(p) == outDir && outputFileNames[d.Name()] { + return nil + } + rel, err := filepath.Rel(absRoot, p) + if err != nil { + return err + } + art, err := readArtifact(p, filepath.ToSlash(rel), opts.CaptureText) + if err != nil { + return err + } + out[art.Path] = art + return nil + }) + if err != nil { + return nil, fmt.Errorf("artifact discovery: %w", err) + } + return out, nil +} + +func absOrEmpty(p string) string { + if p == "" { + return "" + } + abs, err := filepath.Abs(p) + if err != nil { + return "" + } + return abs +} + +// readArtifact hashes the whole file and classifies it. Text source is kept only +// when captureText is set; binaries are never captured. +func readArtifact(abs, rel string, captureText bool) (schema.GoArtifact, error) { + data, err := os.ReadFile(abs) + if err != nil { + return schema.GoArtifact{}, err + } + sum := sha256.Sum256(data) + art := schema.GoArtifact{ + Path: rel, + Format: formatOf(rel, data), + SizeBytes: int64(len(data)), + SHA256: hex.EncodeToString(sum[:]), + Extraction: "none", + } + switch rel { + case "go.mod": + art.Roles = []string{"dependency-manifest"} + case "go.sum": + art.Roles = []string{"lockfile"} + } + if captureText && art.Format != "binary" { + art.Source = string(data) + art.Extraction = "full" + } + return art, nil +} + +// formatOf names the file's format. go.mod and go.sum have their own names; the +// rest follow the Python analyzer's enum. Binary is a NUL byte or invalid UTF-8. +func formatOf(rel string, data []byte) string { + base := filepath.Base(rel) + switch { + case base == "go.mod": + return "gomod" + case base == "go.sum": + return "gosum" + case base == "Dockerfile" || strings.HasPrefix(base, "Dockerfile.") || strings.HasSuffix(base, ".dockerfile"): + return "dockerfile" + case strings.HasPrefix(base, "requirements") && strings.HasSuffix(base, ".txt"): + return "requirements" + } + if bytes.IndexByte(data, 0) >= 0 || !utf8.Valid(data) { + return "binary" + } + switch strings.ToLower(filepath.Ext(base)) { + case ".toml": + return "toml" + case ".yaml", ".yml": + return "yaml" + case ".json": + return "json" + case ".ini", ".cfg": + return "ini" + case ".properties": + return "properties" + } + return "text" +} + +// Facts builds the repository layer for a project: its artifacts, and the +// dependencies declared by the root go.mod. imports are the project's import +// paths (from the symbol table), used to fill provides_imports. +func Facts(root string, app *schema.GoApplication, opts Options) (*schema.RepositoryFacts, error) { + artifacts, err := Discover(root, opts) + if err != nil { + return nil, err + } + facts := &schema.RepositoryFacts{Artifacts: artifacts} + modPath := filepath.Join(root, "go.mod") + modData, err := os.ReadFile(modPath) + if err != nil { + // No root go.mod: artifacts only, no dependencies. + return facts, nil + } + f, err := modfile.Parse("go.mod", modData, nil) + if err != nil { + utils.Warn("repository: go.mod does not parse, dependencies skipped: %v", err) + return facts, nil + } + locked := parseGoSum(filepath.Join(root, "go.sum")) + imports := projectImports(app) + + deps := make([]schema.GoDependency, 0, len(f.Require)) + for _, r := range f.Require { + dep := schema.GoDependency{ + Name: r.Mod.Path, + Ecosystem: Ecosystem, + Spec: r.Mod.Version, + Kind: "runtime", + Direct: !r.Indirect, + DeclaredIn: "go.mod", + Prov: []string{"declared"}, + } + if locked[r.Mod.Path][r.Mod.Version] { + dep.LockedVersion = r.Mod.Version + dep.Prov = append(dep.Prov, "lockfile") + } + deps = append(deps, dep) + } + sort.Slice(deps, func(i, j int) bool { return deps[i].Name < deps[j].Name }) + assignImports(deps, imports) + facts.Dependencies = deps + return facts, nil +} + +// parseGoSum returns, per module path and version, whether go.sum carries both +// the module's h1 line and its /go.mod line. Only such a version is a locked +// version. A missing or unreadable go.sum locks nothing. +func parseGoSum(path string) map[string]map[string]bool { + type seen struct{ h1, gomod bool } + entries := map[string]map[string]*seen{} + data, err := os.ReadFile(path) + if err != nil { + return map[string]map[string]bool{} + } + for _, line := range strings.Split(string(data), "\n") { + fields := strings.Fields(line) + if len(fields) != 3 { + continue + } + mod, ver := fields[0], fields[1] + gomod := strings.HasSuffix(ver, "/go.mod") + ver = strings.TrimSuffix(ver, "/go.mod") + if entries[mod] == nil { + entries[mod] = map[string]*seen{} + } + if entries[mod][ver] == nil { + entries[mod][ver] = &seen{} + } + if gomod { + entries[mod][ver].gomod = true + } else { + entries[mod][ver].h1 = true + } + } + out := map[string]map[string]bool{} + for mod, vers := range entries { + out[mod] = map[string]bool{} + for ver, s := range vers { + out[mod][ver] = s.h1 && s.gomod + } + } + return out +} + +// projectImports is the sorted, deduplicated set of import paths the project's +// (non-test, when the symbol table was built with --skip-tests) Go files use. +func projectImports(app *schema.GoApplication) []string { + seen := map[string]bool{} + for _, f := range app.SymbolTable { + for _, imp := range f.Imports { + seen[imp.Module] = true + } + } + out := make([]string, 0, len(seen)) + for p := range seen { + out = append(out, p) + } + sort.Strings(out) + return out +} + +// assignImports gives each import to the dependency whose module path is its +// longest prefix (the path itself, or a path under it followed by "/"). An +// import matching no requirement is not recorded. +func assignImports(deps []schema.GoDependency, imports []string) { + for _, imp := range imports { + best := -1 + bestLen := -1 + for i, d := range deps { + if (imp == d.Name || strings.HasPrefix(imp, d.Name+"/")) && len(d.Name) > bestLen { + best, bestLen = i, len(d.Name) + } + } + if best >= 0 { + deps[best].ProvidesImports = append(deps[best].ProvidesImports, imp) + } + } +} diff --git a/internal/repository/repository_test.go b/internal/repository/repository_test.go new file mode 100644 index 0000000..0c048ef --- /dev/null +++ b/internal/repository/repository_test.go @@ -0,0 +1,235 @@ +package repository + +import ( + "os" + "path/filepath" + "reflect" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" +) + +// writeTree creates the files under root (path → content). A NUL in content +// makes a file binary. +func writeTree(t *testing.T, root string, files map[string]string) { + t.Helper() + for rel, content := range files { + p := filepath.Join(root, filepath.FromSlash(rel)) + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } +} + +// appWithImports is a symbol-table stub carrying the given import paths in one file. +func appWithImports(paths ...string) *schema.GoApplication { + imps := make([]schema.GoImport, 0, len(paths)) + for _, p := range paths { + imps = append(imps, schema.GoImport{Module: p}) + } + return &schema.GoApplication{SymbolTable: map[string]schema.GoFile{"main.go": {Imports: imps}}} +} + +const goMod = `module example.com/app + +go 1.25.0 + +require ( + cloud.google.com/go v0.1.0 + cloud.google.com/go/auth v0.2.0 + github.com/gorilla/mux v1.8.0 + golang.org/x/text v0.14.0 // indirect +) +` + +// go.sum: mux has both lines for its version (locked); auth too; cloud.google.com/go +// has only its h1 line (not locked); x/text has only its /go.mod line (not locked). +const goSum = `cloud.google.com/go v0.1.0 h1:AAA= +cloud.google.com/go/auth v0.2.0 h1:BBB= +cloud.google.com/go/auth v0.2.0/go.mod h1:CCC= +github.com/gorilla/mux v1.8.0 h1:DDD= +github.com/gorilla/mux v1.8.0/go.mod h1:EEE= +golang.org/x/text v0.14.0/go.mod h1:FFF= +` + +func TestFacts_DependenciesLocksAndImports(t *testing.T) { + root := t.TempDir() + writeTree(t, root, map[string]string{"go.mod": goMod, "go.sum": goSum, "main.go": "package main\n"}) + + facts, err := Facts(root, appWithImports( + "github.com/gorilla/mux", + "cloud.google.com/go/auth/credentials", // longest prefix: cloud.google.com/go/auth + "cloud.google.com/go/storage", // only cloud.google.com/go matches + "fmt", // stdlib: no requirement + ), Options{}) + if err != nil { + t.Fatal(err) + } + + want := []schema.GoDependency{ + {Name: "cloud.google.com/go", Ecosystem: "golang", Spec: "v0.1.0", Kind: "runtime", Direct: true, + DeclaredIn: "go.mod", ProvidesImports: []string{"cloud.google.com/go/storage"}, Prov: []string{"declared"}}, + {Name: "cloud.google.com/go/auth", Ecosystem: "golang", Spec: "v0.2.0", Kind: "runtime", Direct: true, + LockedVersion: "v0.2.0", DeclaredIn: "go.mod", ProvidesImports: []string{"cloud.google.com/go/auth/credentials"}, + Prov: []string{"declared", "lockfile"}}, + {Name: "github.com/gorilla/mux", Ecosystem: "golang", Spec: "v1.8.0", Kind: "runtime", Direct: true, + LockedVersion: "v1.8.0", DeclaredIn: "go.mod", ProvidesImports: []string{"github.com/gorilla/mux"}, + Prov: []string{"declared", "lockfile"}}, + {Name: "golang.org/x/text", Ecosystem: "golang", Spec: "v0.14.0", Kind: "runtime", Direct: false, + DeclaredIn: "go.mod", Prov: []string{"declared"}}, + } + if !reflect.DeepEqual(facts.Dependencies, want) { + t.Errorf("dependencies:\n got %+v\n want %+v", facts.Dependencies, want) + } +} + +func TestFacts_ArtifactsClassifiedAndSkipped(t *testing.T) { + root := t.TempDir() + writeTree(t, root, map[string]string{ + "go.mod": goMod, + "go.sum": goSum, + "main.go": "package main\n", + "main_test.go": "package main\n", + "Dockerfile": "FROM scratch\n", + "config/app.yaml": "a: 1\n", + "config/settings.env": "KEY=value\n", + "blob.dat": "ab\x00cd", + ".git/HEAD": "ref: refs/heads/main\n", + "vendor/x/y.txt": "vendored\n", + }) + + facts, err := Facts(root, appWithImports(), Options{}) + if err != nil { + t.Fatal(err) + } + got := map[string]schema.GoArtifact{} + for p, a := range facts.Artifacts { + got[p] = a + } + wantFormats := map[string]string{ + "go.mod": "gomod", + "go.sum": "gosum", + "Dockerfile": "dockerfile", + "config/app.yaml": "yaml", + "config/settings.env": "text", + "blob.dat": "binary", + } + if len(got) != len(wantFormats) { + t.Errorf("artifacts = %v, want exactly %v", keys(got), keys(wantFormats)) + } + for p, f := range wantFormats { + a, ok := got[p] + if !ok { + t.Errorf("missing artifact %s", p) + continue + } + if a.Format != f { + t.Errorf("%s format = %q, want %q", p, a.Format, f) + } + if a.Extraction != "none" || a.Source != "" { + t.Errorf("%s captured source without --artifact-text: %+v", p, a) + } + if len(a.SHA256) != 64 { + t.Errorf("%s sha256 = %q, want 64 hex chars", p, a.SHA256) + } + } + if r := got["go.mod"].Roles; !reflect.DeepEqual(r, []string{"dependency-manifest"}) { + t.Errorf("go.mod roles = %v", r) + } + if r := got["go.sum"].Roles; !reflect.DeepEqual(r, []string{"lockfile"}) { + t.Errorf("go.sum roles = %v", r) + } +} + +func TestDiscover_CaptureTextOnlyForText(t *testing.T) { + root := t.TempDir() + writeTree(t, root, map[string]string{"go.mod": goMod, "blob.dat": "ab\x00cd"}) + + on, err := Discover(root, Options{CaptureText: true}) + if err != nil { + t.Fatal(err) + } + if on["go.mod"].Source != goMod || on["go.mod"].Extraction != "full" { + t.Errorf("go.mod with --artifact-text: source/extraction = %q/%q", on["go.mod"].Source, on["go.mod"].Extraction) + } + if on["blob.dat"].Source != "" || on["blob.dat"].Extraction != "none" { + t.Errorf("binary captured source: %+v", on["blob.dat"]) + } +} + +func TestDiscover_ExcludesOwnOutputsAndCache(t *testing.T) { + root := t.TempDir() + cache := filepath.Join(root, "cache") + writeTree(t, root, map[string]string{ + "README.md": "# app\n", + "analysis.json": "{}", + "graph.cypher": "CREATE;", + "cache/entry.bin": "x\x00", + "notes/analysis.json": "{}", // not the output dir: kept + }) + + got, err := Discover(root, Options{OutputDir: root, CacheDir: cache}) + if err != nil { + t.Fatal(err) + } + if _, ok := got["analysis.json"]; ok { + t.Errorf("own output analysis.json was captured") + } + if _, ok := got["graph.cypher"]; ok { + t.Errorf("own output graph.cypher was captured") + } + if _, ok := got["cache/entry.bin"]; ok { + t.Errorf("cache subtree was captured") + } + if _, ok := got["notes/analysis.json"]; !ok { + t.Errorf("unrelated notes/analysis.json was wrongly excluded") + } + if _, ok := got["README.md"]; !ok { + t.Errorf("README.md missing") + } +} + +func TestFacts_NoGoModMeansArtifactsOnly(t *testing.T) { + root := t.TempDir() + writeTree(t, root, map[string]string{"Dockerfile": "FROM scratch\n"}) + + facts, err := Facts(root, appWithImports("fmt"), Options{}) + if err != nil { + t.Fatal(err) + } + if len(facts.Dependencies) != 0 { + t.Errorf("dependencies without go.mod: %+v", facts.Dependencies) + } + if _, ok := facts.Artifacts["Dockerfile"]; !ok { + t.Errorf("Dockerfile artifact missing") + } +} + +func TestParseGoSum_OnlyFullPairsLock(t *testing.T) { + root := t.TempDir() + writeTree(t, root, map[string]string{"go.sum": goSum}) + got := parseGoSum(filepath.Join(root, "go.sum")) + if !got["github.com/gorilla/mux"]["v1.8.0"] { + t.Errorf("mux v1.8.0 has both lines; want locked") + } + if got["cloud.google.com/go"]["v0.1.0"] { + t.Errorf("cloud.google.com/go has only h1; want not locked") + } + if got["golang.org/x/text"]["v0.14.0"] { + t.Errorf("golang.org/x/text has only /go.mod; want not locked") + } + if n := len(parseGoSum(filepath.Join(root, "absent"))); n != 0 { + t.Errorf("missing go.sum locked %d modules", n) + } +} + +func keys[V any](m map[string]V) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + return out +} diff --git a/internal/schema/schema.go b/internal/schema/schema.go index 84335d5..99bf2c2 100644 --- a/internal/schema/schema.go +++ b/internal/schema/schema.go @@ -214,6 +214,44 @@ type GoApplication struct { // Externals holds the library (out-of-project) functions the resolver saw // called, keyed by the full callee signature (".[Type.]Name"). Externals map[string]GoExternalCallee `json:"externals,omitempty"` + // Repository holds the repository-artifact facts (non-.go project files and + // go.mod/go.sum dependencies), set by the analyzer after symbol-table + // construction. Never cached or serialized in the v1 model: it is recomputed + // on every run, since it reads files outside the analyzed Go source. + Repository *RepositoryFacts `json:"-"` +} + +// RepositoryFacts is the repository-artifact layer (spec v2-artifacts): every +// non-.go project file as an artifact, and the go.mod requires as dependencies. +type RepositoryFacts struct { + // Artifacts is keyed by repo-relative POSIX path. + Artifacts map[string]GoArtifact + // Dependencies is sorted by module path. + Dependencies []GoDependency +} + +// GoArtifact is one non-.go project file. +type GoArtifact struct { + Path string + Format string + Roles []string + SizeBytes int64 + SHA256 string + Source string // whole file text, only when --artifact-text is on + Extraction string // none | full +} + +// GoDependency is one go.mod require, evidence-tagged by go.sum. +type GoDependency struct { + Name string + Ecosystem string + Spec string + Kind string + Direct bool + LockedVersion string // go.sum version of Spec, when both its h1 and /go.mod lines exist + DeclaredIn string // path of the go.mod artifact + ProvidesImports []string // project import paths that resolve to this module + Prov []string // declared | lockfile } // GoExternalCallee describes a library function or method called from the diff --git a/internal/schema/v2/ids.go b/internal/schema/v2/ids.go index e2e0788..2b2cf5f 100644 --- a/internal/schema/v2/ids.go +++ b/internal/schema/v2/ids.go @@ -15,18 +15,26 @@ import ( // Durable id grammar (the app segment is outermost so a polyglot app shares one // `can:///` prefix across languages; the language follows it): // -// can://///// -// can://myapp/myapp/go/src/util.go/Hasher/Hash(string)uint64 +// can:///// module +// can:////// type +// can://///// method +// can:////// package-level function +// can://myapp/myapp/go/util.go/Hasher/Hash e.g. a method +// +// Each member segment is the bare name: the package import path is dropped (it is +// always before the last dot of the signature, and Go names contain no dots). +// Closures nest under their enclosing callable as closure_N. // // External (out-of-project) symbols are language-neutral: // -// can:////@external// +// can:////@external/// +// can://myapp/myapp/@external/strings/Builder/WriteString // // Ordinal id grammar (statements and synthetic vertices, addressed WITHIN their // callable): // -// @: e.g. …/Hash(string)uint64@15:2 -// @ e.g. …/Hash(string)uint64@entry +// @: e.g. …/Hasher/Hash@15:2 +// @ e.g. …/Hasher/Hash@entry // // The delimiters `/`, `@`, `:` are fixed by the keystone; do not substitute. const ( @@ -91,13 +99,29 @@ func ModuleID(appID, relPath string) string { } // ExternalID builds the id of an out-of-project symbol: -// /@external//. It is language-neutral. An empty sig -// (an import-level ghost, no symbol) yields /@external/. +// /@external/// (or .../ for a +// package-level function). sig is the callee signature without the import-path +// prefix ("Client.Zone"); its dots become path segments, matching the in-project +// form. It is language-neutral. An empty sig (an import-level ghost, no symbol) +// yields /@external/. func ExternalID(appID, importPath, sig string) string { if sig == "" { return appID + idSep + "@external" + idSep + importPath } - return appID + idSep + "@external" + idSep + importPath + idSep + sig + return appID + idSep + "@external" + idSep + importPath + idSep + strings.ReplaceAll(sig, ".", idSep) +} + +// memberName is the member segment a symbol contributes under its parent: the +// part of its signature after the last dot. The package import path is always +// before that dot and Go identifiers contain no dots, so this strips the path +// and keeps the bare name (a function or type name, a method name, or a closure +// name such as closure_1). Precedent: codeanalyzer-typescript idFromSig and +// codeanalyzer-python child_id. +func memberName(sig string) string { + if i := strings.LastIndex(sig, "."); i >= 0 { + return sig[i+1:] + } + return sig } // GlobalOrdinal joins a callable id and a local ordinal: callableID + local @@ -110,17 +134,18 @@ func GlobalOrdinal(callableID, local string) string { return callableID + ordinalSep + local } -// TypeID builds a type id under its module. sig is the type's signatureOf() -// output — the last (and only) segment the type contributes. +// TypeID builds a type id under its module: /. sig is the type's +// signatureOf() output; the package path is dropped (see memberName). func TypeID(moduleID, sig string) string { - return moduleID + idSep + sig + return moduleID + idSep + memberName(sig) } -// CallableID builds a callable id under its parent. For a method the parent is -// its receiver Type id; for a module-level function the parent is the Module -// id. sig is the callable's signatureOf() output — the last path segment. +// CallableID builds a callable id under its parent: / for a +// method (parent is the receiver Type id), / for a package-level +// function (parent is the Module id), /closure_N for a closure. sig is +// the callable's signatureOf() output; the package path is dropped (see memberName). func CallableID(parentID, sig string) string { - return parentID + idSep + sig + return parentID + idSep + memberName(sig) } // OrdinalID addresses a real body node (statement / call) within its callable diff --git a/internal/schema/v2/ids_member_test.go b/internal/schema/v2/ids_member_test.go new file mode 100644 index 0000000..4cb931a --- /dev/null +++ b/internal/schema/v2/ids_member_test.go @@ -0,0 +1,76 @@ +package v2 + +import "testing" + +// The member segment is the signature's last dot-separated part, so the package +// import path (which may itself contain dots) never reaches the id. +func TestMemberIDs_DropPackagePath(t *testing.T) { + mod := ModuleID("can://ns/app", "go/w.go") + cases := []struct { + name string + got string + want string + }{ + {"type", TypeID(mod, "github.com/acme/app/worker.Worker"), mod + "/Worker"}, + {"function", CallableID(mod, "github.com/acme/app/worker.Run"), mod + "/Run"}, + {"method", CallableID(TypeID(mod, "github.com/acme/app/worker.Worker"), "github.com/acme/app/worker.Worker.Run"), mod + "/Worker/Run"}, + {"closure", CallableID(mod+"/Run", "github.com/acme/app/worker.Run.closure_2"), mod + "/Run/closure_2"}, + {"dotted import path", CallableID(mod, "gopkg.in/yaml.v2.Marshal"), mod + "/Marshal"}, + {"init", CallableID(mod, "github.com/acme/app/worker.init"), mod + "/init"}, + } + for _, c := range cases { + if c.got != c.want { + t.Errorf("%s: got %q, want %q", c.name, c.got, c.want) + } + } +} + +// Library callees: the import path stays whole, and the member's dots become +// segments, so Type and Member are separate segments. +func TestExternalID_PathForm(t *testing.T) { + const app = "can://ns/app" + cases := []struct { + importPath, sig, want string + }{ + {"strings", "Builder.WriteString", app + "/@external/strings/Builder/WriteString"}, + {"fmt", "Println", app + "/@external/fmt/Println"}, + {"database/sql", "DB.Query", app + "/@external/database/sql/DB/Query"}, + {"cloud.google.com/go/compute/metadata", "Client.Zone", app + "/@external/cloud.google.com/go/compute/metadata/Client/Zone"}, + {"net/http", "", app + "/@external/net/http"}, + } + for _, c := range cases { + if got := ExternalID(app, c.importPath, c.sig); got != c.want { + t.Errorf("ExternalID(%q, %q) = %q, want %q", c.importPath, c.sig, got, c.want) + } + } +} + +// Two library symbols with the same member name in different packages must keep +// distinct ids: the import path is the disambiguator. +func TestExternalID_SameMemberDifferentPackagesDistinct(t *testing.T) { + const app = "can://ns/app" + a := ExternalID(app, "encoding/json", "Decoder.Decode") + b := ExternalID(app, "encoding/xml", "Decoder.Decode") + if a == b { + t.Fatalf("encoding/json and encoding/xml Decoder.Decode collide at %s", a) + } +} + +// Within one module, every member name is unique: Go forbids duplicate package- +// scope identifiers and duplicate method names per type, so the bare names are +// safe as keys under their parent. +func TestMemberIDs_UniqueWithinModule(t *testing.T) { + mod := ModuleID("can://ns/app", "go/a.go") + ids := map[string]bool{} + for _, id := range []string{ + TypeID(mod, "pkg.T"), + CallableID(mod, "pkg.F"), + CallableID(TypeID(mod, "pkg.T"), "pkg.T.M"), + CallableID(TypeID(mod, "pkg.T"), "pkg.T.N"), + } { + if ids[id] { + t.Errorf("duplicate id %s", id) + } + ids[id] = true + } +} diff --git a/internal/schema/v2/ids_test.go b/internal/schema/v2/ids_test.go index 72e4ee3..7361348 100644 --- a/internal/schema/v2/ids_test.go +++ b/internal/schema/v2/ids_test.go @@ -97,7 +97,7 @@ func TestIDs_AppOutermost(t *testing.T) { cases := []struct{ got, want string }{ {app, "can://shop/shop"}, {ModuleID(app, "pkg/a.go"), "can://shop/shop/go/pkg/a.go"}, - {ExternalID(app, "database/sql", "DB.Query"), "can://shop/shop/@external/database/sql/DB.Query"}, + {ExternalID(app, "database/sql", "DB.Query"), "can://shop/shop/@external/database/sql/DB/Query"}, {GlobalOrdinal("X/F", "@entry"), "X/F@entry"}, {GlobalOrdinal("X/F", "12:3"), "X/F@12:3"}, } diff --git a/internal/schema/v2/schema.go b/internal/schema/v2/schema.go index 800c694..3c5ca1a 100644 --- a/internal/schema/v2/schema.go +++ b/internal/schema/v2/schema.go @@ -97,6 +97,12 @@ type Application struct { // reaches, keyed by their `can:////@external//` id. // Absent when the analysis saw no library call. ExternalSymbols map[string]ExternalSymbol `json:"external_symbols,omitempty"` + // Artifacts holds every non-.go project file, keyed by its repo-relative path + // (spec v2-artifacts). Level-independent; absent when the project has no files. + Artifacts map[string]Artifact `json:"artifacts,omitempty"` + // Dependencies holds the go.mod requires, sorted by module path. Absent when + // the project has no root go.mod. + Dependencies []Dependency `json:"dependencies,omitempty"` // ParamIn and ParamOut are the L4 interprocedural parameter edges between // global body-node ids. Absent below -a 4. ParamIn []ParamEdge `json:"param_in,omitempty"` @@ -179,6 +185,9 @@ type Type struct { ID string `json:"id"` Kind string `json:"kind"` // struct | interface | alias | defined Span Span `json:"span"` + // Signature is the type's full import-qualified name (one signatureOf()). The + // id carries only the bare type name, so this is where the package path lives. + Signature string `json:"signature"` // BaseTypes are embedded type ids (struct/interface embedding — the explicit // spine). BaseTypes []string `json:"base_types,omitempty"` @@ -208,7 +217,8 @@ type Callable struct { ID string `json:"id"` Kind string `json:"kind"` // function | method | lambda Span Span `json:"span"` - // Signature is the human-readable last path segment of ID (one signatureOf()). + // Signature is the full import-qualified name (one signatureOf()). The id + // carries only the bare member name, so this field is where the package path lives. Signature string `json:"signature"` // SourceFile names the callable's declaring file (a symbol_table key) when it // differs from the module this callable is nested under — the Go case where a @@ -341,3 +351,31 @@ type Edge struct { // Weight accumulates when merging edges from multiple backends. Weight int `json:"weight,omitempty"` } + +// Artifact is one non-.go project file. Its id is can:////artifact/. +// Source is the whole file text only under --artifact-text; otherwise absent. +type Artifact struct { + ID string `json:"id"` + Kind string `json:"kind"` // always "artifact" + Path string `json:"path"` + Format string `json:"format"` + Roles []string `json:"roles,omitempty"` + SizeBytes int64 `json:"size_bytes"` + SHA256 string `json:"sha256"` + Source string `json:"source,omitempty"` + Extraction string `json:"extraction"` // none | full +} + +// Dependency is one declared module requirement from go.mod. Its package is +// pkg:/, version-less; the version lives on the record. +type Dependency struct { + Name string `json:"name"` + Ecosystem string `json:"ecosystem"` // always "golang" + Spec string `json:"spec"` // version required by go.mod + Kind string `json:"kind"` // always "runtime" (go.mod has no dev/optional split) + Direct bool `json:"direct"` // false for `// indirect` + LockedVersion string `json:"locked_version,omitempty"` + DeclaredIn string `json:"declared_in"` // the go.mod artifact id + ProvidesImports []string `json:"provides_imports,omitempty"` + Prov []string `json:"prov"` // declared | lockfile +} diff --git a/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go b/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go index b4b7d6a..f0d73ac 100644 --- a/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go +++ b/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go @@ -116,7 +116,7 @@ func TestInitCollision_DistinctNodesAndEdges(t *testing.T) { continue } for _, fn := range mod.Functions { - if strings.HasSuffix(fn.ID, "middleware.init") { + if strings.HasSuffix(fn.ID, "/init") { initNodeIDs[fn.ID] = true } } @@ -131,7 +131,7 @@ func TestInitCollision_DistinctNodesAndEdges(t *testing.T) { // edge resolved to ITS OWN file's init, so the id is self-consistent). sawInitEdge := false for _, e := range out.Application.CallGraph { - if !strings.HasSuffix(e.Src, "middleware.init") { + if !strings.HasSuffix(e.Src, "/init") { continue } sawInitEdge = true diff --git a/internal/syntactic_analysis/v2emit/emit.go b/internal/syntactic_analysis/v2emit/emit.go index 3f0279e..5268552 100644 --- a/internal/syntactic_analysis/v2emit/emit.go +++ b/internal/syntactic_analysis/v2emit/emit.go @@ -254,6 +254,7 @@ func (e *emitter) envelope(app *schema.GoApplication, maxLevel int, analyzerVers }, } out.Application.ExternalSymbols = emitExternalSymbols(e.appID, app.Externals, e.sdg.standIns()) + out.Application.Artifacts, out.Application.Dependencies = emitRepository(e.appID, app.Repository) if e.flow.on { out.KLimit = kLimit } @@ -291,7 +292,7 @@ func emitModule(appID, modRel string, file schema.GoFile, idx sigIndex, srcs *so } for name, t := range file.Types { - mod.Types[name] = emitType(modID, modRel, t, idx, srcs, flow, l4) + mod.Types[name] = emitType(appID, modRel, t, idx, srcs, flow, l4) } for sig, fn := range file.Functions { mod.Functions[sig] = emitCallable(modID, modRel, modRel, li, fn, idx, srcs, flow, l4) @@ -303,12 +304,13 @@ func emitModule(appID, modRel string, file schema.GoFile, idx sigIndex, srcs *so // kind and splitting embedding (base_types) from computed satisfaction. Methods // resolve into the type's callables{}. modRel is the module the type lives in; // its methods may be declared in other files (handled in emitCallable). -func emitType(modID, modRel string, t schema.GoType, idx sigIndex, srcs *sourceCache, flow l3Overlay, l4 *l4Overlay) v2.Type { +func emitType(appID, modRel string, t schema.GoType, idx sigIndex, srcs *sourceCache, flow l3Overlay, l4 *l4Overlay) v2.Type { li := srcs.index(modRel) - typeID := v2.TypeID(modID, t.Signature) + typeID := v2.TypeID(v2.ModuleID(appID, modRel), t.Signature) out := v2.Type{ ID: typeID, Kind: typeKind(t), + Signature: t.Signature, Span: lineSpan(li, t.StartLine, t.EndLine), BaseTypes: nonEmpty(t.BaseTypes), Callables: make(map[string]v2.Callable, len(t.Methods)), @@ -325,7 +327,7 @@ func emitType(modID, modRel string, t schema.GoType, idx sigIndex, srcs *sourceC } } for sig, m := range t.Methods { - out.Callables[sig] = emitCallable(typeID, modRel, modRel, li, m, idx, srcs, flow, l4) + out.Callables[sig] = emitCallable(methodParentID(appID, modRel, t, m), modRel, modRel, li, m, idx, srcs, flow, l4) } return out } diff --git a/internal/syntactic_analysis/v2emit/gate_l2_test.go b/internal/syntactic_analysis/v2emit/gate_l2_test.go index d402ed6..412d424 100644 --- a/internal/syntactic_analysis/v2emit/gate_l2_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l2_test.go @@ -110,8 +110,8 @@ func TestGateL2_NamedEdgePresent(t *testing.T) { out := buildMultipackageL2(t) const ( - src = "can://multipackage/multipackage/go/worker/worker.go/example.com/multipackage/worker.Worker/example.com/multipackage/worker.Worker.Run" - dst = "can://multipackage/multipackage/go/worker/worker.go/example.com/multipackage/worker.Worker/example.com/multipackage/worker.Worker.execute" + src = "can://multipackage/multipackage/go/worker/worker.go/Worker/Run" + dst = "can://multipackage/multipackage/go/worker/worker.go/Worker/execute" ) found := false for _, e := range out.Application.CallGraph { @@ -133,8 +133,8 @@ func TestGateL2_CrossPackageEdge(t *testing.T) { out := buildMultipackageL2(t) const ( - src = "can://multipackage/multipackage/go/main.go/example.com/multipackage.main" - dst = "can://multipackage/multipackage/go/server/server.go/example.com/multipackage/server.New" + src = "can://multipackage/multipackage/go/main.go/main" + dst = "can://multipackage/multipackage/go/server/server.go/New" ) found := false for _, e := range out.Application.CallGraph { diff --git a/internal/syntactic_analysis/v2emit/gate_l3_test.go b/internal/syntactic_analysis/v2emit/gate_l3_test.go index a9f17fc..3b98d89 100644 --- a/internal/syntactic_analysis/v2emit/gate_l3_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l3_test.go @@ -502,13 +502,12 @@ func TestGateL3_LineDirectiveRealPositions(t *testing.T) { func TestSymbolTable_MethodDeclaredAboveItsType(t *testing.T) { fx := buildL3(t, "dataflow") const sig = "example.com/dataflow/flow.early.Bump" - const fID = "example.com/dataflow/flow.f" find := func(out *v2.Analysis) (v2.Callable, bool) { for _, mod := range out.Application.SymbolTable { for _, ty := range mod.Types { for _, m := range ty.Callables { - if strings.HasSuffix(m.ID, "/"+sig) { + if strings.HasSuffix(m.ID, "/early/Bump") { return m, true } } @@ -531,7 +530,7 @@ func TestSymbolTable_MethodDeclaredAboveItsType(t *testing.T) { if call == nil { t.Fatalf("%s has no call site", sig) } - if call.Callee == nil || !strings.HasSuffix(*call.Callee, "/"+fID) { + if call.Callee == nil || !strings.HasSuffix(*call.Callee, "/f") { t.Fatalf("call to f not resolved at -a 2: callee = %v", call.Callee) } sawEdge := false diff --git a/internal/syntactic_analysis/v2emit/gate_l4_done_test.go b/internal/syntactic_analysis/v2emit/gate_l4_done_test.go index 6803719..c22142b 100644 --- a/internal/syntactic_analysis/v2emit/gate_l4_done_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l4_done_test.go @@ -144,7 +144,7 @@ func TestGateL4_HTTPToSQLPath(t *testing.T) { method := func(typ, m string) string { return v2.CallableID(v2.TypeID(mod, pkg+typ), pkg+typ+"."+m) } sink := vertex(v2.ExternalID(app, "database/sql", "DB.Query"), "formal_in", "query") - if sink != "can://httpsql/httpsql/@external/database/sql/DB.Query@formal_in:1" { + if sink != "can://httpsql/httpsql/@external/database/sql/DB/Query@formal_in:1" { t.Fatalf("sink vertex %s", sink) } find := method("sqlRepo", "Find") diff --git a/internal/syntactic_analysis/v2emit/gate_test.go b/internal/syntactic_analysis/v2emit/gate_test.go index 41822be..86382da 100644 --- a/internal/syntactic_analysis/v2emit/gate_test.go +++ b/internal/syntactic_analysis/v2emit/gate_test.go @@ -578,7 +578,6 @@ func TestGateL1_BodyNodeIDs(t *testing.T) { func TestGateL2_GoStatementArgumentCalls(t *testing.T) { out := buildDataflowL2V2(t) fn := functionByName(t, mustModule(t, out, "flow/flow.go"), "GoArgs") - const pkg = "example.com/dataflow/flow." for _, w := range []struct { key, callee string goroutine bool @@ -599,7 +598,7 @@ func TestGateL2_GoStatementArgumentCalls(t *testing.T) { if w.callee == "" { continue } - want := "can://dataflow/dataflow/go/flow/flow.go/" + pkg + w.callee + want := "can://dataflow/dataflow/go/flow/flow.go/" + w.callee if n.Callee == nil || *n.Callee != want { t.Errorf("GoArgs %s callee = %v, want %s", w.key, n.Callee, want) } diff --git a/internal/syntactic_analysis/v2emit/repository.go b/internal/syntactic_analysis/v2emit/repository.go new file mode 100644 index 0000000..b68924f --- /dev/null +++ b/internal/syntactic_analysis/v2emit/repository.go @@ -0,0 +1,57 @@ +package v2emit + +import ( + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// artifactID is an artifact's durable id: the application root, then the +// artifact segment and the repo-relative path (spec v2-artifacts). +func artifactID(appID, relPath string) string { + return appID + "/artifact/" + relPath +} + +// emitRepository maps the repository facts onto the v2 layer. Artifacts are +// keyed by path; dependencies keep the facts' order (sorted by module path) and +// point at their go.mod artifact by id. A nil facts set emits nothing. +func emitRepository(appID string, facts *schema.RepositoryFacts) (map[string]v2.Artifact, []v2.Dependency) { + if facts == nil { + return nil, nil + } + var arts map[string]v2.Artifact + if len(facts.Artifacts) > 0 { + arts = make(map[string]v2.Artifact, len(facts.Artifacts)) + for path, a := range facts.Artifacts { + arts[path] = v2.Artifact{ + ID: artifactID(appID, path), + Kind: "artifact", + Path: a.Path, + Format: a.Format, + Roles: a.Roles, + SizeBytes: a.SizeBytes, + SHA256: a.SHA256, + Source: a.Source, + Extraction: a.Extraction, + } + } + } + var deps []v2.Dependency + for _, d := range facts.Dependencies { + declared := "" + if d.DeclaredIn != "" { + declared = artifactID(appID, d.DeclaredIn) + } + deps = append(deps, v2.Dependency{ + Name: d.Name, + Ecosystem: d.Ecosystem, + Spec: d.Spec, + Kind: d.Kind, + Direct: d.Direct, + LockedVersion: d.LockedVersion, + DeclaredIn: declared, + ProvidesImports: d.ProvidesImports, + Prov: d.Prov, + }) + } + return arts, deps +} diff --git a/internal/syntactic_analysis/v2emit/sigindex.go b/internal/syntactic_analysis/v2emit/sigindex.go index 00ed238..83a3f07 100644 --- a/internal/syntactic_analysis/v2emit/sigindex.go +++ b/internal/syntactic_analysis/v2emit/sigindex.go @@ -55,15 +55,28 @@ func buildSigIndex(appID string, symbolTable map[string]schema.GoFile) sigIndex idx.addCallable(modID, relPath, fn) } for _, t := range file.Types { - typeID := v2.TypeID(modID, t.Signature) for _, m := range t.Methods { - idx.addCallable(typeID, relPath, m) + idx.addCallable(methodParentID(appID, relPath, t, m), relPath, m) } } } return idx } +// methodParentID is the id a method nests under. A method declared in its +// receiver type's file nests under that type's id. A method declared in another +// file nests under the same type in the DECLARING file's module, so its id names +// where the method is written (spec v2-go-ids § Amendment). The JSON node itself +// stays in the receiver type's callables{}; only its id moves. modRel is the +// receiver's module (the file the type lives in). +func methodParentID(appID, modRel string, t schema.GoType, c schema.GoCallable) string { + mod := modRel + if c.Path != "" { + mod = c.Path + } + return v2.TypeID(v2.ModuleID(appID, mod), t.Signature) +} + // addCallable records one callable and recurses into its closures, matching the // id nesting emitCallable produces. relPath is the module file this callable is // reached through; it keys the disambiguating (signature,path) entry. diff --git a/scripts/eval_e2e.sh b/scripts/eval_e2e.sh new file mode 100755 index 0000000..34188ec --- /dev/null +++ b/scripts/eval_e2e.sh @@ -0,0 +1,187 @@ +#!/usr/bin/env bash +# eval_e2e.sh - end-to-end check: Go app -> analysis.json -> Neo4j projection -> loaded graph. +# +# Usage (from the repository root): +# scripts/eval_e2e.sh [--json-only] APP_DIR [OUT_DIR] +# +# APP_DIR a Go module root (must contain go.mod). +# OUT_DIR where analysis.json and the run log go (default: APP_DIR). +# --json-only write and check analysis.json only: no Neo4j push, no graph check, +# no password or running database needed. +# +# Example: +# NEO4J_PASSWORD='...' scripts/eval_e2e.sh /path/to/microservices-demo/src/frontend +# scripts/eval_e2e.sh --json-only /path/to/microservices-demo/src/frontend +# +# Environment: +# LEVEL analysis level for analysis.json (default 2; 1..4) +# APP_NAME --app-name anchor (default: APP_DIR base name, as cango does) +# CANGO analyzer binary. When unset, it is built from this checkout into +# OUT_DIR/cango with `go build ./cmd/codeanalyzer`. +# NEO4J_URI Bolt URI (default bolt://localhost:7687) +# NEO4J_USER username (default neo4j) +# NEO4J_PASSWORD password (default neo4j). Passed to cango through its env var; never echoed. +# EAGER 1 = reload: the push runs with --eager, purging this application's +# subgraph (its can:// prefix only) first. Default 0 (additive upsert). +# CYPHER_SHELL cypher-shell binary for verification (auto-detected when unset). When it +# cannot be found or connected, the count check is reported as skipped. +# +# Steps: +# 1. Validate APP_DIR and the requested level. +# 2. Build the analyzer (unless CANGO is set). +# 3. cango -a LEVEL -i APP_DIR -o OUT_DIR -> OUT_DIR/analysis.json +# 4. cango --emit neo4j --neo4j-uri URI ... -> live push (MERGE, idempotent) +# 5. Verify analysis.json (schema, level, app id) and that the database holds the +# pushed node count under this application's can:// prefix. +# +# Exit status: 0 when every check passes (or is skipped with a reason), non-zero otherwise. +set -euo pipefail + +JSON_ONLY=0 +if [ "${1:-}" = "--json-only" ]; then JSON_ONLY=1; shift; fi + +if [ $# -lt 1 ]; then + sed -n '2,/^set -euo/p' "$0" | sed '$d' >&2 + exit 2 +fi + +REPO=$(cd "$(dirname "$0")/.." && pwd) +APP=$(cd "$1" && pwd) +OUT=${2:-$APP} +mkdir -p "$OUT" +OUT=$(cd "$OUT" && pwd) + +LEVEL=${LEVEL:-2} +URI=${NEO4J_URI:-bolt://localhost:7687} +USER=${NEO4J_USER:-neo4j} +PASS=${NEO4J_PASSWORD:-neo4j} +APP_NAME=${APP_NAME:-$(basename "$APP")} +# The run log lives outside OUT: a log inside the project would be captured as an artifact. +LOG=${E2E_LOG:-${TMPDIR:-/tmp}/cldk-e2e-${APP_NAME}.log} + +fail=0 +step() { echo; echo "== $1"; } +pass() { echo " PASS: $1"; } +bad() { echo " FAIL: $1" >&2; fail=1; } +skip() { echo " SKIP: $1"; } + +# Strings that become id segments or Cypher literals must be plain names. +case "$APP_NAME" in + *[!A-Za-z0-9._-]*|"") echo "APP_NAME '$APP_NAME' must contain only letters, digits, '.', '_' or '-'" >&2; exit 2 ;; +esac +case "$LEVEL" in 1|2|3|4) ;; *) echo "LEVEL must be 1..4 (got '$LEVEL')" >&2; exit 2 ;; esac +[ -f "$APP/go.mod" ] || { echo "APP_DIR '$APP' has no go.mod; not a Go module root" >&2; exit 2; } + +: >"$LOG" + +step "1/5 analyzer binary" +if [ -n "${CANGO:-}" ]; then + [ -x "$CANGO" ] || { echo "CANGO '$CANGO' is not executable" >&2; exit 2; } + echo " using $CANGO" +else + # Build outside the project: a binary inside it would be captured as an artifact. + CANGO="${E2E_BUILD_DIR:-${TMPDIR:-/tmp}}/cldk-e2e-cango-${APP_NAME}" + (cd "$REPO" && go build -o "$CANGO" ./cmd/codeanalyzer) >>"$LOG" 2>&1 + echo " built $CANGO from $REPO" +fi + +step "2/5 analysis.json (-a $LEVEL)" +"$CANGO" -a "$LEVEL" --app-name "$APP_NAME" -i "$APP" -o "$OUT" >>"$LOG" 2>&1 +[ -s "$OUT/analysis.json" ] && pass "wrote $OUT/analysis.json ($(wc -c <"$OUT/analysis.json" | tr -d ' ') bytes)" \ + || { bad "analysis.json missing or empty (see $LOG)"; exit 1; } + +if [ "$JSON_ONLY" = "1" ]; then + step "3/5 Neo4j projection"; skip "--json-only: no push" +else +step "3/5 Neo4j projection (live push to $URI as $USER)" +# EAGER=1 reloads: --eager purges this application's subgraph (its can://// +# prefix only) before the push, so nodes from an earlier id format do not linger. +# A plain string, not an array: an empty array trips `set -u` on bash 3.2 (macOS). +EAGER_FLAG="" +[ "${EAGER:-0}" = "1" ] && EAGER_FLAG="--eager" +push_out=$(NEO4J_PASSWORD="$PASS" "$CANGO" --emit neo4j --neo4j-uri "$URI" --neo4j-user "$USER" \ + --app-name "$APP_NAME" $EAGER_FLAG -i "$APP" -o "$OUT" 2>&1) && push_rc=0 || push_rc=$? +printf '%s\n' "$push_out" >>"$LOG" +if [ "$push_rc" -ne 0 ]; then + bad "projection push failed (exit $push_rc): $(printf '%s' "$push_out" | tail -1)" + exit 1 +fi +pushed=$(printf '%s\n' "$push_out" | sed -n 's/^pushed \([0-9]*\) nodes, \([0-9]*\) relationships.*/\1 \2/p' | tail -1) +[ -n "$pushed" ] && pass "pushed $(printf '%s' "$pushed" | cut -d' ' -f1) nodes, $(printf '%s' "$pushed" | cut -d' ' -f2) relationships" \ + || bad "push output had no 'pushed N nodes' line" +fi + +step "4/5 verify analysis.json" +if command -v python3 >/dev/null 2>&1; then + if python3 -I - "$OUT/analysis.json" "$LEVEL" "$APP_NAME" <<'PY' +import json, sys +path, level, app = sys.argv[1], int(sys.argv[2]), sys.argv[3] +d = json.load(open(path, encoding="utf-8")) +errs = [] +if d.get("language") != "go": errs.append(f"language is {d.get('language')!r}") +if d.get("max_level") != level: errs.append(f"max_level is {d.get('max_level')!r}, expected {level}") +app_id = d.get("application", {}).get("id", "") +if not app_id.startswith(f"can://"): errs.append(f"application.id {app_id!r} is not a can:// id") +if app not in app_id: errs.append(f"application.id {app_id!r} lacks the app name {app!r}") +st = d.get("application", {}).get("symbol_table", {}) +if not st: errs.append("symbol_table is empty") +if errs: + print("\n".join(" FAIL: " + e for e in errs)); sys.exit(1) +print(f" PASS: schema {d.get('schema_version')}, level {level}, {len(st)} files, " + f"{len(d['application'].get('call_graph', []))} call edges, id {app_id}") +PY + then :; else fail=1; fi +else + skip "python3 not found; analysis.json not inspected" +fi + +if [ "$JSON_ONLY" = "1" ]; then + step "5/5 verify loaded graph"; skip "--json-only: graph not checked" +else +step "5/5 verify loaded graph" +CS=${CYPHER_SHELL:-} +if [ -z "$CS" ]; then + CS=$(command -v cypher-shell || true) + [ -n "$CS" ] || for c in /opt/homebrew/bin/cypher-shell /usr/local/bin/cypher-shell; do + [ -x "$c" ] && { CS=$c; break; } + done +fi +if [ -z "$CS" ]; then + skip "cypher-shell not found; set CYPHER_SHELL to enable the count check" +else + shell() { "$CS" -a "$URI" -u "$USER" -p "$PASS" --format plain "$@" 2>/dev/null; } + if ! shell "RETURN 1" >/dev/null; then + skip "cannot authenticate to $URI with cypher-shell; the count check was not run" + else + # Nodes keyed by this application's id root, the same prefix the projector scopes on. + prefix="can://${APP_NAME}/" + db_nodes=$(shell "MATCH (n:GoCanNode) WHERE n.id STARTS WITH '$prefix' RETURN count(n) AS c" | tail -1 | tr -d '"[:space:]') + want_nodes=$(printf '%s' "$pushed" | cut -d' ' -f1) + # Dependency :Package nodes are neutral (pkg: purls, shared across apps), so they are + # not under the prefix. They are part of the push count and checked separately. + db_pkgs=$(shell "MATCH (n:Package) WHERE n.id STARTS WITH 'pkg:' RETURN count(n) AS c" | tail -1 | tr -d '"[:space:]') + echo " info: $db_pkgs :Package nodes in the database" + # Pushed nodes are app-scoped (GoCanNode under the prefix) plus neutral :Package nodes. + if [ -n "$db_nodes" ] && [ "$db_nodes" -gt 0 ] && [ "$((db_nodes + ${db_pkgs:-0}))" -ge "$want_nodes" ]; then + pass "database holds $db_nodes GoCanNode rows under $prefix and ${db_pkgs:-0} :Package nodes (pushed $want_nodes nodes)" + else + bad "database has ${db_nodes:-no} GoCanNode rows under $prefix plus ${db_pkgs:-0} :Package; expected at least $want_nodes" + fi + labels=$(shell "MATCH (n:GoCallable) WHERE n.id STARTS WITH '$prefix' RETURN count(n) AS c" | tail -1 | tr -d '"[:space:]') + echo " info: $labels GoCallable nodes under $prefix" + fi +fi + +fi + +echo +if [ "$fail" -eq 0 ]; then + if [ "$JSON_ONLY" = "1" ]; then + echo "E2E OK (json-only): $APP_NAME -> $OUT/analysis.json" + else + echo "E2E OK: $APP_NAME -> $OUT/analysis.json -> Neo4j at $URI" + fi +else + echo "E2E FAILED for $APP_NAME (log: $LOG)" >&2 +fi +exit "$fail"