From 75e0411ac60a36bee310bfd43514601680016ad1 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Thu, 19 Nov 2026 21:35:55 -0500 Subject: [PATCH 01/11] docs(design): repository-artifact and dependency datamodel for schema v2 Spec and CLAUDE.md schema decisions from the design loop: application.artifacts (all non-.go files, metadata-first, --artifact-text opt-in), can:///artifact ids, version-less pkg:golang package ids, go.mod requires as dependencies with go.sum locks, L1 at every level. Config keys deferred. Co-Authored-By: Claude Haiku 5.5 --- CLAUDE.md | 11 +++ docs/design/specs/v2-artifacts.md | 131 ++++++++++++++++++++++++++++++ 2 files changed, 142 insertions(+) create mode 100644 docs/design/specs/v2-artifacts.md diff --git a/CLAUDE.md b/CLAUDE.md index f4425d4..9091930 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -129,6 +129,17 @@ Reuses Group A ids/spans and the L3 vocabulary unchanged; every L4 fact is an ad - A Go-specific `select` cfg kind was not added (`select` is a `switch` with `switch_case` edges). +### Repository-artifact decisions (spec `docs/design/specs/v2-artifacts.md`, branch `feat/artifacts`) +Made with the user in the 2026-11-19 design loop, anchored on `codeanalyzer-python`'s `PyArtifact` / `PyDependency`. +- **Placement:** `application.artifacts{: artifact}` (map); `application.dependencies[]` (list). +- **Scope:** all non-`.go` project files [user, 2026-11-19]. Binaries are metadata only (`source:""`) [user]. +- **Artifact id:** `can:///artifact/`, app-scoped (Python parity) [user]. The Go projector's `:Artifact` flips to `CanNode=true`. +- **Package id:** version-less `pkg:golang/` [user]; version lives on the dependency edge. +- **Source text:** only with `--artifact-text`, default off [user]. `config_keys` deferred [user]. +- **Dependencies:** every `go.mod` require; `direct=false` on `// indirect`; `kind:"runtime"`; `go.sum` supplies `locked_version` only, `go.sum`-only entries excluded [user]. +- **`provides_imports`:** longest-prefix match against `go.mod` module paths [user]. +- **Level:** L1, every level, not gated [user]. + ## Neo4j-projection decisions The graph projection (`--emit neo4j`) is a **second projection of the same v2 envelope**, diff --git a/docs/design/specs/v2-artifacts.md b/docs/design/specs/v2-artifacts.md new file mode 100644 index 0000000..b06bfe8 --- /dev/null +++ b/docs/design/specs/v2-artifacts.md @@ -0,0 +1,131 @@ +# Spec: repository artifacts and dependencies (codeanalyzer-go) + +> **Cross-repo spec.** Canonical home is `codellm-devkit/.github → docs/design/specs/`, +> beside the coordinating parent issue. Staged locally on `feat/artifacts` until that repo +> is available (same precedent as `v2-l1-emission.md`). This is the design transcript of the +> datamodel loop, linked from the tracking record, not pasted into it. + +## Contract-Impact Triage + +**Does this change schema v2 output?** Yes. It adds two application-scoped fields +(`artifacts`, `dependencies`), new id shapes for artifact and package nodes, one new CLI flag +(`--artifact-text`), and populates the already-declared Neo4j neutral layer (`:Artifact`, +`:Package`, `HAS_ARTIFACT`, `DECLARES_DEPENDENCY`, `LOCKS`). + +| Change type | Analyzers | SDKs | Docs | +| --- | --- | --- | --- | +| Schema v2 evolution (additive) | **codeanalyzer-go** (emit + projector); codeanalyzer-python: none (parity adopted, no id change) | **python-sdk** (Go backend reconstruction of `:Artifact` / `:Package`) | `CLAUDE.md` § Schema decisions, this spec | + +Anchor: `codeanalyzer-python` already emits this layer (`PyArtifact`, `PyDependency`, +`PyConfigKey`, `application.artifacts`, `application.dependencies`, +`codeanalyzer/schema/py_schema.py`), and `python-sdk/cldk/analysis/commons/artifacts.py` +rebuilds it from Neo4j, with the neutral labels shared across analyzers. + +## Problem + +The v2 schema and projector declare the neutral repository-artifact layer, but the Go +analyzer never populates it. `go.mod` is read only for `go mod download`. Against the Go +analysis of `microservices-demo/src/frontend`, `MATCH (n:Artifact)` returns 0 rows. + +## Design-loop decisions (the transcript) + +Each decision was made with the user, one question per decision, in the order below. + +### Application placement +- **`application.artifacts`** is a **map keyed by repo-relative path**, as Python does. Stays + reachable from `application` by containment (keystone rule). +- **`application.dependencies`** is a **list** (Python parity). + +### Artifact scope and fields +- **Scope: all non-`.go` project files** (user decision, broader than the recommended manifests-only + scope). `.go` files are modules, not artifacts. Directories `.git/` and `vendor/` are excluded + (implementation default, see Open items). +- **Artifact fields** (Python `PyArtifact` parity): + | field | type | notes | + | --- | --- | --- | + | `id` | string | `can:///artifact/` | + | `kind` | string | `"artifact"` | + | `path` | string | repo-relative POSIX path (also the map key) | + | `format` | string | `toml\|yaml\|json\|ini\|properties\|requirements\|dockerfile\|text\|binary`, plus Go's `gomod` / `gosum` (Open items) | + | `roles` | string[] | `dependency-manifest` for `go.mod`, `lockfile` for `go.sum`; others only when recognized | + | `size_bytes` | int | always | + | `sha256` | string | full file hash, always | + | `source` | string | whole file text **only** with `--artifact-text`; otherwise `""` | + | `extraction` | string | `none` \| `partial` \| `full`; `full` when `source` is set, else `none` | +- **Binary files**: metadata only. `format:"binary"`, `size_bytes` and `sha256` set, + `source:""` (user decision). +- **`--artifact-text`** (default **off**): when on, `source` carries the whole file text + (user decision). Off keeps default output to metadata. +- **`config_keys`** is **deferred** (user decision): not emitted in this change. No + `:ConfigKey` rows are written. + +### Artifact id +- **`can:///artifact/`** (Python parity, user decision). App-scoped, so the + projector's `can:////` prefix scoping covers artifact nodes. This changes the + Go projector's `:Artifact` flag to `CanNode=true`; the current Go schema comment ("neutral, + not a can:// id") is superseded. + +### Package and dependencies +- **Package id: version-less `pkg:golang/`** (user decision), for example + `pkg:golang/github.com/gorilla/mux`. Mirrors Python's `pkg:pypi/`. The version lives + on the edge and the record, so every dependent app converges on one `:Package`. +- **Dependency records come from every `go.mod` `require`** (user decision). Direct and + indirect entries alike. + | field | source | + | --- | --- | + | `name` | module path | + | `ecosystem` | `"golang"` | + | `spec` | version in `go.mod` (for example `v1.8.0`) | + | `kind` | `"runtime"` for every entry (`go.mod` has no dev/optional split) | + | `direct` | `false` when the line carries `// indirect`, else `true` | + | `locked_version` | version pinned in `go.sum` for that module, when present | + | `declared_in` | the `go.mod` artifact id | + | `provides_imports` | import paths in the project that resolve to this module (see below) | + | `prov` | `["declared"]`, plus `"lockfile"` when `go.sum` supplies `locked_version` | +- **`go.sum`-only entries are not dependencies** (user decision, the non-recommended option + would have included them; the recommended default is not to, because `go.sum` retains + entries for pruned modules). `go.sum` contributes only `locked_version` and `prov`. +- **`provides_imports`**: populated by longest-prefix match of each project import path + against the `go.mod` module paths (user decision, Python parity). Import paths that match + no requirement are **not** recorded in this change (see Open items). + +### Analysis level +- **L1, every level** (user decision). Parse-only facts, identical at `-a 1` through `-a 4`. + Not gated. + +### Neo4j projection (the neutral layer, unchanged labels) +- `:Artifact {id, path, format}` (keyed by `id`, `CanNode=true`), under `HAS_ARTIFACT` + from `:GoApplication`. +- `:Package {id, ecosystem, name}` keyed by `id`. +- `DECLARES_DEPENDENCY` `(:Artifact)→(:Package)`, `_k = kind` (`runtime`), carrying `spec`, + `direct`, `prov` as properties. +- `LOCKS` `(:Artifact)→(:Package)` from `go.sum`, carrying `locked_version`. +- No `:ConfigKey` rows and no `DEFINES_CONFIG` rows in this change. + +## Open items (flagged for review, not asked) + +These are defaults I chose while writing the spec. Confirm them during review: + +1. **`format` values for go.mod and go.sum.** I used `gomod` and `gosum`, because the Python + enum has no Go case. The alternative is `text`. +2. **Excluded directories.** `.git/` and `vendor/` are skipped. `vendor/` is third-party + source, not project configuration. +3. **Imports with no matching requirement.** Not recorded. They are `unresolved_imports` in + Python. That field is deferred with the repository-provenance field. +4. **`locked_version` when `go.sum` has several lines for one module** (the `h1:` and + `/go.mod` hashes). Take the version common to both lines; error if they disagree. + +## Consumers + +- **codeanalyzer-go**: emits the fields; the projector writes the rows above. Tests: a fixture + with `go.mod` and `go.sum`, a projector test for the new rows, and a `--artifact-text` + on/off test. +- **python-sdk**: the Go backend reads `:Artifact` / `:Package` through the shared + `cldk/analysis/commons/artifacts.py` reconstructors. The Go backend needs its artifact getters + added. Scope of that rung is decided at decomposition. + +## Out of scope (deferred) + +- Config keys and `DEFINES_CONFIG` (user decision). +- `unresolved_imports`, repository provenance (`repository`), `config_uses`. +- Codeanalyzer-python: no change. From cf2cab1ab626901c59070d0c5421f78f50c86d6c Mon Sep 17 00:00:00 2001 From: lambertpw Date: Thu, 19 Nov 2026 22:08:12 -0500 Subject: [PATCH 02/11] feat(artifacts): emit repository artifacts and go.mod dependencies at L1 Adds application.artifacts (every non-.go project file: format, roles, size, sha256; source only with --artifact-text) and application.dependencies (go.mod requires with direct, locked_version from go.sum, provides_imports by longest prefix). The Neo4j projector writes :Artifact (app-scoped), :Package (pkg:golang/), HAS_ARTIFACT, DECLARES_DEPENDENCY and LOCKS. Config keys are deferred. Refs codellm-devkit/codeanalyzer-go#45 Co-Authored-By: Claude Haiku 5.5 --- README.md | 1 + cmd/codeanalyzer/main.go | 4 + docs/design/specs/v2-artifacts.md | 24 ++ internal/core/analyzer.go | 20 ++ internal/core/repository_test.go | 109 +++++++ internal/neo4j/project.go | 44 +++ internal/neo4j/repository_test.go | 97 ++++++ internal/neo4j/schema.go | 13 +- internal/options/options.go | 3 + internal/repository/repository.go | 279 ++++++++++++++++++ internal/repository/repository_test.go | 235 +++++++++++++++ internal/schema/schema.go | 38 +++ internal/schema/v2/schema.go | 34 +++ internal/syntactic_analysis/v2emit/emit.go | 1 + .../syntactic_analysis/v2emit/repository.go | 57 ++++ 15 files changed, 954 insertions(+), 5 deletions(-) create mode 100644 internal/core/repository_test.go create mode 100644 internal/neo4j/repository_test.go create mode 100644 internal/repository/repository.go create mode 100644 internal/repository/repository_test.go create mode 100644 internal/syntactic_analysis/v2emit/repository.go diff --git a/README.md b/README.md index 0d61c82..f063d6d 100644 --- a/README.md +++ b/README.md @@ -173,6 +173,7 @@ Flags: -f, --format string Output format: json|msgpack (default "json") --graph-field-depth int Field-path depth bound for L3 dataflow nodes (k limit; applies at -a 3 and -a 4) (default 3) -h, --help help for cango + --artifact-text Capture each text artifact's whole source in analysis.json (default: metadata only) -i, --input string Project root to analyze (required) -j, --jobs int Worker parallelism (default: CPU cores) --library-models string JSON file of library stand-in flows overriding the all-to-all default (-a 4 and --emit neo4j only) diff --git a/cmd/codeanalyzer/main.go b/cmd/codeanalyzer/main.go index d2b5379..3cceee3 100644 --- a/cmd/codeanalyzer/main.go +++ b/cmd/codeanalyzer/main.go @@ -46,6 +46,7 @@ func rootCmd() *cobra.Command { libraryModels string targetFiles []string skipTests bool + artifactText bool eager bool cacheDir string jobs int @@ -172,6 +173,7 @@ via CLDK(language="go").analysis(project_path=...).`, AnalyzerVersion: version, TargetFiles: targetFiles, SkipTests: skipTests, + ArtifactText: artifactText, Eager: eager, CacheDir: cacheDir, Jobs: jobs, @@ -238,6 +240,8 @@ via CLDK(language="go").analysis(project_path=...).`, f.StringSliceVarP(&targetFiles, "target-files", "t", nil, "Restrict analysis to specific files (incremental mode)") f.BoolVar(&skipTests, "skip-tests", true, "Skip *_test.go files") + f.BoolVar(&artifactText, "artifact-text", false, + "Capture each text artifact's whole source in analysis.json (default: metadata only)") f.BoolVar(&eager, "eager", false, "Force clean rebuild (ignore cache)") f.StringVarP(&cacheDir, "cache-dir", "c", "", "Cache directory (default: ~/.cldk/go-cache)") f.IntVarP(&jobs, "jobs", "j", 0, "Worker parallelism (default: CPU cores)") diff --git a/docs/design/specs/v2-artifacts.md b/docs/design/specs/v2-artifacts.md index b06bfe8..6d8ed76 100644 --- a/docs/design/specs/v2-artifacts.md +++ b/docs/design/specs/v2-artifacts.md @@ -115,6 +115,30 @@ These are defaults I chose while writing the spec. Confirm them during review: 4. **`locked_version` when `go.sum` has several lines for one module** (the `h1:` and `/go.mod` hashes). Take the version common to both lines; error if they disagree. +## Resolution (implementation, branch `feat/artifacts`) + +The open items were resolved with the defaults below. Two are deviations from the wording +above, marked **(deviation)**: + +1. **`format`**: `gomod` for `go.mod`, `gosum` for `go.sum`, adopted. +2. **Excluded directories**: `.git/` and `vendor/` are skipped at any depth. The tool's own + outputs are also skipped when they sit in the project: `analysis.json` and `graph.cypher` + in the `-o` directory, and the `-c` cache subtree. Without this, a run that writes into the + project would capture the previous run's output. +3. **Unrecorded imports**: an import matching no `go.mod` requirement is not recorded. +4. **Locked version (deviation)**: a requirement is locked only when go.sum has **both** the + `h1:` and the `/go.mod` line for the exact required version. A version with only one line + is left unlocked (`locked_version` absent, `prov` without `lockfile`), with no error. A + disagreement cannot arise, since the lookup is keyed by the required version. +5. **Artifact id root (deviation)**: the code's grammar is `can:////…`, the + same root every other node uses, so an artifact id is + `can:////artifact/`. The `can:///artifact/` form in the + decisions above is shorthand for that root. + +Also decided at implementation: `source` is omitted from JSON when empty (`omitempty`), not +written as `""`; only the root `go.mod` contributes dependencies (nested modules are artifacts +only); `dependencies` is absent when there is no root `go.mod`. + ## Consumers - **codeanalyzer-go**: emits the fields; the projector writes the rows above. Tests: a fixture diff --git a/internal/core/analyzer.go b/internal/core/analyzer.go index a82ac2c..b9dbf34 100644 --- a/internal/core/analyzer.go +++ b/internal/core/analyzer.go @@ -32,6 +32,7 @@ import ( "github.com/codellm-devkit/codeanalyzer-go/internal/analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" "github.com/codellm-devkit/codeanalyzer-go/internal/options" + "github.com/codellm-devkit/codeanalyzer-go/internal/repository" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" "github.com/codellm-devkit/codeanalyzer-go/internal/sdg" "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" @@ -93,6 +94,25 @@ func New(opts options.AnalysisOptions) *Analyzer { // Analyze runs the full analysis pipeline and returns a GoApplication. func (a *Analyzer) Analyze() (*schema.GoApplication, error) { + app, err := a.analyze() + if err != nil { + return nil, err + } + // The repository layer reads files outside the Go source, so it is computed + // on every run, after the (possibly cached) application is ready. + facts, err := repository.Facts(a.opts.InputPath, app, repository.Options{ + CaptureText: a.opts.ArtifactText, + OutputDir: a.opts.OutputDir, + CacheDir: a.opts.CacheDir, + }) + if err != nil { + return nil, fmt.Errorf("repository layer: %w", err) + } + app.Repository = facts + return app, nil +} + +func (a *Analyzer) analyze() (*schema.GoApplication, error) { if _, err := resolveIdentity(a.opts); err != nil { return nil, err } diff --git a/internal/core/repository_test.go b/internal/core/repository_test.go new file mode 100644 index 0000000..1f72580 --- /dev/null +++ b/internal/core/repository_test.go @@ -0,0 +1,109 @@ +package core_test + +import ( + "encoding/json" + "os" + "path/filepath" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/core" + "github.com/codellm-devkit/codeanalyzer-go/internal/options" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis/v2emit" +) + +// repoProject is a Go module with no requires (so no network materialization), +// one .go file, and two non-.go files. +func repoProject(t *testing.T) string { + t.Helper() + dir := t.TempDir() + files := map[string]string{ + "go.mod": "module example.com/repoapp\n\ngo 1.25.0\n", + "go.sum": "", + "main.go": "package main\n\nfunc main() {}\n", + "README.md": "# repo app\n", + } + for rel, content := range files { + if err := os.WriteFile(filepath.Join(dir, rel), []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + return dir +} + +// emitJSON runs Analyze at level 1 and returns the v2 analysis.json bytes. +func emitJSON(t *testing.T, dir string, artifactText bool) []byte { + t.Helper() + opts := options.AnalysisOptions{ + InputPath: dir, AppName: "repoapp", Level: options.LevelSymbolTable, + SkipTests: true, CacheDir: t.TempDir(), OutputDir: t.TempDir(), ArtifactText: artifactText, + } + app, err := core.New(opts).Analyze() + if err != nil { + t.Fatal(err) + } + identity, err := v2.ResolveIdentity("repoapp", nil) + if err != nil { + t.Fatal(err) + } + payload := v2emit.Emit(app, identity, dir, int(opts.Level), "test", 0, nil, nil) + out, err := json.Marshal(payload) + if err != nil { + t.Fatal(err) + } + return out +} + +func TestAnalyze_ArtifactsDefaultMetadataOnly(t *testing.T) { + dir := repoProject(t) + var doc struct { + Application struct { + Artifacts map[string]v2.Artifact `json:"artifacts"` + } `json:"application"` + } + if err := json.Unmarshal(emitJSON(t, dir, false), &doc); err != nil { + t.Fatal(err) + } + art, ok := doc.Application.Artifacts["README.md"] + if !ok { + t.Fatalf("README.md missing from artifacts: %v", doc.Application.Artifacts) + } + if art.ID != "can://repoapp/repoapp/artifact/README.md" { + t.Errorf("artifact id = %q", art.ID) + } + if art.Format != "text" || art.Extraction != "none" || art.Source != "" { + t.Errorf("default artifact should be metadata only: %+v", art) + } + if _, ok := doc.Application.Artifacts["main.go"]; ok { + t.Errorf("a .go file became an artifact") + } +} + +func TestAnalyze_ArtifactTextCapturesSource(t *testing.T) { + dir := repoProject(t) + var doc struct { + Application struct { + Artifacts map[string]v2.Artifact `json:"artifacts"` + } `json:"application"` + } + if err := json.Unmarshal(emitJSON(t, dir, true), &doc); err != nil { + t.Fatal(err) + } + art := doc.Application.Artifacts["README.md"] + if art.Source != "# repo app\n" || art.Extraction != "full" { + t.Errorf("--artifact-text: source/extraction = %q/%q", art.Source, art.Extraction) + } + mod := doc.Application.Artifacts["go.mod"] + if mod.Format != "gomod" || len(mod.Roles) != 1 || mod.Roles[0] != "dependency-manifest" { + t.Errorf("go.mod artifact = %+v", mod) + } +} + +func TestAnalyze_ArtifactsReproducibleAcrossRuns(t *testing.T) { + dir := repoProject(t) + a := emitJSON(t, dir, false) + b := emitJSON(t, dir, false) + if string(a) != string(b) { + t.Errorf("two runs over the same project differ") + } +} diff --git a/internal/neo4j/project.go b/internal/neo4j/project.go index 78fde41..04111d1 100644 --- a/internal/neo4j/project.go +++ b/internal/neo4j/project.go @@ -2,6 +2,7 @@ package neo4j import ( "encoding/json" + "sort" "strings" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" @@ -33,6 +34,7 @@ func Project(a *v2.Analysis) GraphRows { declared: collectDeclaredIDs(a), } p.projectApplication() + p.projectRepository() for _, mod := range a.Application.SymbolTable { p.projectModule(mod) } @@ -84,6 +86,48 @@ func (p *projector) appRef() NodeRef { return NodeRef{Label: LabelApplication, KeyProp: "id", Value: p.analysis.Application.ID} } +// ─── repository layer (spec v2-artifacts) ──────────────────────────────────── + +// projectRepository writes the artifact and dependency rows. An artifact is an +// app-scoped node (its id is a can:// id) hung off the application; a dependency +// is a neutral :Package keyed by its version-less purl, declared by the go.mod +// artifact and, when go.sum locks it, locked by the go.sum artifact. +func (p *projector) projectRepository() { + arts := p.analysis.Application.Artifacts + paths := make([]string, 0, len(arts)) + for path := range arts { + paths = append(paths, path) + } + sort.Strings(paths) + for _, path := range paths { + art := arts[path] + ref := p.b.Node( + []string{LabelArtifact, MarkerLabel}, + "id", art.ID, + Props{"path": art.Path, "format": art.Format}, + "", // an artifact belongs to no single module + ) + p.b.Edge(RelHasArtifact, p.appRef(), ref, nil, "") + } + + goSum, hasGoSum := arts["go.sum"] + for _, dep := range p.analysis.Application.Dependencies { + pkgRef := p.b.Node( + []string{LabelPackage}, + "id", "pkg:"+dep.Ecosystem+"/"+dep.Name, + Props{"ecosystem": dep.Ecosystem, "name": dep.Name}, + "", + ) + declaredBy := NodeRef{Label: LabelArtifact, KeyProp: "id", Value: dep.DeclaredIn} + p.b.Edge(RelDeclaresDependency, declaredBy, pkgRef, + Props{"spec": dep.Spec, "direct": dep.Direct, "prov": dep.Prov}, dep.Kind) + if dep.LockedVersion != "" && hasGoSum { + p.b.Edge(RelLocks, NodeRef{Label: LabelArtifact, KeyProp: "id", Value: goSum.ID}, pkgRef, + Props{"locked_version": dep.LockedVersion}, "") + } + } +} + // ─── module ────────────────────────────────────────────────────────────────── func (p *projector) projectModule(m v2.Module) { diff --git a/internal/neo4j/repository_test.go b/internal/neo4j/repository_test.go new file mode 100644 index 0000000..b2554f6 --- /dev/null +++ b/internal/neo4j/repository_test.go @@ -0,0 +1,97 @@ +package neo4j + +import ( + "testing" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// repoAnalysis is a minimal L1 analysis carrying only the repository layer: +// go.mod and go.sum artifacts, and two dependencies (one locked, one not). +func repoAnalysis() *v2.Analysis { + const app = "can://frontend/frontend" + goMod := app + "/artifact/go.mod" + goSum := app + "/artifact/go.sum" + return &v2.Analysis{ + IdentityVersion: 2, SchemaVersion: v2.SchemaVersion, Language: v2.Language, MaxLevel: 1, + Application: v2.Application{ + IdentityVersion: 2, ID: app, Kind: "application", + Artifacts: map[string]v2.Artifact{ + "go.mod": {ID: goMod, Kind: "artifact", Path: "go.mod", Format: "gomod", Roles: []string{"dependency-manifest"}}, + "go.sum": {ID: goSum, Kind: "artifact", Path: "go.sum", Format: "gosum", Roles: []string{"lockfile"}}, + }, + Dependencies: []v2.Dependency{ + {Name: "github.com/gorilla/mux", Ecosystem: "golang", Spec: "v1.8.0", Kind: "runtime", Direct: true, + LockedVersion: "v1.8.0", DeclaredIn: goMod, Prov: []string{"declared", "lockfile"}}, + {Name: "golang.org/x/text", Ecosystem: "golang", Spec: "v0.14.0", Kind: "runtime", Direct: false, + DeclaredIn: goMod, Prov: []string{"declared"}}, + }, + }, + } +} + +func TestProject_RepositoryLayerRows(t *testing.T) { + rows := Project(repoAnalysis()) + + nodes := map[string]NodeRow{} + for _, n := range rows.Nodes { + nodes[n.Value] = n + } + + goMod := nodes["can://frontend/frontend/artifact/go.mod"] + if !hasLabel(goMod, LabelArtifact) || !hasLabel(goMod, MarkerLabel) { + t.Errorf("artifact node labels = %v, want Artifact + %s (app-scoped)", goMod.Labels, MarkerLabel) + } + mux := nodes["pkg:golang/github.com/gorilla/mux"] + if !hasLabel(mux, LabelPackage) || hasLabel(mux, MarkerLabel) { + t.Errorf("package node labels = %v, want Package only (neutral, no marker)", mux.Labels) + } + + edges := map[string][]EdgeRow{} + for _, e := range rows.Edges { + edges[e.Type] = append(edges[e.Type], e) + } + if len(edges[RelHasArtifact]) != 2 { + t.Errorf("HAS_ARTIFACT edges = %d, want 2 (go.mod, go.sum)", len(edges[RelHasArtifact])) + } + + decl := edges[RelDeclaresDependency] + if len(decl) != 2 { + t.Fatalf("DECLARES_DEPENDENCY edges = %d, want 2", len(decl)) + } + for _, e := range decl { + if e.From.Value != "can://frontend/frontend/artifact/go.mod" || e.To.Label != LabelPackage { + t.Errorf("DECLARES_DEPENDENCY wired %s -> %s:%s, want go.mod -> Package", e.From.Value, e.To.Label, e.To.Value) + } + if e.Key != "runtime" { + t.Errorf("DECLARES_DEPENDENCY _k = %q, want runtime", e.Key) + } + } + + locks := edges[RelLocks] + if len(locks) != 1 { + t.Fatalf("LOCKS edges = %d, want 1 (only the locked dependency)", len(locks)) + } + if locks[0].From.Value != "can://frontend/frontend/artifact/go.sum" || + locks[0].To.Value != "pkg:golang/github.com/gorilla/mux" || + locks[0].Props["locked_version"] != "v1.8.0" { + t.Errorf("LOCKS = %+v, want go.sum -> mux with locked_version v1.8.0", locks[0]) + } +} + +func TestProject_NoRepositoryLayerEmitsNoRows(t *testing.T) { + a := repoAnalysis() + a.Application.Artifacts, a.Application.Dependencies = nil, nil + rows := Project(a) + for _, n := range rows.Nodes { + if hasLabel(n, LabelArtifact) || hasLabel(n, LabelPackage) { + t.Errorf("repository node %v emitted without facts", n.Labels) + } + } + for _, e := range rows.Edges { + switch e.Type { + case RelHasArtifact, RelDeclaresDependency, RelLocks: + t.Errorf("repository edge %s emitted without facts", e.Type) + } + } +} diff --git a/internal/neo4j/schema.go b/internal/neo4j/schema.go index 18853b4..274d39d 100644 --- a/internal/neo4j/schema.go +++ b/internal/neo4j/schema.go @@ -227,12 +227,15 @@ var Nodes = []NodeSpec{ Props: []string{"name", "path", "signature", "module"}, }, - // Neutral layer. Keyed by own id (not a can:// id), so NO marker label and - // outside prefix scoping — a sibling analyzer converges on the same node. + // Repository layer. An artifact id is can:////artifact/ + // (spec v2-artifacts, Python parity), so it is app-scoped: it carries the + // :GoCanNode marker and falls under the prefix scope on wipe. { - Label: LabelArtifact, MergeLabel: LabelArtifact, Key: "id", CanNode: false, + Label: LabelArtifact, MergeLabel: LabelArtifact, Key: "id", CanNode: true, Props: []string{"path", "format"}, }, + // Neutral layer. Keyed by its pkg: purl (not a can:// id), so NO marker label + // and outside prefix scoping — a sibling analyzer converges on the same node. { Label: LabelPackage, MergeLabel: LabelPackage, Key: "id", CanNode: false, Props: []string{"ecosystem", "name"}, @@ -265,8 +268,8 @@ var Rels = []RelSpec{ // Repository layer. {Type: RelHasArtifact, From: []string{LabelApplication}, To: []string{LabelArtifact}}, - {Type: RelDeclaresDependency, From: []string{LabelArtifact}, To: []string{LabelPackage}, Discriminant: "kind"}, - {Type: RelLocks, From: []string{LabelArtifact}, To: []string{LabelPackage}}, + {Type: RelDeclaresDependency, From: []string{LabelArtifact}, To: []string{LabelPackage}, Props: []string{"spec", "direct", "prov"}, Discriminant: "kind"}, + {Type: RelLocks, From: []string{LabelArtifact}, To: []string{LabelPackage}, Props: []string{"locked_version"}}, {Type: RelDefinesConfig, From: []string{LabelArtifact}, To: []string{LabelConfigKey}}, {Type: RelProvides, From: []string{LabelPackage}, To: []string{LabelExternal}}, {Type: RelUnresolvedImport, From: []string{LabelApplication}, To: []string{LabelExternal}}, diff --git a/internal/options/options.go b/internal/options/options.go index 13f0af9..ebeb300 100644 --- a/internal/options/options.go +++ b/internal/options/options.go @@ -57,6 +57,9 @@ type AnalysisOptions struct { TargetFiles []string // SkipTests skips test files (files ending in _test.go). SkipTests bool + // ArtifactText captures each text artifact's whole source (--artifact-text). + // Off by default: artifacts carry metadata only. + ArtifactText bool // Eager forces a clean rebuild ignoring any cache. Eager bool // CacheDir is where per-file caches and intermediate data are stored. diff --git a/internal/repository/repository.go b/internal/repository/repository.go new file mode 100644 index 0000000..45906d1 --- /dev/null +++ b/internal/repository/repository.go @@ -0,0 +1,279 @@ +// Package repository collects the repository-artifact layer (spec +// docs/design/specs/v2-artifacts.md): every non-.go project file as an artifact, +// and the go.mod requires as dependencies, with go.sum supplying locked versions. +// +// It reads files outside the Go source the symbol table sees, so it runs on every +// analysis (never from the cache) and is independent of the analysis level. +package repository + +import ( + "bytes" + "crypto/sha256" + "encoding/hex" + "fmt" + "io/fs" + "os" + "path/filepath" + "sort" + "strings" + "unicode/utf8" + + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" + "golang.org/x/mod/modfile" +) + +// Ecosystem is the purl type used for Go modules (pkg:golang/). +const Ecosystem = "golang" + +// Options controls discovery. +type Options struct { + // CaptureText puts each text artifact's whole source into Source (--artifact-text). + CaptureText bool + // OutputDir and CacheDir are the tool's own output locations. When either lies + // inside the project, its files are not artifacts: a run that writes analysis.json + // into the project must not capture the previous run's output. + OutputDir string + CacheDir string +} + +// skipDirNames are directories never walked: VCS metadata and vendored third-party code. +var skipDirNames = map[string]bool{".git": true, "vendor": true} + +// outputFileNames are the files this tool writes into OutputDir. +var outputFileNames = map[string]bool{"analysis.json": true, "graph.cypher": true} + +// Discover returns every non-.go regular file under root as an artifact, keyed by +// its repo-relative POSIX path. Files are visited in lexical order. +func Discover(root string, opts Options) (map[string]schema.GoArtifact, error) { + absRoot, err := filepath.Abs(root) + if err != nil { + return nil, err + } + outDir := absOrEmpty(opts.OutputDir) + cacheDir := absOrEmpty(opts.CacheDir) + out := map[string]schema.GoArtifact{} + err = filepath.WalkDir(absRoot, func(p string, d fs.DirEntry, werr error) error { + if werr != nil { + return werr + } + if d.IsDir() { + if p != absRoot && (skipDirNames[d.Name()] || (cacheDir != "" && p == cacheDir)) { + return filepath.SkipDir + } + return nil + } + if !d.Type().IsRegular() || strings.HasSuffix(d.Name(), ".go") { + return nil + } + if outDir != "" && filepath.Dir(p) == outDir && outputFileNames[d.Name()] { + return nil + } + rel, err := filepath.Rel(absRoot, p) + if err != nil { + return err + } + art, err := readArtifact(p, filepath.ToSlash(rel), opts.CaptureText) + if err != nil { + return err + } + out[art.Path] = art + return nil + }) + if err != nil { + return nil, fmt.Errorf("artifact discovery: %w", err) + } + return out, nil +} + +func absOrEmpty(p string) string { + if p == "" { + return "" + } + abs, err := filepath.Abs(p) + if err != nil { + return "" + } + return abs +} + +// readArtifact hashes the whole file and classifies it. Text source is kept only +// when captureText is set; binaries are never captured. +func readArtifact(abs, rel string, captureText bool) (schema.GoArtifact, error) { + data, err := os.ReadFile(abs) + if err != nil { + return schema.GoArtifact{}, err + } + sum := sha256.Sum256(data) + art := schema.GoArtifact{ + Path: rel, + Format: formatOf(rel, data), + SizeBytes: int64(len(data)), + SHA256: hex.EncodeToString(sum[:]), + Extraction: "none", + } + switch rel { + case "go.mod": + art.Roles = []string{"dependency-manifest"} + case "go.sum": + art.Roles = []string{"lockfile"} + } + if captureText && art.Format != "binary" { + art.Source = string(data) + art.Extraction = "full" + } + return art, nil +} + +// formatOf names the file's format. go.mod and go.sum have their own names; the +// rest follow the Python analyzer's enum. Binary is a NUL byte or invalid UTF-8. +func formatOf(rel string, data []byte) string { + base := filepath.Base(rel) + switch { + case base == "go.mod": + return "gomod" + case base == "go.sum": + return "gosum" + case base == "Dockerfile" || strings.HasPrefix(base, "Dockerfile.") || strings.HasSuffix(base, ".dockerfile"): + return "dockerfile" + case strings.HasPrefix(base, "requirements") && strings.HasSuffix(base, ".txt"): + return "requirements" + } + if bytes.IndexByte(data, 0) >= 0 || !utf8.Valid(data) { + return "binary" + } + switch strings.ToLower(filepath.Ext(base)) { + case ".toml": + return "toml" + case ".yaml", ".yml": + return "yaml" + case ".json": + return "json" + case ".ini", ".cfg": + return "ini" + case ".properties": + return "properties" + } + return "text" +} + +// Facts builds the repository layer for a project: its artifacts, and the +// dependencies declared by the root go.mod. imports are the project's import +// paths (from the symbol table), used to fill provides_imports. +func Facts(root string, app *schema.GoApplication, opts Options) (*schema.RepositoryFacts, error) { + artifacts, err := Discover(root, opts) + if err != nil { + return nil, err + } + facts := &schema.RepositoryFacts{Artifacts: artifacts} + modPath := filepath.Join(root, "go.mod") + modData, err := os.ReadFile(modPath) + if err != nil { + // No root go.mod: artifacts only, no dependencies. + return facts, nil + } + f, err := modfile.Parse("go.mod", modData, nil) + if err != nil { + utils.Warn("repository: go.mod does not parse, dependencies skipped: %v", err) + return facts, nil + } + locked := parseGoSum(filepath.Join(root, "go.sum")) + imports := projectImports(app) + + deps := make([]schema.GoDependency, 0, len(f.Require)) + for _, r := range f.Require { + dep := schema.GoDependency{ + Name: r.Mod.Path, + Ecosystem: Ecosystem, + Spec: r.Mod.Version, + Kind: "runtime", + Direct: !r.Indirect, + DeclaredIn: "go.mod", + Prov: []string{"declared"}, + } + if locked[r.Mod.Path][r.Mod.Version] { + dep.LockedVersion = r.Mod.Version + dep.Prov = append(dep.Prov, "lockfile") + } + deps = append(deps, dep) + } + sort.Slice(deps, func(i, j int) bool { return deps[i].Name < deps[j].Name }) + assignImports(deps, imports) + facts.Dependencies = deps + return facts, nil +} + +// parseGoSum returns, per module path and version, whether go.sum carries both +// the module's h1 line and its /go.mod line. Only such a version is a locked +// version. A missing or unreadable go.sum locks nothing. +func parseGoSum(path string) map[string]map[string]bool { + type seen struct{ h1, gomod bool } + entries := map[string]map[string]*seen{} + data, err := os.ReadFile(path) + if err != nil { + return map[string]map[string]bool{} + } + for _, line := range strings.Split(string(data), "\n") { + fields := strings.Fields(line) + if len(fields) != 3 { + continue + } + mod, ver := fields[0], fields[1] + gomod := strings.HasSuffix(ver, "/go.mod") + ver = strings.TrimSuffix(ver, "/go.mod") + if entries[mod] == nil { + entries[mod] = map[string]*seen{} + } + if entries[mod][ver] == nil { + entries[mod][ver] = &seen{} + } + if gomod { + entries[mod][ver].gomod = true + } else { + entries[mod][ver].h1 = true + } + } + out := map[string]map[string]bool{} + for mod, vers := range entries { + out[mod] = map[string]bool{} + for ver, s := range vers { + out[mod][ver] = s.h1 && s.gomod + } + } + return out +} + +// projectImports is the sorted, deduplicated set of import paths the project's +// (non-test, when the symbol table was built with --skip-tests) Go files use. +func projectImports(app *schema.GoApplication) []string { + seen := map[string]bool{} + for _, f := range app.SymbolTable { + for _, imp := range f.Imports { + seen[imp.Module] = true + } + } + out := make([]string, 0, len(seen)) + for p := range seen { + out = append(out, p) + } + sort.Strings(out) + return out +} + +// assignImports gives each import to the dependency whose module path is its +// longest prefix (the path itself, or a path under it followed by "/"). An +// import matching no requirement is not recorded. +func assignImports(deps []schema.GoDependency, imports []string) { + for _, imp := range imports { + best := -1 + bestLen := -1 + for i, d := range deps { + if (imp == d.Name || strings.HasPrefix(imp, d.Name+"/")) && len(d.Name) > bestLen { + best, bestLen = i, len(d.Name) + } + } + if best >= 0 { + deps[best].ProvidesImports = append(deps[best].ProvidesImports, imp) + } + } +} diff --git a/internal/repository/repository_test.go b/internal/repository/repository_test.go new file mode 100644 index 0000000..0c048ef --- /dev/null +++ b/internal/repository/repository_test.go @@ -0,0 +1,235 @@ +package repository + +import ( + "os" + "path/filepath" + "reflect" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" +) + +// writeTree creates the files under root (path → content). A NUL in content +// makes a file binary. +func writeTree(t *testing.T, root string, files map[string]string) { + t.Helper() + for rel, content := range files { + p := filepath.Join(root, filepath.FromSlash(rel)) + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } +} + +// appWithImports is a symbol-table stub carrying the given import paths in one file. +func appWithImports(paths ...string) *schema.GoApplication { + imps := make([]schema.GoImport, 0, len(paths)) + for _, p := range paths { + imps = append(imps, schema.GoImport{Module: p}) + } + return &schema.GoApplication{SymbolTable: map[string]schema.GoFile{"main.go": {Imports: imps}}} +} + +const goMod = `module example.com/app + +go 1.25.0 + +require ( + cloud.google.com/go v0.1.0 + cloud.google.com/go/auth v0.2.0 + github.com/gorilla/mux v1.8.0 + golang.org/x/text v0.14.0 // indirect +) +` + +// go.sum: mux has both lines for its version (locked); auth too; cloud.google.com/go +// has only its h1 line (not locked); x/text has only its /go.mod line (not locked). +const goSum = `cloud.google.com/go v0.1.0 h1:AAA= +cloud.google.com/go/auth v0.2.0 h1:BBB= +cloud.google.com/go/auth v0.2.0/go.mod h1:CCC= +github.com/gorilla/mux v1.8.0 h1:DDD= +github.com/gorilla/mux v1.8.0/go.mod h1:EEE= +golang.org/x/text v0.14.0/go.mod h1:FFF= +` + +func TestFacts_DependenciesLocksAndImports(t *testing.T) { + root := t.TempDir() + writeTree(t, root, map[string]string{"go.mod": goMod, "go.sum": goSum, "main.go": "package main\n"}) + + facts, err := Facts(root, appWithImports( + "github.com/gorilla/mux", + "cloud.google.com/go/auth/credentials", // longest prefix: cloud.google.com/go/auth + "cloud.google.com/go/storage", // only cloud.google.com/go matches + "fmt", // stdlib: no requirement + ), Options{}) + if err != nil { + t.Fatal(err) + } + + want := []schema.GoDependency{ + {Name: "cloud.google.com/go", Ecosystem: "golang", Spec: "v0.1.0", Kind: "runtime", Direct: true, + DeclaredIn: "go.mod", ProvidesImports: []string{"cloud.google.com/go/storage"}, Prov: []string{"declared"}}, + {Name: "cloud.google.com/go/auth", Ecosystem: "golang", Spec: "v0.2.0", Kind: "runtime", Direct: true, + LockedVersion: "v0.2.0", DeclaredIn: "go.mod", ProvidesImports: []string{"cloud.google.com/go/auth/credentials"}, + Prov: []string{"declared", "lockfile"}}, + {Name: "github.com/gorilla/mux", Ecosystem: "golang", Spec: "v1.8.0", Kind: "runtime", Direct: true, + LockedVersion: "v1.8.0", DeclaredIn: "go.mod", ProvidesImports: []string{"github.com/gorilla/mux"}, + Prov: []string{"declared", "lockfile"}}, + {Name: "golang.org/x/text", Ecosystem: "golang", Spec: "v0.14.0", Kind: "runtime", Direct: false, + DeclaredIn: "go.mod", Prov: []string{"declared"}}, + } + if !reflect.DeepEqual(facts.Dependencies, want) { + t.Errorf("dependencies:\n got %+v\n want %+v", facts.Dependencies, want) + } +} + +func TestFacts_ArtifactsClassifiedAndSkipped(t *testing.T) { + root := t.TempDir() + writeTree(t, root, map[string]string{ + "go.mod": goMod, + "go.sum": goSum, + "main.go": "package main\n", + "main_test.go": "package main\n", + "Dockerfile": "FROM scratch\n", + "config/app.yaml": "a: 1\n", + "config/settings.env": "KEY=value\n", + "blob.dat": "ab\x00cd", + ".git/HEAD": "ref: refs/heads/main\n", + "vendor/x/y.txt": "vendored\n", + }) + + facts, err := Facts(root, appWithImports(), Options{}) + if err != nil { + t.Fatal(err) + } + got := map[string]schema.GoArtifact{} + for p, a := range facts.Artifacts { + got[p] = a + } + wantFormats := map[string]string{ + "go.mod": "gomod", + "go.sum": "gosum", + "Dockerfile": "dockerfile", + "config/app.yaml": "yaml", + "config/settings.env": "text", + "blob.dat": "binary", + } + if len(got) != len(wantFormats) { + t.Errorf("artifacts = %v, want exactly %v", keys(got), keys(wantFormats)) + } + for p, f := range wantFormats { + a, ok := got[p] + if !ok { + t.Errorf("missing artifact %s", p) + continue + } + if a.Format != f { + t.Errorf("%s format = %q, want %q", p, a.Format, f) + } + if a.Extraction != "none" || a.Source != "" { + t.Errorf("%s captured source without --artifact-text: %+v", p, a) + } + if len(a.SHA256) != 64 { + t.Errorf("%s sha256 = %q, want 64 hex chars", p, a.SHA256) + } + } + if r := got["go.mod"].Roles; !reflect.DeepEqual(r, []string{"dependency-manifest"}) { + t.Errorf("go.mod roles = %v", r) + } + if r := got["go.sum"].Roles; !reflect.DeepEqual(r, []string{"lockfile"}) { + t.Errorf("go.sum roles = %v", r) + } +} + +func TestDiscover_CaptureTextOnlyForText(t *testing.T) { + root := t.TempDir() + writeTree(t, root, map[string]string{"go.mod": goMod, "blob.dat": "ab\x00cd"}) + + on, err := Discover(root, Options{CaptureText: true}) + if err != nil { + t.Fatal(err) + } + if on["go.mod"].Source != goMod || on["go.mod"].Extraction != "full" { + t.Errorf("go.mod with --artifact-text: source/extraction = %q/%q", on["go.mod"].Source, on["go.mod"].Extraction) + } + if on["blob.dat"].Source != "" || on["blob.dat"].Extraction != "none" { + t.Errorf("binary captured source: %+v", on["blob.dat"]) + } +} + +func TestDiscover_ExcludesOwnOutputsAndCache(t *testing.T) { + root := t.TempDir() + cache := filepath.Join(root, "cache") + writeTree(t, root, map[string]string{ + "README.md": "# app\n", + "analysis.json": "{}", + "graph.cypher": "CREATE;", + "cache/entry.bin": "x\x00", + "notes/analysis.json": "{}", // not the output dir: kept + }) + + got, err := Discover(root, Options{OutputDir: root, CacheDir: cache}) + if err != nil { + t.Fatal(err) + } + if _, ok := got["analysis.json"]; ok { + t.Errorf("own output analysis.json was captured") + } + if _, ok := got["graph.cypher"]; ok { + t.Errorf("own output graph.cypher was captured") + } + if _, ok := got["cache/entry.bin"]; ok { + t.Errorf("cache subtree was captured") + } + if _, ok := got["notes/analysis.json"]; !ok { + t.Errorf("unrelated notes/analysis.json was wrongly excluded") + } + if _, ok := got["README.md"]; !ok { + t.Errorf("README.md missing") + } +} + +func TestFacts_NoGoModMeansArtifactsOnly(t *testing.T) { + root := t.TempDir() + writeTree(t, root, map[string]string{"Dockerfile": "FROM scratch\n"}) + + facts, err := Facts(root, appWithImports("fmt"), Options{}) + if err != nil { + t.Fatal(err) + } + if len(facts.Dependencies) != 0 { + t.Errorf("dependencies without go.mod: %+v", facts.Dependencies) + } + if _, ok := facts.Artifacts["Dockerfile"]; !ok { + t.Errorf("Dockerfile artifact missing") + } +} + +func TestParseGoSum_OnlyFullPairsLock(t *testing.T) { + root := t.TempDir() + writeTree(t, root, map[string]string{"go.sum": goSum}) + got := parseGoSum(filepath.Join(root, "go.sum")) + if !got["github.com/gorilla/mux"]["v1.8.0"] { + t.Errorf("mux v1.8.0 has both lines; want locked") + } + if got["cloud.google.com/go"]["v0.1.0"] { + t.Errorf("cloud.google.com/go has only h1; want not locked") + } + if got["golang.org/x/text"]["v0.14.0"] { + t.Errorf("golang.org/x/text has only /go.mod; want not locked") + } + if n := len(parseGoSum(filepath.Join(root, "absent"))); n != 0 { + t.Errorf("missing go.sum locked %d modules", n) + } +} + +func keys[V any](m map[string]V) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + return out +} diff --git a/internal/schema/schema.go b/internal/schema/schema.go index 84335d5..99bf2c2 100644 --- a/internal/schema/schema.go +++ b/internal/schema/schema.go @@ -214,6 +214,44 @@ type GoApplication struct { // Externals holds the library (out-of-project) functions the resolver saw // called, keyed by the full callee signature (".[Type.]Name"). Externals map[string]GoExternalCallee `json:"externals,omitempty"` + // Repository holds the repository-artifact facts (non-.go project files and + // go.mod/go.sum dependencies), set by the analyzer after symbol-table + // construction. Never cached or serialized in the v1 model: it is recomputed + // on every run, since it reads files outside the analyzed Go source. + Repository *RepositoryFacts `json:"-"` +} + +// RepositoryFacts is the repository-artifact layer (spec v2-artifacts): every +// non-.go project file as an artifact, and the go.mod requires as dependencies. +type RepositoryFacts struct { + // Artifacts is keyed by repo-relative POSIX path. + Artifacts map[string]GoArtifact + // Dependencies is sorted by module path. + Dependencies []GoDependency +} + +// GoArtifact is one non-.go project file. +type GoArtifact struct { + Path string + Format string + Roles []string + SizeBytes int64 + SHA256 string + Source string // whole file text, only when --artifact-text is on + Extraction string // none | full +} + +// GoDependency is one go.mod require, evidence-tagged by go.sum. +type GoDependency struct { + Name string + Ecosystem string + Spec string + Kind string + Direct bool + LockedVersion string // go.sum version of Spec, when both its h1 and /go.mod lines exist + DeclaredIn string // path of the go.mod artifact + ProvidesImports []string // project import paths that resolve to this module + Prov []string // declared | lockfile } // GoExternalCallee describes a library function or method called from the diff --git a/internal/schema/v2/schema.go b/internal/schema/v2/schema.go index 800c694..ce3eefa 100644 --- a/internal/schema/v2/schema.go +++ b/internal/schema/v2/schema.go @@ -97,6 +97,12 @@ type Application struct { // reaches, keyed by their `can:////@external//` id. // Absent when the analysis saw no library call. ExternalSymbols map[string]ExternalSymbol `json:"external_symbols,omitempty"` + // Artifacts holds every non-.go project file, keyed by its repo-relative path + // (spec v2-artifacts). Level-independent; absent when the project has no files. + Artifacts map[string]Artifact `json:"artifacts,omitempty"` + // Dependencies holds the go.mod requires, sorted by module path. Absent when + // the project has no root go.mod. + Dependencies []Dependency `json:"dependencies,omitempty"` // ParamIn and ParamOut are the L4 interprocedural parameter edges between // global body-node ids. Absent below -a 4. ParamIn []ParamEdge `json:"param_in,omitempty"` @@ -341,3 +347,31 @@ type Edge struct { // Weight accumulates when merging edges from multiple backends. Weight int `json:"weight,omitempty"` } + +// Artifact is one non-.go project file. Its id is can:////artifact/. +// Source is the whole file text only under --artifact-text; otherwise absent. +type Artifact struct { + ID string `json:"id"` + Kind string `json:"kind"` // always "artifact" + Path string `json:"path"` + Format string `json:"format"` + Roles []string `json:"roles,omitempty"` + SizeBytes int64 `json:"size_bytes"` + SHA256 string `json:"sha256"` + Source string `json:"source,omitempty"` + Extraction string `json:"extraction"` // none | full +} + +// Dependency is one declared module requirement from go.mod. Its package is +// pkg:/, version-less; the version lives on the record. +type Dependency struct { + Name string `json:"name"` + Ecosystem string `json:"ecosystem"` // always "golang" + Spec string `json:"spec"` // version required by go.mod + Kind string `json:"kind"` // always "runtime" (go.mod has no dev/optional split) + Direct bool `json:"direct"` // false for `// indirect` + LockedVersion string `json:"locked_version,omitempty"` + DeclaredIn string `json:"declared_in"` // the go.mod artifact id + ProvidesImports []string `json:"provides_imports,omitempty"` + Prov []string `json:"prov"` // declared | lockfile +} diff --git a/internal/syntactic_analysis/v2emit/emit.go b/internal/syntactic_analysis/v2emit/emit.go index 3f0279e..1371aa4 100644 --- a/internal/syntactic_analysis/v2emit/emit.go +++ b/internal/syntactic_analysis/v2emit/emit.go @@ -254,6 +254,7 @@ func (e *emitter) envelope(app *schema.GoApplication, maxLevel int, analyzerVers }, } out.Application.ExternalSymbols = emitExternalSymbols(e.appID, app.Externals, e.sdg.standIns()) + out.Application.Artifacts, out.Application.Dependencies = emitRepository(e.appID, app.Repository) if e.flow.on { out.KLimit = kLimit } diff --git a/internal/syntactic_analysis/v2emit/repository.go b/internal/syntactic_analysis/v2emit/repository.go new file mode 100644 index 0000000..b68924f --- /dev/null +++ b/internal/syntactic_analysis/v2emit/repository.go @@ -0,0 +1,57 @@ +package v2emit + +import ( + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// artifactID is an artifact's durable id: the application root, then the +// artifact segment and the repo-relative path (spec v2-artifacts). +func artifactID(appID, relPath string) string { + return appID + "/artifact/" + relPath +} + +// emitRepository maps the repository facts onto the v2 layer. Artifacts are +// keyed by path; dependencies keep the facts' order (sorted by module path) and +// point at their go.mod artifact by id. A nil facts set emits nothing. +func emitRepository(appID string, facts *schema.RepositoryFacts) (map[string]v2.Artifact, []v2.Dependency) { + if facts == nil { + return nil, nil + } + var arts map[string]v2.Artifact + if len(facts.Artifacts) > 0 { + arts = make(map[string]v2.Artifact, len(facts.Artifacts)) + for path, a := range facts.Artifacts { + arts[path] = v2.Artifact{ + ID: artifactID(appID, path), + Kind: "artifact", + Path: a.Path, + Format: a.Format, + Roles: a.Roles, + SizeBytes: a.SizeBytes, + SHA256: a.SHA256, + Source: a.Source, + Extraction: a.Extraction, + } + } + } + var deps []v2.Dependency + for _, d := range facts.Dependencies { + declared := "" + if d.DeclaredIn != "" { + declared = artifactID(appID, d.DeclaredIn) + } + deps = append(deps, v2.Dependency{ + Name: d.Name, + Ecosystem: d.Ecosystem, + Spec: d.Spec, + Kind: d.Kind, + Direct: d.Direct, + LockedVersion: d.LockedVersion, + DeclaredIn: declared, + ProvidesImports: d.ProvidesImports, + Prov: d.Prov, + }) + } + return arts, deps +} From 144cbcb656c4d5b9ff3c78f59f4199ee84fbefdc Mon Sep 17 00:00:00 2001 From: lambertpw Date: Fri, 20 Nov 2026 07:17:32 -0500 Subject: [PATCH 03/11] docs(design): schema v2 id format for Go callables and types Spec from the design loop: types /, methods //, functions /, closures keep closure_N, external ids /@external///, Neo4j signature from the JSON field. Follows codeanalyzer-typescript idFromSig and codeanalyzer-python child_id. --- docs/design/specs/v2-go-ids.md | 117 +++++++++++++++++++++++++++++++++ 1 file changed, 117 insertions(+) create mode 100644 docs/design/specs/v2-go-ids.md diff --git a/docs/design/specs/v2-go-ids.md b/docs/design/specs/v2-go-ids.md new file mode 100644 index 0000000..9ec996e --- /dev/null +++ b/docs/design/specs/v2-go-ids.md @@ -0,0 +1,117 @@ +# Spec: schema v2 id format for Go callables and types (codeanalyzer-go) + +> **Cross-repo spec.** The python-sdk Go backend keys types and fields by the last id segment, +> so the canonical home is `codellm-devkit/.github → docs/design/specs/`, beside the +> coordinating record. Staged locally on `feat/v2-go-ids` until that repo is available (same +> precedent as `v2-l1-emission.md` and `v2-artifacts.md`). This is the design transcript of the +> datamodel loop, linked from the tracking issue, not pasted into it. + +## Contract-Impact Triage + +**Does this change schema v2 output?** Yes. It changes the id shape of `type`, method, and +function nodes, and of `external_symbols` keys. It does not change the module, field, +artifact, or ordinal-id shapes, and it adds no field. + +| Change type | Analyzers | SDKs | Docs | +| --- | --- | --- | --- | +| Schema v2 id-format change | **codeanalyzer-go** (`internal/schema/v2/ids.go`, `v2emit`, Neo4j projector) | **python-sdk** Go backend (`cldk/analysis/go/neo4j/neo4j_backend.py` keys types and fields by last id segment; verify methods at the SDK rung) | `CLAUDE.md` § Schema decisions, `canonical-schema.md` references, this spec | + +Convention anchor: `codeanalyzer-typescript/src/schema/ids.ts` (`idFromSig`: strip the module +prefix from the dotted signature, split the tail on `.`) and `codeanalyzer-python/codeanalyzer/schema/ids.py` +(`child_id`, `ordinal_id`, `external_id`, `artifact_id`). + +## Problem + +Go ids embed the full import-qualified signature as one segment. For a method this repeats the +package path twice: + +``` +can://boutique/frontend/go/middleware.go/github.com/GoogleCloudPlatform/microservices-demo/src/frontend.logHandler/github.com/GoogleCloudPlatform/microservices-demo/src/frontend.logHandler.ServeHTTP/github.com/GoogleCloudPlatform/microservices-demo/src/frontend.logHandler.ServeHTTP.closure_1@82:3/2/actual_out:2 +``` + +The segment is not a path segment. A parser that splits on `/` (the Neo4j projector's +`lastSegment`) cannot recover a Go callable's name from its id, and the id is long and +unreadable. + +## Design-loop decisions (the transcript) + +Each decision was made with the user, one per question, in spine order. + +### Module (unchanged) +`can:////go/`. Already the convention. No question was needed. + +### Type +- **Tail = ``** (user decision, recommended). The Go package path is stripped from the + signature, then the type name is appended to the module: `/logHandler`. Precedent: + TypeScript `idFromSig`, Python `child_id`. The package path is not repeated, since the module + already names the scope. +- The package path that is stripped is the `pkgPath` of the symbol's signature, which equals + the package's import path. It is the same for every symbol in one file. + +### Callable +- **Function: `/`**, e.g. `middleware.go/ensureSessionID`. No receiver. +- **Method: `//`** (user decision, recommended). A method nests under its + receiver type, matching containment (`type.callables`). Precedent: TypeScript split-on-dots, + Python `child_id`. Example: `middleware.go/logHandler/ServeHTTP`. +- **Closure: `/closure_N`** (user decision, recommended). N is the source-order index + within the enclosing callable, as the analyzer numbers them today. Example: + `middleware.go/logHandler/ServeHTTP/closure_1`. Position is not used, because the body + ordinal key already carries it. +- **Ordinal and synthetic keys are unchanged**: `@`, `@entry`, `@exit`, + `@formal_in:`, `@actual_in:` / `@actual_out`, and the `/N` disambiguator for nested + calls at one position. Nothing in those changes. + +### Field +`/`, already in this shape. No change. + +### External symbols (decided earlier with the user) +`/@external///`, the TypeScript `/` form. The +import path stays whole, since it contains dots and slashes. A package-level function is +`/@external//`. + +### Call body node and call-graph edge +Unchanged in shape. A call node's `callee` and a call-graph edge's endpoints take the new +callable and external ids. No new field. + +### Projector: signature property (user decision) +The Neo4j `signature` property on `:GoCallable` and `:GoType` is copied from the node's JSON +`signature` field, not derived from the id. Today the projector takes the last `/` or `@` +segment of the id (`lastSegment`), which yields `Type.Method` and loses the import-qualified +name the JSON carries. The id no longer needs to carry it. + +## Consequences + +- **Parsing**: a Go id splits cleanly on `/`: module, then type, then method. The `@` ordinal + suffix is unchanged. +- **Uniqueness**: ids are unique within a file. Go forbids duplicate package-scope identifiers, + and forbids a duplicate method name on one type. Two files in one package have distinct + module ids, so the stripped package path never causes a collision. +- **Cross-file methods**: a method declared in another file nests under its receiver type's + module, as today (`source_file` handles its text). Its id is `//`. +- **python-sdk**: `cldk/analysis/go/neo4j/neo4j_backend.py` keys types by `ty.id.split("/")[-1]`, + which still returns the type name, and fields by `fld.id.split("/")[-1]`, which is unchanged. + Callables keyed by a dotted sig will change meaning: the last segment is now the bare method + name. The SDK rung must verify this. + +## Open items (flagged for review) + +1. **Name collision inside a file**: a type `X` and a function `X` cannot share a file's package + scope, so the ids cannot collide. Confirm no other Go construct gives two nodes the same + `//` (a type's method set is unique by name). Expected: none. +2. **Generic types**: a generic type's signature includes type parameters. The `[...]` part is + not part of the segment name. Decide at implementation: strip the type parameters (default) + or keep them. +3. **Dot-free function names in `external_symbols` keys**: keys become `//`. + Confirm the key is still unique when two packages share a last path element. + +## Tracking + +One issue in `codeanalyzer-go`, rungs as checklist items (user decision). Base branch +`feat/artifacts`, since the artifact code is the base for this work (user decision). Analyzer +ships first; the python-sdk Go backend follows once the id shape is released. + +## Out of scope + +- Module, field, artifact, and ordinal-id shapes. +- Any new node kind, edge kind, or field. +- Renaming the `signature` field (kept as-is). From 0f68195b53eee9ffd80952d01df3e78bd5270ecb Mon Sep 17 00:00:00 2001 From: lambertpw Date: Fri, 20 Nov 2026 07:28:50 -0500 Subject: [PATCH 04/11] feat(ids): Go callable and type ids use the bare member name (TS/Python convention) Type /, method //, function /, closure /closure_N; the package import path is dropped (the member is the signature part after its last dot). External ids /@external///. Type nodes gain a signature field (full import-qualified name) so the Neo4j :GoType signature no longer depends on the id. Projector copies signature from the JSON field. Refs codellm-devkit/codeanalyzer-go#46 --- CLAUDE.md | 22 ++++-- docs/design/specs/v2-go-ids.md | 13 ++++ internal/neo4j/project.go | 2 +- internal/neo4j/signature_test.go | 75 ++++++++++++++++++ internal/schema/v2/ids.go | 55 ++++++++++---- internal/schema/v2/ids_member_test.go | 76 +++++++++++++++++++ internal/schema/v2/ids_test.go | 2 +- internal/schema/v2/schema.go | 6 +- .../v2emit/determinism_freshbuild_test.go | 4 +- internal/syntactic_analysis/v2emit/emit.go | 1 + .../syntactic_analysis/v2emit/gate_l2_test.go | 8 +- .../syntactic_analysis/v2emit/gate_l3_test.go | 4 +- .../v2emit/gate_l4_done_test.go | 2 +- .../syntactic_analysis/v2emit/gate_test.go | 2 +- 14 files changed, 239 insertions(+), 33 deletions(-) create mode 100644 internal/neo4j/signature_test.go create mode 100644 internal/schema/v2/ids_member_test.go diff --git a/CLAUDE.md b/CLAUDE.md index 9091930..36b310e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -15,10 +15,21 @@ committed spec is the full transcript: `docs/design/specs/v2-l1-emission.md`. - **`application.id`** = `can://` (app outermost, so a polyglot app shares one prefix), where `` is the **`--app-name`** value, defaulting to the input directory's base name (per the CLI contract; the SDK's Neo4j backend must use the same anchor). Module ids are - `can:///go/`; externals are language-neutral `can:///@external//`. + `can:///go/`; externals are language-neutral `can:///@external///` (see § Id member segments). L3/L4 build ids on this unchanged. -- **Callable signature** = existing `signatureOf()` output as the last `can://` path - segment; receiver folded in for methods. One canonicalizer, unchanged from v1. +- **Id member segments (spec `docs/design/specs/v2-go-ids.md`, issue #46)**: a type is + `/`; a method is `//`; a package-level function is + `/`; a closure is `/closure_N`. The package import path is dropped: + the member segment is the signature's part after its last dot (Go names contain no dots). + Follows `codeanalyzer-typescript` `idFromSig` and `codeanalyzer-python` `child_id` + [user, 2026-11-20]. Externals: `/@external///` (import path + whole) [user, 2026-11-20]. +- **Callable and type `signature`** = the full `signatureOf()` output (import-qualified), a + JSON field on both (types gained it, additive, for the Neo4j `:GoType.signature`) [user, + 2026-11-20]. The id carries only the bare member name, so this field is where the package + path lives. One canonicalizer, unchanged from v1. +- **Generic types**: `signatureOf()` carries no type parameters, so a generic type or method + ids as `//`, with no `[T]` in the id. - **`span.bytes`** = **UTF-8 byte offsets** into `module.source` (Go strings are natively UTF-8; avoids the UTF-16/rune slicing mismatch TS hit in cants#179). @@ -64,8 +75,9 @@ committed spec is the full transcript: `docs/design/specs/v2-l1-emission.md`. `(*strings.Builder).WriteString`) is no longer dropped: the call node's `callee` and the `call_graph` edge `dst` are the callee's `@external` id. Builtins, type conversions (no longer minted as `.__new__`) and calls through function values stay `null`. -- **Id** = `can:///@external//` where `` is the callee signature - minus the `.` prefix (`Println`, `Builder.WriteString`); it may also name a +- **Id** = `can:///@external///` where `` is the callee signature + minus the `.` prefix; its dots become segments (`Println` → `…/fmt/Println`, + `Builder.WriteString` → `…/strings/Builder/WriteString`); it may also name a library *interface* method (`@external/io/Writer.Write`). Methods on generic library types record the declared (origin) signature, not the instance. `@external` is language-neutral (not `go/`), so a polyglot app shares one external namespace. diff --git a/docs/design/specs/v2-go-ids.md b/docs/design/specs/v2-go-ids.md index 9ec996e..4ed8d62 100644 --- a/docs/design/specs/v2-go-ids.md +++ b/docs/design/specs/v2-go-ids.md @@ -110,6 +110,19 @@ One issue in `codeanalyzer-go`, rungs as checklist items (user decision). Base b `feat/artifacts`, since the artifact code is the base for this work (user decision). Analyzer ships first; the python-sdk Go backend follows once the id shape is released. +## Implementation notes (2026-11-20) + +- **Type signature field (amended)**: the spec said this change adds no field. The Neo4j + `:GoType.signature` needs the full name, which the type id no longer carries, so `Type` + gained a `signature` JSON field (the full `signatureOf()`), mirroring `Callable`. The + projector copies it. Decided with the user. +- **Member name rule**: implemented as "the part after the last dot of the signature". This + is the same as stripping the import path, since Go identifiers contain no dots. +- **Open item 2 (generics) resolved**: `signatureOf()` has no type parameters, so a generic + type `Set[T]` ids as `set/set.go/Set` and its method as `set/set.go/Set/Add`. +- **Open item 1 (collisions) resolved**: the generics fixture and the chi/init fixtures + produce distinct ids. `init` functions are distinct per file by module. + ## Out of scope - Module, field, artifact, and ordinal-id shapes. diff --git a/internal/neo4j/project.go b/internal/neo4j/project.go index 04111d1..782ee2b 100644 --- a/internal/neo4j/project.go +++ b/internal/neo4j/project.go @@ -162,7 +162,7 @@ func (p *projector) projectType(modRef NodeRef, module string, t v2.Type) { "id", t.ID, Props{ "kind": t.Kind, // struct|interface|alias|defined (decision T1) - "signature": lastSegment(t.ID), + "signature": t.Signature, "span_json": spanJSON(t.Span), }, module, diff --git a/internal/neo4j/signature_test.go b/internal/neo4j/signature_test.go new file mode 100644 index 0000000..1df9ffc --- /dev/null +++ b/internal/neo4j/signature_test.go @@ -0,0 +1,75 @@ +package neo4j + +import ( + "testing" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// The :GoCallable signature property is the JSON signature field, the full +// import-qualified name, not a segment parsed out of the id (the id now carries +// only the bare member name). +func TestProject_CallableSignatureFromJSONField(t *testing.T) { + const ( + sig = "github.com/acme/app/worker.Worker.Run" + mod = "can://ns/app/go/worker.go" + ) + a := &v2.Analysis{ + IdentityVersion: 2, SchemaVersion: v2.SchemaVersion, Language: v2.Language, MaxLevel: 1, + Application: v2.Application{ + IdentityVersion: 2, ID: "can://ns/app", Kind: "application", + SymbolTable: map[string]v2.Module{ + "worker.go": { + ID: mod, Kind: "module", + Types: map[string]v2.Type{ + "Worker": { + ID: mod + "/Worker", Kind: "struct", Signature: "github.com/acme/app/worker.Worker", + Callables: map[string]v2.Callable{ + "Run": {ID: mod + "/Worker/Run", Kind: "method", Signature: sig}, + }, + }, + }, + }, + }, + }, + } + rows := Project(a) + for _, n := range rows.Nodes { + if hasLabel(n, LabelCallable) && n.Value == mod+"/Worker/Run" { + if got := n.Props["signature"]; got != sig { + t.Errorf("signature property = %v, want %q", got, sig) + } + return + } + } + t.Fatalf("no :GoCallable row for %s", mod+"/Worker/Run") +} + +// The :GoType signature is the type JSON's signature field (full name), not the +// bare type name the id ends with. +func TestProject_TypeSignatureFromJSONField(t *testing.T) { + const ( + sig = "github.com/acme/app/worker.Worker" + mod = "can://ns/app/go/worker.go" + ) + a := &v2.Analysis{ + IdentityVersion: 2, SchemaVersion: v2.SchemaVersion, Language: v2.Language, MaxLevel: 1, + Application: v2.Application{ + IdentityVersion: 2, ID: "can://ns/app", Kind: "application", + SymbolTable: map[string]v2.Module{ + "worker.go": {ID: mod, Kind: "module", Types: map[string]v2.Type{ + "Worker": {ID: mod + "/Worker", Kind: "struct", Signature: sig}, + }}, + }, + }, + } + for _, n := range Project(a).Nodes { + if hasLabel(n, LabelType) && n.Value == mod+"/Worker" { + if got := n.Props["signature"]; got != sig { + t.Errorf("type signature property = %v, want %q", got, sig) + } + return + } + } + t.Fatalf("no :GoType row for %s", mod+"/Worker") +} diff --git a/internal/schema/v2/ids.go b/internal/schema/v2/ids.go index e2e0788..2b2cf5f 100644 --- a/internal/schema/v2/ids.go +++ b/internal/schema/v2/ids.go @@ -15,18 +15,26 @@ import ( // Durable id grammar (the app segment is outermost so a polyglot app shares one // `can:///` prefix across languages; the language follows it): // -// can://///// -// can://myapp/myapp/go/src/util.go/Hasher/Hash(string)uint64 +// can:///// module +// can:////// type +// can://///// method +// can:////// package-level function +// can://myapp/myapp/go/util.go/Hasher/Hash e.g. a method +// +// Each member segment is the bare name: the package import path is dropped (it is +// always before the last dot of the signature, and Go names contain no dots). +// Closures nest under their enclosing callable as closure_N. // // External (out-of-project) symbols are language-neutral: // -// can:////@external// +// can:////@external/// +// can://myapp/myapp/@external/strings/Builder/WriteString // // Ordinal id grammar (statements and synthetic vertices, addressed WITHIN their // callable): // -// @: e.g. …/Hash(string)uint64@15:2 -// @ e.g. …/Hash(string)uint64@entry +// @: e.g. …/Hasher/Hash@15:2 +// @ e.g. …/Hasher/Hash@entry // // The delimiters `/`, `@`, `:` are fixed by the keystone; do not substitute. const ( @@ -91,13 +99,29 @@ func ModuleID(appID, relPath string) string { } // ExternalID builds the id of an out-of-project symbol: -// /@external//. It is language-neutral. An empty sig -// (an import-level ghost, no symbol) yields /@external/. +// /@external/// (or .../ for a +// package-level function). sig is the callee signature without the import-path +// prefix ("Client.Zone"); its dots become path segments, matching the in-project +// form. It is language-neutral. An empty sig (an import-level ghost, no symbol) +// yields /@external/. func ExternalID(appID, importPath, sig string) string { if sig == "" { return appID + idSep + "@external" + idSep + importPath } - return appID + idSep + "@external" + idSep + importPath + idSep + sig + return appID + idSep + "@external" + idSep + importPath + idSep + strings.ReplaceAll(sig, ".", idSep) +} + +// memberName is the member segment a symbol contributes under its parent: the +// part of its signature after the last dot. The package import path is always +// before that dot and Go identifiers contain no dots, so this strips the path +// and keeps the bare name (a function or type name, a method name, or a closure +// name such as closure_1). Precedent: codeanalyzer-typescript idFromSig and +// codeanalyzer-python child_id. +func memberName(sig string) string { + if i := strings.LastIndex(sig, "."); i >= 0 { + return sig[i+1:] + } + return sig } // GlobalOrdinal joins a callable id and a local ordinal: callableID + local @@ -110,17 +134,18 @@ func GlobalOrdinal(callableID, local string) string { return callableID + ordinalSep + local } -// TypeID builds a type id under its module. sig is the type's signatureOf() -// output — the last (and only) segment the type contributes. +// TypeID builds a type id under its module: /. sig is the type's +// signatureOf() output; the package path is dropped (see memberName). func TypeID(moduleID, sig string) string { - return moduleID + idSep + sig + return moduleID + idSep + memberName(sig) } -// CallableID builds a callable id under its parent. For a method the parent is -// its receiver Type id; for a module-level function the parent is the Module -// id. sig is the callable's signatureOf() output — the last path segment. +// CallableID builds a callable id under its parent: / for a +// method (parent is the receiver Type id), / for a package-level +// function (parent is the Module id), /closure_N for a closure. sig is +// the callable's signatureOf() output; the package path is dropped (see memberName). func CallableID(parentID, sig string) string { - return parentID + idSep + sig + return parentID + idSep + memberName(sig) } // OrdinalID addresses a real body node (statement / call) within its callable diff --git a/internal/schema/v2/ids_member_test.go b/internal/schema/v2/ids_member_test.go new file mode 100644 index 0000000..4cb931a --- /dev/null +++ b/internal/schema/v2/ids_member_test.go @@ -0,0 +1,76 @@ +package v2 + +import "testing" + +// The member segment is the signature's last dot-separated part, so the package +// import path (which may itself contain dots) never reaches the id. +func TestMemberIDs_DropPackagePath(t *testing.T) { + mod := ModuleID("can://ns/app", "go/w.go") + cases := []struct { + name string + got string + want string + }{ + {"type", TypeID(mod, "github.com/acme/app/worker.Worker"), mod + "/Worker"}, + {"function", CallableID(mod, "github.com/acme/app/worker.Run"), mod + "/Run"}, + {"method", CallableID(TypeID(mod, "github.com/acme/app/worker.Worker"), "github.com/acme/app/worker.Worker.Run"), mod + "/Worker/Run"}, + {"closure", CallableID(mod+"/Run", "github.com/acme/app/worker.Run.closure_2"), mod + "/Run/closure_2"}, + {"dotted import path", CallableID(mod, "gopkg.in/yaml.v2.Marshal"), mod + "/Marshal"}, + {"init", CallableID(mod, "github.com/acme/app/worker.init"), mod + "/init"}, + } + for _, c := range cases { + if c.got != c.want { + t.Errorf("%s: got %q, want %q", c.name, c.got, c.want) + } + } +} + +// Library callees: the import path stays whole, and the member's dots become +// segments, so Type and Member are separate segments. +func TestExternalID_PathForm(t *testing.T) { + const app = "can://ns/app" + cases := []struct { + importPath, sig, want string + }{ + {"strings", "Builder.WriteString", app + "/@external/strings/Builder/WriteString"}, + {"fmt", "Println", app + "/@external/fmt/Println"}, + {"database/sql", "DB.Query", app + "/@external/database/sql/DB/Query"}, + {"cloud.google.com/go/compute/metadata", "Client.Zone", app + "/@external/cloud.google.com/go/compute/metadata/Client/Zone"}, + {"net/http", "", app + "/@external/net/http"}, + } + for _, c := range cases { + if got := ExternalID(app, c.importPath, c.sig); got != c.want { + t.Errorf("ExternalID(%q, %q) = %q, want %q", c.importPath, c.sig, got, c.want) + } + } +} + +// Two library symbols with the same member name in different packages must keep +// distinct ids: the import path is the disambiguator. +func TestExternalID_SameMemberDifferentPackagesDistinct(t *testing.T) { + const app = "can://ns/app" + a := ExternalID(app, "encoding/json", "Decoder.Decode") + b := ExternalID(app, "encoding/xml", "Decoder.Decode") + if a == b { + t.Fatalf("encoding/json and encoding/xml Decoder.Decode collide at %s", a) + } +} + +// Within one module, every member name is unique: Go forbids duplicate package- +// scope identifiers and duplicate method names per type, so the bare names are +// safe as keys under their parent. +func TestMemberIDs_UniqueWithinModule(t *testing.T) { + mod := ModuleID("can://ns/app", "go/a.go") + ids := map[string]bool{} + for _, id := range []string{ + TypeID(mod, "pkg.T"), + CallableID(mod, "pkg.F"), + CallableID(TypeID(mod, "pkg.T"), "pkg.T.M"), + CallableID(TypeID(mod, "pkg.T"), "pkg.T.N"), + } { + if ids[id] { + t.Errorf("duplicate id %s", id) + } + ids[id] = true + } +} diff --git a/internal/schema/v2/ids_test.go b/internal/schema/v2/ids_test.go index 72e4ee3..7361348 100644 --- a/internal/schema/v2/ids_test.go +++ b/internal/schema/v2/ids_test.go @@ -97,7 +97,7 @@ func TestIDs_AppOutermost(t *testing.T) { cases := []struct{ got, want string }{ {app, "can://shop/shop"}, {ModuleID(app, "pkg/a.go"), "can://shop/shop/go/pkg/a.go"}, - {ExternalID(app, "database/sql", "DB.Query"), "can://shop/shop/@external/database/sql/DB.Query"}, + {ExternalID(app, "database/sql", "DB.Query"), "can://shop/shop/@external/database/sql/DB/Query"}, {GlobalOrdinal("X/F", "@entry"), "X/F@entry"}, {GlobalOrdinal("X/F", "12:3"), "X/F@12:3"}, } diff --git a/internal/schema/v2/schema.go b/internal/schema/v2/schema.go index ce3eefa..3c5ca1a 100644 --- a/internal/schema/v2/schema.go +++ b/internal/schema/v2/schema.go @@ -185,6 +185,9 @@ type Type struct { ID string `json:"id"` Kind string `json:"kind"` // struct | interface | alias | defined Span Span `json:"span"` + // Signature is the type's full import-qualified name (one signatureOf()). The + // id carries only the bare type name, so this is where the package path lives. + Signature string `json:"signature"` // BaseTypes are embedded type ids (struct/interface embedding — the explicit // spine). BaseTypes []string `json:"base_types,omitempty"` @@ -214,7 +217,8 @@ type Callable struct { ID string `json:"id"` Kind string `json:"kind"` // function | method | lambda Span Span `json:"span"` - // Signature is the human-readable last path segment of ID (one signatureOf()). + // Signature is the full import-qualified name (one signatureOf()). The id + // carries only the bare member name, so this field is where the package path lives. Signature string `json:"signature"` // SourceFile names the callable's declaring file (a symbol_table key) when it // differs from the module this callable is nested under — the Go case where a diff --git a/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go b/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go index b4b7d6a..f0d73ac 100644 --- a/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go +++ b/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go @@ -116,7 +116,7 @@ func TestInitCollision_DistinctNodesAndEdges(t *testing.T) { continue } for _, fn := range mod.Functions { - if strings.HasSuffix(fn.ID, "middleware.init") { + if strings.HasSuffix(fn.ID, "/init") { initNodeIDs[fn.ID] = true } } @@ -131,7 +131,7 @@ func TestInitCollision_DistinctNodesAndEdges(t *testing.T) { // edge resolved to ITS OWN file's init, so the id is self-consistent). sawInitEdge := false for _, e := range out.Application.CallGraph { - if !strings.HasSuffix(e.Src, "middleware.init") { + if !strings.HasSuffix(e.Src, "/init") { continue } sawInitEdge = true diff --git a/internal/syntactic_analysis/v2emit/emit.go b/internal/syntactic_analysis/v2emit/emit.go index 1371aa4..55e81a4 100644 --- a/internal/syntactic_analysis/v2emit/emit.go +++ b/internal/syntactic_analysis/v2emit/emit.go @@ -310,6 +310,7 @@ func emitType(modID, modRel string, t schema.GoType, idx sigIndex, srcs *sourceC out := v2.Type{ ID: typeID, Kind: typeKind(t), + Signature: t.Signature, Span: lineSpan(li, t.StartLine, t.EndLine), BaseTypes: nonEmpty(t.BaseTypes), Callables: make(map[string]v2.Callable, len(t.Methods)), diff --git a/internal/syntactic_analysis/v2emit/gate_l2_test.go b/internal/syntactic_analysis/v2emit/gate_l2_test.go index d402ed6..412d424 100644 --- a/internal/syntactic_analysis/v2emit/gate_l2_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l2_test.go @@ -110,8 +110,8 @@ func TestGateL2_NamedEdgePresent(t *testing.T) { out := buildMultipackageL2(t) const ( - src = "can://multipackage/multipackage/go/worker/worker.go/example.com/multipackage/worker.Worker/example.com/multipackage/worker.Worker.Run" - dst = "can://multipackage/multipackage/go/worker/worker.go/example.com/multipackage/worker.Worker/example.com/multipackage/worker.Worker.execute" + src = "can://multipackage/multipackage/go/worker/worker.go/Worker/Run" + dst = "can://multipackage/multipackage/go/worker/worker.go/Worker/execute" ) found := false for _, e := range out.Application.CallGraph { @@ -133,8 +133,8 @@ func TestGateL2_CrossPackageEdge(t *testing.T) { out := buildMultipackageL2(t) const ( - src = "can://multipackage/multipackage/go/main.go/example.com/multipackage.main" - dst = "can://multipackage/multipackage/go/server/server.go/example.com/multipackage/server.New" + src = "can://multipackage/multipackage/go/main.go/main" + dst = "can://multipackage/multipackage/go/server/server.go/New" ) found := false for _, e := range out.Application.CallGraph { diff --git a/internal/syntactic_analysis/v2emit/gate_l3_test.go b/internal/syntactic_analysis/v2emit/gate_l3_test.go index a9f17fc..ddec91a 100644 --- a/internal/syntactic_analysis/v2emit/gate_l3_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l3_test.go @@ -508,7 +508,7 @@ func TestSymbolTable_MethodDeclaredAboveItsType(t *testing.T) { for _, mod := range out.Application.SymbolTable { for _, ty := range mod.Types { for _, m := range ty.Callables { - if strings.HasSuffix(m.ID, "/"+sig) { + if strings.HasSuffix(m.ID, "/early/Bump") { return m, true } } @@ -531,7 +531,7 @@ func TestSymbolTable_MethodDeclaredAboveItsType(t *testing.T) { if call == nil { t.Fatalf("%s has no call site", sig) } - if call.Callee == nil || !strings.HasSuffix(*call.Callee, "/"+fID) { + if call.Callee == nil || !strings.HasSuffix(*call.Callee, "/f") { t.Fatalf("call to f not resolved at -a 2: callee = %v", call.Callee) } sawEdge := false diff --git a/internal/syntactic_analysis/v2emit/gate_l4_done_test.go b/internal/syntactic_analysis/v2emit/gate_l4_done_test.go index 6803719..c22142b 100644 --- a/internal/syntactic_analysis/v2emit/gate_l4_done_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l4_done_test.go @@ -144,7 +144,7 @@ func TestGateL4_HTTPToSQLPath(t *testing.T) { method := func(typ, m string) string { return v2.CallableID(v2.TypeID(mod, pkg+typ), pkg+typ+"."+m) } sink := vertex(v2.ExternalID(app, "database/sql", "DB.Query"), "formal_in", "query") - if sink != "can://httpsql/httpsql/@external/database/sql/DB.Query@formal_in:1" { + if sink != "can://httpsql/httpsql/@external/database/sql/DB/Query@formal_in:1" { t.Fatalf("sink vertex %s", sink) } find := method("sqlRepo", "Find") diff --git a/internal/syntactic_analysis/v2emit/gate_test.go b/internal/syntactic_analysis/v2emit/gate_test.go index 41822be..321cdaf 100644 --- a/internal/syntactic_analysis/v2emit/gate_test.go +++ b/internal/syntactic_analysis/v2emit/gate_test.go @@ -599,7 +599,7 @@ func TestGateL2_GoStatementArgumentCalls(t *testing.T) { if w.callee == "" { continue } - want := "can://dataflow/dataflow/go/flow/flow.go/" + pkg + w.callee + want := "can://dataflow/dataflow/go/flow/flow.go/" + w.callee if n.Callee == nil || *n.Callee != want { t.Errorf("GoArgs %s callee = %v, want %s", w.key, n.Callee, want) } From 7ed66ce9b6ea211238752158e445d3d59b402820 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Fri, 20 Nov 2026 07:56:57 -0500 Subject: [PATCH 05/11] test(v2emit): drop unused constants after the id-format change Remove the fID and pkg constants left unused by the bare-member id expectations; golangci-lint's unused check failed CI on them. Refs codellm-devkit/codeanalyzer-go#46 --- internal/syntactic_analysis/v2emit/gate_l3_test.go | 1 - internal/syntactic_analysis/v2emit/gate_test.go | 1 - 2 files changed, 2 deletions(-) diff --git a/internal/syntactic_analysis/v2emit/gate_l3_test.go b/internal/syntactic_analysis/v2emit/gate_l3_test.go index ddec91a..3b98d89 100644 --- a/internal/syntactic_analysis/v2emit/gate_l3_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l3_test.go @@ -502,7 +502,6 @@ func TestGateL3_LineDirectiveRealPositions(t *testing.T) { func TestSymbolTable_MethodDeclaredAboveItsType(t *testing.T) { fx := buildL3(t, "dataflow") const sig = "example.com/dataflow/flow.early.Bump" - const fID = "example.com/dataflow/flow.f" find := func(out *v2.Analysis) (v2.Callable, bool) { for _, mod := range out.Application.SymbolTable { diff --git a/internal/syntactic_analysis/v2emit/gate_test.go b/internal/syntactic_analysis/v2emit/gate_test.go index 321cdaf..86382da 100644 --- a/internal/syntactic_analysis/v2emit/gate_test.go +++ b/internal/syntactic_analysis/v2emit/gate_test.go @@ -578,7 +578,6 @@ func TestGateL1_BodyNodeIDs(t *testing.T) { func TestGateL2_GoStatementArgumentCalls(t *testing.T) { out := buildDataflowL2V2(t) fn := functionByName(t, mustModule(t, out, "flow/flow.go"), "GoArgs") - const pkg = "example.com/dataflow/flow." for _, w := range []struct { key, callee string goroutine bool From 210a5111e461a27e0636c3ca0f6063ef183af07a Mon Sep 17 00:00:00 2001 From: lambertpw Date: Fri, 20 Nov 2026 07:59:07 -0500 Subject: [PATCH 06/11] scripts: add end-to-end eval (analysis.json -> Neo4j projection -> load) eval_e2e.sh runs cango -a 2 on an app, pushes the Neo4j projection live, and checks analysis.json and the loaded graph (app-scoped GoCanNode rows plus :Package nodes). EAGER=1 reloads: the push runs with --eager, purging the application's subgraph first. --- scripts/eval_e2e.sh | 166 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 166 insertions(+) create mode 100755 scripts/eval_e2e.sh diff --git a/scripts/eval_e2e.sh b/scripts/eval_e2e.sh new file mode 100755 index 0000000..84d1558 --- /dev/null +++ b/scripts/eval_e2e.sh @@ -0,0 +1,166 @@ +#!/usr/bin/env bash +# eval_e2e.sh - end-to-end check: Go app -> analysis.json -> Neo4j projection -> loaded graph. +# +# Usage (from the repository root): +# scripts/eval_e2e.sh APP_DIR [OUT_DIR] +# +# APP_DIR a Go module root (must contain go.mod). +# OUT_DIR where analysis.json and the run log go (default: APP_DIR). +# +# Example: +# NEO4J_PASSWORD='...' scripts/eval_e2e.sh /path/to/microservices-demo/src/frontend +# +# Environment: +# LEVEL analysis level for analysis.json (default 2; 1..4) +# APP_NAME --app-name anchor (default: APP_DIR base name, as cango does) +# CANGO analyzer binary. When unset, it is built from this checkout into +# OUT_DIR/cango with `go build ./cmd/codeanalyzer`. +# NEO4J_URI Bolt URI (default bolt://localhost:7687) +# NEO4J_USER username (default neo4j) +# NEO4J_PASSWORD password (default neo4j). Passed to cango through its env var; never echoed. +# EAGER 1 = reload: the push runs with --eager, purging this application's +# subgraph (its can:// prefix only) first. Default 0 (additive upsert). +# CYPHER_SHELL cypher-shell binary for verification (auto-detected when unset). When it +# cannot be found or connected, the count check is reported as skipped. +# +# Steps: +# 1. Validate APP_DIR and the requested level. +# 2. Build the analyzer (unless CANGO is set). +# 3. cango -a LEVEL -i APP_DIR -o OUT_DIR -> OUT_DIR/analysis.json +# 4. cango --emit neo4j --neo4j-uri URI ... -> live push (MERGE, idempotent) +# 5. Verify analysis.json (schema, level, app id) and that the database holds the +# pushed node count under this application's can:// prefix. +# +# Exit status: 0 when every check passes (or is skipped with a reason), non-zero otherwise. +set -euo pipefail + +if [ $# -lt 1 ]; then + sed -n '2,/^set -euo/p' "$0" | sed '$d' >&2 + exit 2 +fi + +REPO=$(cd "$(dirname "$0")/.." && pwd) +APP=$(cd "$1" && pwd) +OUT=${2:-$APP} +mkdir -p "$OUT" +OUT=$(cd "$OUT" && pwd) + +LEVEL=${LEVEL:-2} +URI=${NEO4J_URI:-bolt://localhost:7687} +USER=${NEO4J_USER:-neo4j} +PASS=${NEO4J_PASSWORD:-neo4j} +APP_NAME=${APP_NAME:-$(basename "$APP")} +# The run log lives outside OUT: a log inside the project would be captured as an artifact. +LOG=${E2E_LOG:-${TMPDIR:-/tmp}/cldk-e2e-${APP_NAME}.log} + +fail=0 +step() { echo; echo "== $1"; } +pass() { echo " PASS: $1"; } +bad() { echo " FAIL: $1" >&2; fail=1; } +skip() { echo " SKIP: $1"; } + +# Strings that become id segments or Cypher literals must be plain names. +case "$APP_NAME" in + *[!A-Za-z0-9._-]*|"") echo "APP_NAME '$APP_NAME' must contain only letters, digits, '.', '_' or '-'" >&2; exit 2 ;; +esac +case "$LEVEL" in 1|2|3|4) ;; *) echo "LEVEL must be 1..4 (got '$LEVEL')" >&2; exit 2 ;; esac +[ -f "$APP/go.mod" ] || { echo "APP_DIR '$APP' has no go.mod; not a Go module root" >&2; exit 2; } + +: >"$LOG" + +step "1/5 analyzer binary" +if [ -n "${CANGO:-}" ]; then + [ -x "$CANGO" ] || { echo "CANGO '$CANGO' is not executable" >&2; exit 2; } + echo " using $CANGO" +else + CANGO="$OUT/cango" + (cd "$REPO" && go build -o "$CANGO" ./cmd/codeanalyzer) >>"$LOG" 2>&1 + echo " built $CANGO from $REPO" +fi + +step "2/5 analysis.json (-a $LEVEL)" +"$CANGO" -a "$LEVEL" --app-name "$APP_NAME" -i "$APP" -o "$OUT" >>"$LOG" 2>&1 +[ -s "$OUT/analysis.json" ] && pass "wrote $OUT/analysis.json ($(wc -c <"$OUT/analysis.json" | tr -d ' ') bytes)" \ + || { bad "analysis.json missing or empty (see $LOG)"; exit 1; } + +step "3/5 Neo4j projection (live push to $URI as $USER)" +# EAGER=1 reloads: --eager purges this application's subgraph (its can://// +# prefix only) before the push, so nodes from an earlier id format do not linger. +EAGER_FLAG=() +[ "${EAGER:-0}" = "1" ] && EAGER_FLAG=(--eager) +push_out=$(NEO4J_PASSWORD="$PASS" "$CANGO" --emit neo4j --neo4j-uri "$URI" --neo4j-user "$USER" \ + --app-name "$APP_NAME" "${EAGER_FLAG[@]}" -i "$APP" -o "$OUT" 2>&1) && push_rc=0 || push_rc=$? +printf '%s\n' "$push_out" >>"$LOG" +if [ "$push_rc" -ne 0 ]; then + bad "projection push failed (exit $push_rc): $(printf '%s' "$push_out" | tail -1)" + exit 1 +fi +pushed=$(printf '%s\n' "$push_out" | sed -n 's/^pushed \([0-9]*\) nodes, \([0-9]*\) relationships.*/\1 \2/p' | tail -1) +[ -n "$pushed" ] && pass "pushed $(printf '%s' "$pushed" | cut -d' ' -f1) nodes, $(printf '%s' "$pushed" | cut -d' ' -f2) relationships" \ + || bad "push output had no 'pushed N nodes' line" + +step "4/5 verify analysis.json" +if command -v python3 >/dev/null 2>&1; then + if python3 -I - "$OUT/analysis.json" "$LEVEL" "$APP_NAME" <<'PY' +import json, sys +path, level, app = sys.argv[1], int(sys.argv[2]), sys.argv[3] +d = json.load(open(path, encoding="utf-8")) +errs = [] +if d.get("language") != "go": errs.append(f"language is {d.get('language')!r}") +if d.get("max_level") != level: errs.append(f"max_level is {d.get('max_level')!r}, expected {level}") +app_id = d.get("application", {}).get("id", "") +if not app_id.startswith(f"can://"): errs.append(f"application.id {app_id!r} is not a can:// id") +if app not in app_id: errs.append(f"application.id {app_id!r} lacks the app name {app!r}") +st = d.get("application", {}).get("symbol_table", {}) +if not st: errs.append("symbol_table is empty") +if errs: + print("\n".join(" FAIL: " + e for e in errs)); sys.exit(1) +print(f" PASS: schema {d.get('schema_version')}, level {level}, {len(st)} files, " + f"{len(d['application'].get('call_graph', []))} call edges, id {app_id}") +PY + then :; else fail=1; fi +else + skip "python3 not found; analysis.json not inspected" +fi + +step "5/5 verify loaded graph" +CS=${CYPHER_SHELL:-} +if [ -z "$CS" ]; then + CS=$(command -v cypher-shell || true) + [ -n "$CS" ] || for c in /opt/homebrew/bin/cypher-shell /usr/local/bin/cypher-shell; do + [ -x "$c" ] && { CS=$c; break; } + done +fi +if [ -z "$CS" ]; then + skip "cypher-shell not found; set CYPHER_SHELL to enable the count check" +else + shell() { "$CS" -a "$URI" -u "$USER" -p "$PASS" --format plain "$@" 2>/dev/null; } + if ! shell "RETURN 1" >/dev/null; then + skip "cannot authenticate to $URI with cypher-shell; the count check was not run" + else + # Nodes keyed by this application's id root, the same prefix the projector scopes on. + prefix="can://${APP_NAME}/" + db_nodes=$(shell "MATCH (n:GoCanNode) WHERE n.id STARTS WITH '$prefix' RETURN count(n) AS c" | tail -1 | tr -d '"[:space:]') + want_nodes=$(printf '%s' "$pushed" | cut -d' ' -f1) + # Dependency :Package nodes are neutral (pkg: purls, shared across apps), so they are + # not under the prefix. They are part of the push count and checked separately. + db_pkgs=$(shell "MATCH (n:Package) WHERE n.id STARTS WITH 'pkg:' RETURN count(n) AS c" | tail -1 | tr -d '"[:space:]') + echo " info: $db_pkgs :Package nodes in the database" + # Pushed nodes are app-scoped (GoCanNode under the prefix) plus neutral :Package nodes. + if [ -n "$db_nodes" ] && [ "$db_nodes" -gt 0 ] && [ "$((db_nodes + ${db_pkgs:-0}))" -ge "$want_nodes" ]; then + pass "database holds $db_nodes GoCanNode rows under $prefix and ${db_pkgs:-0} :Package nodes (pushed $want_nodes nodes)" + else + bad "database has ${db_nodes:-no} GoCanNode rows under $prefix plus ${db_pkgs:-0} :Package; expected at least $want_nodes" + fi + labels=$(shell "MATCH (n:GoCallable) WHERE n.id STARTS WITH '$prefix' RETURN count(n) AS c" | tail -1 | tr -d '"[:space:]') + echo " info: $labels GoCallable nodes under $prefix" + fi +fi + +echo +if [ "$fail" -eq 0 ]; then + echo "E2E OK: $APP_NAME -> $OUT/analysis.json -> Neo4j at $URI" +else + echo "E2E FAILED for $APP_NAME (log: $LOG)" >&2 +fi +exit "$fail" From e605867a40b50277aeddc750762c20cf89db71d4 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Fri, 20 Nov 2026 08:39:11 -0500 Subject: [PATCH 07/11] scripts(eval_e2e): fix bash 3.2 unbound array and build outside the project EAGER_FLAG is a plain string, since an empty array trips set -u on macOS bash 3.2. The default cango build goes to the temp dir, not the project, so the binary is not captured as an artifact. --- scripts/eval_e2e.sh | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/scripts/eval_e2e.sh b/scripts/eval_e2e.sh index 84d1558..27de8cb 100755 --- a/scripts/eval_e2e.sh +++ b/scripts/eval_e2e.sh @@ -73,7 +73,8 @@ if [ -n "${CANGO:-}" ]; then [ -x "$CANGO" ] || { echo "CANGO '$CANGO' is not executable" >&2; exit 2; } echo " using $CANGO" else - CANGO="$OUT/cango" + # Build outside the project: a binary inside it would be captured as an artifact. + CANGO="${E2E_BUILD_DIR:-${TMPDIR:-/tmp}}/cldk-e2e-cango-${APP_NAME}" (cd "$REPO" && go build -o "$CANGO" ./cmd/codeanalyzer) >>"$LOG" 2>&1 echo " built $CANGO from $REPO" fi @@ -86,10 +87,11 @@ step "2/5 analysis.json (-a $LEVEL)" step "3/5 Neo4j projection (live push to $URI as $USER)" # EAGER=1 reloads: --eager purges this application's subgraph (its can://// # prefix only) before the push, so nodes from an earlier id format do not linger. -EAGER_FLAG=() -[ "${EAGER:-0}" = "1" ] && EAGER_FLAG=(--eager) +# A plain string, not an array: an empty array trips `set -u` on bash 3.2 (macOS). +EAGER_FLAG="" +[ "${EAGER:-0}" = "1" ] && EAGER_FLAG="--eager" push_out=$(NEO4J_PASSWORD="$PASS" "$CANGO" --emit neo4j --neo4j-uri "$URI" --neo4j-user "$USER" \ - --app-name "$APP_NAME" "${EAGER_FLAG[@]}" -i "$APP" -o "$OUT" 2>&1) && push_rc=0 || push_rc=$? + --app-name "$APP_NAME" $EAGER_FLAG -i "$APP" -o "$OUT" 2>&1) && push_rc=0 || push_rc=$? printf '%s\n' "$push_out" >>"$LOG" if [ "$push_rc" -ne 0 ]; then bad "projection push failed (exit $push_rc): $(printf '%s' "$push_out" | tail -1)" From 85e2ed141acfc1b0e626957b470436d44204a68b Mon Sep 17 00:00:00 2001 From: lambertpw Date: Fri, 20 Nov 2026 09:18:38 -0500 Subject: [PATCH 08/11] scripts(eval_e2e): add --json-only to write and check analysis.json only Skips the Neo4j push and the graph check, so no password or running database is needed. The build and the analysis.json check still run. --- scripts/eval_e2e.sh | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/scripts/eval_e2e.sh b/scripts/eval_e2e.sh index 27de8cb..34188ec 100755 --- a/scripts/eval_e2e.sh +++ b/scripts/eval_e2e.sh @@ -2,13 +2,16 @@ # eval_e2e.sh - end-to-end check: Go app -> analysis.json -> Neo4j projection -> loaded graph. # # Usage (from the repository root): -# scripts/eval_e2e.sh APP_DIR [OUT_DIR] +# scripts/eval_e2e.sh [--json-only] APP_DIR [OUT_DIR] # -# APP_DIR a Go module root (must contain go.mod). -# OUT_DIR where analysis.json and the run log go (default: APP_DIR). +# APP_DIR a Go module root (must contain go.mod). +# OUT_DIR where analysis.json and the run log go (default: APP_DIR). +# --json-only write and check analysis.json only: no Neo4j push, no graph check, +# no password or running database needed. # # Example: # NEO4J_PASSWORD='...' scripts/eval_e2e.sh /path/to/microservices-demo/src/frontend +# scripts/eval_e2e.sh --json-only /path/to/microservices-demo/src/frontend # # Environment: # LEVEL analysis level for analysis.json (default 2; 1..4) @@ -34,6 +37,9 @@ # Exit status: 0 when every check passes (or is skipped with a reason), non-zero otherwise. set -euo pipefail +JSON_ONLY=0 +if [ "${1:-}" = "--json-only" ]; then JSON_ONLY=1; shift; fi + if [ $# -lt 1 ]; then sed -n '2,/^set -euo/p' "$0" | sed '$d' >&2 exit 2 @@ -84,6 +90,9 @@ step "2/5 analysis.json (-a $LEVEL)" [ -s "$OUT/analysis.json" ] && pass "wrote $OUT/analysis.json ($(wc -c <"$OUT/analysis.json" | tr -d ' ') bytes)" \ || { bad "analysis.json missing or empty (see $LOG)"; exit 1; } +if [ "$JSON_ONLY" = "1" ]; then + step "3/5 Neo4j projection"; skip "--json-only: no push" +else step "3/5 Neo4j projection (live push to $URI as $USER)" # EAGER=1 reloads: --eager purges this application's subgraph (its can://// # prefix only) before the push, so nodes from an earlier id format do not linger. @@ -100,6 +109,7 @@ fi pushed=$(printf '%s\n' "$push_out" | sed -n 's/^pushed \([0-9]*\) nodes, \([0-9]*\) relationships.*/\1 \2/p' | tail -1) [ -n "$pushed" ] && pass "pushed $(printf '%s' "$pushed" | cut -d' ' -f1) nodes, $(printf '%s' "$pushed" | cut -d' ' -f2) relationships" \ || bad "push output had no 'pushed N nodes' line" +fi step "4/5 verify analysis.json" if command -v python3 >/dev/null 2>&1; then @@ -125,6 +135,9 @@ else skip "python3 not found; analysis.json not inspected" fi +if [ "$JSON_ONLY" = "1" ]; then + step "5/5 verify loaded graph"; skip "--json-only: graph not checked" +else step "5/5 verify loaded graph" CS=${CYPHER_SHELL:-} if [ -z "$CS" ]; then @@ -159,9 +172,15 @@ else fi fi +fi + echo if [ "$fail" -eq 0 ]; then - echo "E2E OK: $APP_NAME -> $OUT/analysis.json -> Neo4j at $URI" + if [ "$JSON_ONLY" = "1" ]; then + echo "E2E OK (json-only): $APP_NAME -> $OUT/analysis.json" + else + echo "E2E OK: $APP_NAME -> $OUT/analysis.json -> Neo4j at $URI" + fi else echo "E2E FAILED for $APP_NAME (log: $LOG)" >&2 fi From 000ebad234620a6d8485f7acc2572df0ad00ffef Mon Sep 17 00:00:00 2001 From: lambertpw Date: Fri, 20 Nov 2026 10:47:17 -0500 Subject: [PATCH 09/11] docs(design): cross-file method ids use the declaring file A method declared in another file than its receiver type now gets an id under the declaring module: //. The node stays in the receiver's callables{}; source_file is kept; Neo4j adds GO_DECLARES from the declaring module. Refs codellm-devkit/codeanalyzer-go#46 --- CLAUDE.md | 8 +++++++- docs/design/specs/v2-go-ids.md | 32 ++++++++++++++++++++++++++++++-- 2 files changed, 37 insertions(+), 3 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 36b310e..769a9cf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,7 +19,8 @@ committed spec is the full transcript: `docs/design/specs/v2-l1-emission.md`. L3/L4 build ids on this unchanged. - **Id member segments (spec `docs/design/specs/v2-go-ids.md`, issue #46)**: a type is `/`; a method is `//`; a package-level function is - `/`; a closure is `/closure_N`. The package import path is dropped: + `/` (a method declared in another file uses that file as ``, see + § Cross-file method ids below); a closure is `/closure_N`. The package import path is dropped: the member segment is the signature's part after its last dot (Go names contain no dots). Follows `codeanalyzer-typescript` `idFromSig` and `codeanalyzer-python` `child_id` [user, 2026-11-20]. Externals: `/@external///` (import path @@ -28,6 +29,11 @@ committed spec is the full transcript: `docs/design/specs/v2-l1-emission.md`. JSON field on both (types gained it, additive, for the Neo4j `:GoType.signature`) [user, 2026-11-20]. The id carries only the bare member name, so this field is where the package path lives. One canonicalizer, unchanged from v1. +- **Cross-file method ids** [user, 2026-11-20]: a method declared in another file than its receiver + type gets `//`, e.g. `…/handlers.go/frontendServer/logoutHandler`. + The node stays in the receiver type's `callables{}`. `source_file` is kept for text recovery. + Neo4j adds `GO_DECLARES` from the declaring `:GoModule` to the method, beside `GO_HAS_METHOD`. + Spec: `docs/design/specs/v2-go-ids.md` § Amendment. - **Generic types**: `signatureOf()` carries no type parameters, so a generic type or method ids as `//`, with no `[T]` in the id. - **`span.bytes`** = **UTF-8 byte offsets** into `module.source` (Go strings are diff --git a/docs/design/specs/v2-go-ids.md b/docs/design/specs/v2-go-ids.md index 4ed8d62..b35ba2d 100644 --- a/docs/design/specs/v2-go-ids.md +++ b/docs/design/specs/v2-go-ids.md @@ -86,8 +86,10 @@ name the JSON carries. The id no longer needs to carry it. - **Uniqueness**: ids are unique within a file. Go forbids duplicate package-scope identifiers, and forbids a duplicate method name on one type. Two files in one package have distinct module ids, so the stripped package path never causes a collision. -- **Cross-file methods**: a method declared in another file nests under its receiver type's - module, as today (`source_file` handles its text). Its id is `//`. +- **Cross-file methods** (amended 2026-11-20, see Amendment below): a method declared in another + file gets an id under the file that declares it, `//`. The + method node stays in its receiver type's `callables{}`, and `source_file` names the declaring + file for text recovery. - **python-sdk**: `cldk/analysis/go/neo4j/neo4j_backend.py` keys types by `ty.id.split("/")[-1]`, which still returns the type name, and fields by `fld.id.split("/")[-1]`, which is unchanged. Callables keyed by a dotted sig will change meaning: the last segment is now the bare method @@ -123,6 +125,32 @@ ships first; the python-sdk Go backend follows once the id shape is released. - **Open item 1 (collisions) resolved**: the generics fixture and the chi/init fixtures produce distinct ids. `init` functions are distinct per file by module. +## Amendment (2026-11-20): cross-file method ids + +Decided with the user in the design loop, after the id change landed. + +**Problem.** A method is nested under its receiver type's module, so its id names the file +where the receiver lives, not the file where the method is declared. Example: `frontendServer.logoutHandler` +is declared in `handlers.go` but its id was `…/main.go/frontendServer/logoutHandler`. 22 methods in the +frontend app are affected (those with `source_file` set). + +**Decisions.** + +1. **Id.** A callable's id contains the file where it is declared: + `//`, e.g. `…/handlers.go/frontendServer/logoutHandler`. The type + segment stays. Rejected: `/`, because a method and a package-level + function of the same name in one file would collide. +2. **JSON containment.** The node stays in its receiver type's `callables{}` (containment by + receiver is unchanged). Only its id changes. Moving it under the declaring module was rejected + because it changes containment for every consumer and hides which type the method belongs to. +3. **Text recovery.** `source_file` is kept. The id's module now equals `source_file`, but the + node still nests under the receiver's module, so its span still needs `source_file` to resolve. +4. **Neo4j.** `GO_HAS_METHOD` from the receiver `:GoType` is kept. A `GO_DECLARES` edge is added from + the declaring `:GoModule` to the method, so "which file declares this method" is one query. No + new relationship kind. + +**Release.** Same PR and release as the id format (feat/v2-go-ids). Tracked in #46. + ## Out of scope - Module, field, artifact, and ordinal-id shapes. From dd9349bf92ab166fba734a330a8f090f06d99aeb Mon Sep 17 00:00:00 2001 From: lambertpw Date: Fri, 20 Nov 2026 10:50:51 -0500 Subject: [PATCH 10/11] feat(ids): cross-file methods get ids under their declaring file A method declared in a file other than its receiver type now nests under that file's module: //. The JSON node stays in the receiver type's callables{}, and source_file is kept. The Neo4j projector adds GO_DECLARES from the declaring module to the method, beside GO_HAS_METHOD. 22 frontend methods change, e.g. handlers.go/frontendServer/logoutHandler (was main.go/frontendServer/logoutHandler). Refs codellm-devkit/codeanalyzer-go#46 --- internal/core/crossfile_method_test.go | 161 ++++++++++++++++++ internal/neo4j/crossfile_test.go | 54 ++++++ internal/neo4j/project.go | 6 + internal/syntactic_analysis/v2emit/emit.go | 8 +- .../syntactic_analysis/v2emit/sigindex.go | 17 +- 5 files changed, 240 insertions(+), 6 deletions(-) create mode 100644 internal/core/crossfile_method_test.go create mode 100644 internal/neo4j/crossfile_test.go diff --git a/internal/core/crossfile_method_test.go b/internal/core/crossfile_method_test.go new file mode 100644 index 0000000..aa3060a --- /dev/null +++ b/internal/core/crossfile_method_test.go @@ -0,0 +1,161 @@ +package core_test + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/core" + "github.com/codellm-devkit/codeanalyzer-go/internal/options" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis/v2emit" +) + +// crossFileProject: T is declared in a.go, its method M in b.go, a package-level +// function M in b.go too (same name, different id), and Use in c.go calls both. +func crossFileProject(t *testing.T) string { + t.Helper() + dir := t.TempDir() + files := map[string]string{ + "go.mod": "module example.com/xf\n\ngo 1.25.0\n", + "a.go": "package xf\n\ntype T struct{ n int }\n", + "b.go": "package xf\n\nfunc (t T) M() int { return t.n }\n\nfunc M() int { return 2 }\n", + "c.go": "package xf\n\nfunc Use() int {\n\tvar t T\n\treturn t.M() + M()\n}\n", + } + for rel, content := range files { + if err := os.WriteFile(filepath.Join(dir, rel), []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + return dir +} + +// analyzeL2 runs the real pipeline at -a 2 and returns the analysis.json tree as +// generic maps (the v2 Go structs are not needed for these checks). +func analyzeL2(t *testing.T, dir string) map[string]any { + t.Helper() + opts := options.AnalysisOptions{ + InputPath: dir, AppName: "xf", Level: options.LevelCallGraph, + SkipTests: true, CacheDir: t.TempDir(), OutputDir: t.TempDir(), + } + app, err := core.New(opts).Analyze() + if err != nil { + t.Fatal(err) + } + identity, err := v2.ResolveIdentity("xf", nil) + if err != nil { + t.Fatal(err) + } + payload := v2emit.Emit(app, identity, dir, int(opts.Level), "test", 0, nil, nil) + raw, err := json.Marshal(payload) + if err != nil { + t.Fatal(err) + } + var doc map[string]any + if err := json.Unmarshal(raw, &doc); err != nil { + t.Fatal(err) + } + return doc +} + +func asMap(v any) map[string]any { m, _ := v.(map[string]any); return m } + +// methodNode finds the method whose signature ends in suffix, under any type. +func methodNode(st map[string]any, suffix string) map[string]any { + for _, mod := range st { + for _, ty := range asMap(asMap(mod)["types"]) { + for _, c := range asMap(asMap(ty)["callables"]) { + if strings.HasSuffix(asMap(c)["signature"].(string), suffix) { + return asMap(c) + } + } + } + } + return nil +} + +func TestCrossFileMethod_IDNamesDeclaringFile(t *testing.T) { + doc := analyzeL2(t, crossFileProject(t)) + st := asMap(asMap(doc["application"])["symbol_table"]) + + m := methodNode(st, ".T.M") + if m == nil { + t.Fatal("method T.M not found") + } + const wantID = "can://xf/xf/go/b.go/T/M" + if got := m["id"]; got != wantID { + t.Errorf("method id = %v, want %s (declaring file b.go, not the receiver's a.go)", got, wantID) + } + if got := m["source_file"]; got != "b.go" { + t.Errorf("source_file = %v, want b.go", got) + } +} + +func TestCrossFileMethod_SameNameFunctionDistinct(t *testing.T) { + doc := analyzeL2(t, crossFileProject(t)) + st := asMap(asMap(doc["application"])["symbol_table"]) + fn := asMap(asMap(asMap(st["b.go"])["functions"])) + var fnID string + for _, f := range fn { + if asMap(f)["signature"] == "example.com/xf.M" { + fnID = asMap(f)["id"].(string) + } + } + const wantFn = "can://xf/xf/go/b.go/M" + if fnID != wantFn { + t.Errorf("function M id = %q, want %s", fnID, wantFn) + } + if m := methodNode(st, ".T.M"); m != nil && m["id"] == fnID { + t.Errorf("method and function named M collide at %s", fnID) + } +} + +// The call in Use (c.go) to the cross-file method resolves to the declaring-file +// id: the call-graph edge and the body call node both name b.go. +func TestCrossFileMethod_CallsResolveToDeclaringFile(t *testing.T) { + doc := analyzeL2(t, crossFileProject(t)) + app := asMap(doc["application"]) + const method = "can://xf/xf/go/b.go/T/M" + + var caller string + for _, mod := range asMap(app["symbol_table"]) { + for _, f := range asMap(asMap(mod)["functions"]) { + if asMap(f)["signature"] == "example.com/xf.Use" { + caller = asMap(f)["id"].(string) + } + } + } + if caller == "" { + t.Fatal("Use not found") + } + + edge := false + for _, e := range app["call_graph"].([]any) { + em := asMap(e) + if em["src"] == caller && em["dst"] == method { + edge = true + } + } + if !edge { + t.Errorf("no call_graph edge %s -> %s", caller, method) + } + + resolved := false + for _, mod := range asMap(app["symbol_table"]) { + for _, f := range asMap(asMap(mod)["functions"]) { + if asMap(f)["id"] != caller { + continue + } + for _, bn := range asMap(asMap(f)["body"]) { + if c, ok := asMap(bn)["callee"].(string); ok && c == method { + resolved = true + } + } + } + } + if !resolved { + t.Errorf("no body call node in Use has callee %s", method) + } +} diff --git a/internal/neo4j/crossfile_test.go b/internal/neo4j/crossfile_test.go new file mode 100644 index 0000000..1c7132d --- /dev/null +++ b/internal/neo4j/crossfile_test.go @@ -0,0 +1,54 @@ +package neo4j + +import ( + "testing" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// A method declared in b.go but whose receiver type lives in a.go: GO_HAS_METHOD +// from the type, and GO_DECLARES from the declaring module b.go. +func TestProject_CrossFileMethodDeclaredByItsModule(t *testing.T) { + const ( + app = "can://xf/xf" + typeMod = app + "/go/a.go" + declMod = app + "/go/b.go" + typeID = typeMod + "/T" + methodID = declMod + "/T/M" + ) + a := &v2.Analysis{ + IdentityVersion: 2, SchemaVersion: v2.SchemaVersion, Language: v2.Language, MaxLevel: 1, + Application: v2.Application{ + IdentityVersion: 2, ID: app, Kind: "application", + SymbolTable: map[string]v2.Module{ + "a.go": {ID: typeMod, Kind: "module", Types: map[string]v2.Type{ + "T": {ID: typeID, Kind: "struct", Signature: "example.com/xf.T", + Callables: map[string]v2.Callable{ + "example.com/xf.T.M": {ID: methodID, Kind: "method", Signature: "example.com/xf.T.M", SourceFile: "b.go"}, + }}, + }}, + "b.go": {ID: declMod, Kind: "module"}, + }, + }, + } + rows := Project(a) + + hasMethod, declares := false, false + for _, e := range rows.Edges { + if e.To.Value != methodID { + continue + } + switch e.Type { + case RelHasMethod: + hasMethod = e.From.Value == typeID + case RelDeclares: + declares = e.From.Label == LabelModule && e.From.Value == declMod + } + } + if !hasMethod { + t.Errorf("no GO_HAS_METHOD from receiver type %s to %s", typeID, methodID) + } + if !declares { + t.Errorf("no GO_DECLARES from declaring module %s to %s", declMod, methodID) + } +} diff --git a/internal/neo4j/project.go b/internal/neo4j/project.go index 782ee2b..8790153 100644 --- a/internal/neo4j/project.go +++ b/internal/neo4j/project.go @@ -195,6 +195,12 @@ func (p *projector) projectType(modRef NodeRef, module string, t v2.Type) { for _, m := range t.Callables { mRef := p.projectCallable(tRef, module, m) p.b.Edge(RelHasMethod, tRef, mRef, nil, "") + // A method declared in another file is also declared by that file's module, + // so "which file declares this method" is one query (spec v2-go-ids § Amendment). + if m.SourceFile != "" { + declaring := NodeRef{Label: LabelModule, KeyProp: "id", Value: v2.ModuleID(p.analysis.Application.ID, m.SourceFile)} + p.b.Edge(RelDeclares, declaring, mRef, nil, "") + } } } diff --git a/internal/syntactic_analysis/v2emit/emit.go b/internal/syntactic_analysis/v2emit/emit.go index 55e81a4..5268552 100644 --- a/internal/syntactic_analysis/v2emit/emit.go +++ b/internal/syntactic_analysis/v2emit/emit.go @@ -292,7 +292,7 @@ func emitModule(appID, modRel string, file schema.GoFile, idx sigIndex, srcs *so } for name, t := range file.Types { - mod.Types[name] = emitType(modID, modRel, t, idx, srcs, flow, l4) + mod.Types[name] = emitType(appID, modRel, t, idx, srcs, flow, l4) } for sig, fn := range file.Functions { mod.Functions[sig] = emitCallable(modID, modRel, modRel, li, fn, idx, srcs, flow, l4) @@ -304,9 +304,9 @@ func emitModule(appID, modRel string, file schema.GoFile, idx sigIndex, srcs *so // kind and splitting embedding (base_types) from computed satisfaction. Methods // resolve into the type's callables{}. modRel is the module the type lives in; // its methods may be declared in other files (handled in emitCallable). -func emitType(modID, modRel string, t schema.GoType, idx sigIndex, srcs *sourceCache, flow l3Overlay, l4 *l4Overlay) v2.Type { +func emitType(appID, modRel string, t schema.GoType, idx sigIndex, srcs *sourceCache, flow l3Overlay, l4 *l4Overlay) v2.Type { li := srcs.index(modRel) - typeID := v2.TypeID(modID, t.Signature) + typeID := v2.TypeID(v2.ModuleID(appID, modRel), t.Signature) out := v2.Type{ ID: typeID, Kind: typeKind(t), @@ -327,7 +327,7 @@ func emitType(modID, modRel string, t schema.GoType, idx sigIndex, srcs *sourceC } } for sig, m := range t.Methods { - out.Callables[sig] = emitCallable(typeID, modRel, modRel, li, m, idx, srcs, flow, l4) + out.Callables[sig] = emitCallable(methodParentID(appID, modRel, t, m), modRel, modRel, li, m, idx, srcs, flow, l4) } return out } diff --git a/internal/syntactic_analysis/v2emit/sigindex.go b/internal/syntactic_analysis/v2emit/sigindex.go index 00ed238..83a3f07 100644 --- a/internal/syntactic_analysis/v2emit/sigindex.go +++ b/internal/syntactic_analysis/v2emit/sigindex.go @@ -55,15 +55,28 @@ func buildSigIndex(appID string, symbolTable map[string]schema.GoFile) sigIndex idx.addCallable(modID, relPath, fn) } for _, t := range file.Types { - typeID := v2.TypeID(modID, t.Signature) for _, m := range t.Methods { - idx.addCallable(typeID, relPath, m) + idx.addCallable(methodParentID(appID, relPath, t, m), relPath, m) } } } return idx } +// methodParentID is the id a method nests under. A method declared in its +// receiver type's file nests under that type's id. A method declared in another +// file nests under the same type in the DECLARING file's module, so its id names +// where the method is written (spec v2-go-ids § Amendment). The JSON node itself +// stays in the receiver type's callables{}; only its id moves. modRel is the +// receiver's module (the file the type lives in). +func methodParentID(appID, modRel string, t schema.GoType, c schema.GoCallable) string { + mod := modRel + if c.Path != "" { + mod = c.Path + } + return v2.TypeID(v2.ModuleID(appID, mod), t.Signature) +} + // addCallable records one callable and recurses into its closures, matching the // id nesting emitCallable produces. relPath is the module file this callable is // reached through; it keys the disambiguating (signature,path) entry. From 19c862a6517a82d2667528fc565917ecaa441768 Mon Sep 17 00:00:00 2001 From: lambertpw Date: Fri, 20 Nov 2026 10:53:17 -0500 Subject: [PATCH 11/11] fix(neo4j): purge every changed module before upserting any An eager push purged and upserted module by module. A node is owned by one module but can carry another module's id prefix (a method declared in rpc.go, owned by main.go), so purging rpc.go after main.go had upserted it deleted the method. All purges now run before any upsert. Refs codellm-devkit/codeanalyzer-go#46 --- internal/neo4j/bolt.go | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/internal/neo4j/bolt.go b/internal/neo4j/bolt.go index da5b001..8a96fa4 100644 --- a/internal/neo4j/bolt.go +++ b/internal/neo4j/bolt.go @@ -129,6 +129,20 @@ func (w *boltWriter) run() error { } } + // Purge every changed module before upserting any of them. A module's purge is + // by id prefix, and a node a module OWNS may carry another module's prefix (a + // method declared in rpc.go but owned by main.go's receiver). Purging rpc.go + // after main.go had upserted that method would delete it, so all purges run first. + if w.cfg.Eager && scoped { + for _, m := range sortedModuleKeys(byModule) { + if !changed[m] { + continue + } + if err := w.purgeModule(m); err != nil { + return err + } + } + } // Upsert shared nodes (MERGE-only, every run) then changed-module nodes. if err := w.upsertNodes(shared); err != nil { return err @@ -137,11 +151,6 @@ func (w *boltWriter) run() error { if !changed[m] { continue } - if w.cfg.Eager && scoped { - if err := w.purgeModule(m); err != nil { - return err - } - } if err := w.upsertNodes(byModule[m]); err != nil { return err }