0.5.0 (4/5): telemetry switches, dg init, JSON output and listen output formats

conftest.py

@@ -77,6 +77,11 @@ def _disable_telemetry(monkeypatch):
     """
     monkeypatch.setenv("DEEPCTL_TELEMETRY_DISABLED", "1")
     monkeypatch.setenv("DEEPCTL_TELEMETRY_DSN", "http://k@127.0.0.1:9/1")
+    # A developer's DO_NOT_TRACK or DEEPCTL_TELEMETRY=log must not change
+    # what telemetry's own tests see. (CI is cleared only by those tests:
+    # other suites read it as an automation signal.)
+    monkeypatch.delenv("DO_NOT_TRACK", raising=False)
+    monkeypatch.delenv("DEEPCTL_TELEMETRY", raising=False)
 
 
 @pytest.fixture(autouse=True)

docs/contracts/help-cache.md

@@ -77,7 +77,7 @@ The launcher serves an entry only when every one of the following holds. Anythin
 4. **Each name in `env`** has the same value in the launcher's environment, and `null` means the variable must be unset. On Windows, look the names up case-insensitively, as the OS does. A variable that isn't listed doesn't matter. `DEEPCTL_PROG_NAME` and `DEEPCTL_HELP_CACHE` are never listed: the launcher sets them, and `prog` already records the name.
 5. **`text_sha256`** (lowercase hex) equals the SHA-256 of the entry bytes the launcher read. This binds the text to its key: Python writes the text first and the key second, so text from another run under this key is a miss and is never served.
 
-A bundle install (`-I`, no user site, `HOME` is the real home) lists about 10 files and 18 environment variables. Python writes no key over the limits (32 files, 64 variables), so an install with a long `sys.path` (an editable install, say) gets no entry and always takes the Python path.
+A bundle install (`-I`, no user site, `HOME` is the real home) lists about 10 files and 21 environment variables. Python writes no key over the limits (32 files, 64 variables), so an install with a long `sys.path` (an editable install, say) gets no entry and always takes the Python path.
 
 **Paths and values that aren't valid Unicode.**
 - **Unix:** a path or environment value with bytes that don't decode under the filesystem encoding reaches Python as a lone surrogate (`\udcXX`).

docs/telemetry.md

@@ -0,0 +1,64 @@
+# Telemetry
+
+dg sends one error report when a command fails with an unexpected error. Nothing is sent when a command succeeds, for a usage error or another error dg reports to you, or for `--help` and `--version`. The README's Telemetry section says where reports go.
+
+## When it runs
+
+- **At most one report per run**, and only for an unexpected error.
+- **No network or reporting code on the success path.** The reporting library is loaded only after an unexpected error, so a command that succeeds opens no connection for telemetry.
+- **A short time budget.** dg waits about 0.6 s at most for a report, loading included, and doesn't wait at all when no report was queued. With blocked or slow egress, a failing command takes at most that much longer.
+
+## Turn it off
+
+Any one of these turns telemetry off. They are checked in this order:
+
+| Switch | Scope |
+|---|---|
+| `--no-telemetry` | One command |
+| `DEEPCTL_TELEMETRY_DISABLED=1` | A shell session, a CI job |
+| `DEEPCTL_TELEMETRY=off` | A shell session, a CI job |
+| `DO_NOT_TRACK=1` | Every tool that follows the [Console Do Not Track](https://consoledonottrack.com/) convention |
+| `CI` set (as CI services do) | Any CI job |
+| `telemetry.enabled: false` in your `config.yaml` | Always |
+
+For `DEEPCTL_TELEMETRY_DISABLED`, `DO_NOT_TRACK` and `CI`, any value except an empty one, `0`, `false`, `no` or `off` turns telemetry off. `DEEPCTL_TELEMETRY` turns it off when set to `off`, `0`, `false`, `no` or `disabled`; `log` is described below, and any other value changes nothing. A config file dg won't load (unreadable, too large, not valid YAML, a repeated `telemetry` key, or one dg doesn't trust in an AI agent session) counts as turned off too. A project's `deepgram.yaml` can't change these settings. The last line of `dg --help` shows whether telemetry is on.
+
+## See what would be sent
+
+Run a command with `DEEPCTL_TELEMETRY=log`. If it fails with an unexpected error and telemetry is on, dg prints the report it would have sent to stderr, as one JSON line, and sends nothing.
+
+```sh
+DEEPCTL_TELEMETRY=log dg listen recording.wav
+```
+
+## Field schema
+
+A report holds these fields and nothing else. A value that doesn't match its expected shape (an identifier, a version string, a command path) is dropped, never shortened or escaped. Names from code outside dg and the libraries it calls, such as a plugin's, are sent as `<other>`.
+
+<!-- fields:start -->
+| Field | Contents |
+|---|---|
+| `event_id` | Random report ID (32 hex characters) |
+| `timestamp` | When the error happened |
+| `platform` | Always `python` |
+| `level` | Always `error` |
+| `release` | `deepctl@` and the dg version |
+| `environment` | Always `production` |
+| `sdk.name` | Name of the reporting library |
+| `sdk.version` | Version of the reporting library |
+| `tags.cmd.path` | Command path, such as `dg listen` |
+| `tags.cmd.flags` | Names of the flags used, never their values, such as `--model,--output` |
+| `tags.cli.os` | Operating system, such as `Darwin` |
+| `tags.cli.arch` | CPU architecture, such as `arm64` |
+| `tags.cli.python` | Python version, such as `3.12.4` |
+| `exception.values[].type` | Exception type, and the type of each exception that led to it |
+| `exception.values[].mechanism.type` | How the error was caught, such as `generic` |
+| `exception.values[].mechanism.handled` | Whether dg caught it |
+| `exception.values[].stacktrace.frames[].module` | Module of each stack frame |
+| `exception.values[].stacktrace.frames[].function` | Function of each stack frame |
+| `exception.values[].stacktrace.frames[].lineno` | Line number of each stack frame |
+<!-- fields:end -->
+
+A report never holds flag values or arguments, file names or paths, environment variables, API keys, request or response data, error messages, log output, variable values, source code, your hostname or your username.
+
+A test (`packages/deepctl-telemetry/tests/unit/test_contract.py`) sends a report through the real error path and checks that its fields are exactly the ones in this table.

packages/deepctl-cmd-init/src/deepctl_cmd_init/clones.py

@@ -0,0 +1,192 @@
+"""dg init's record of the clones it made, in dg's own state directory.
+
+A rerun of ``dg init`` resumes in a directory only when this record names
+it: nothing inside the directory (which anyone could have packed into an
+archive) is trusted for that. Each record is one owner-only file under
+``{state_dir}/init-clones/`` (0700), named after the directory's real path,
+holding that path, the template, its repository, the commit dg cloned and
+the phases done, and the clone directory's device and inode, so a
+directory deleted and made again at the same path is not taken for it.
+"""
+
+from __future__ import annotations
+
+import errno
+import hashlib
+import json
+import os
+import re
+import secrets
+import stat
+import sys
+from dataclasses import asdict, dataclass
+from typing import Any
+
+from deepctl_core.login_state import StateDirError, _open_dir, state_dir
+
+DIR_NAME = "init-clones"
+_MAX_BYTES = 16 * 1024
+_COMMIT = re.compile(r"[0-9a-f]{40}|[0-9a-f]{64}")
+_NOFOLLOW = getattr(os, "O_NOFOLLOW", 0)
+_NONBLOCK = getattr(os, "O_NONBLOCK", 0)
+_DIRECTORY = getattr(os, "O_DIRECTORY", 0)
+_BINARY = getattr(os, "O_BINARY", 0)
+
+
+@dataclass(frozen=True)
+class CloneRecord:
+    path: str
+    template: str
+    url: str
+    commit: str
+    phases: list[str]
+    dev: int
+    ino: int
+
+
+def _name(real_path: str) -> str:
+    return hashlib.sha256(real_path.encode("utf-8", "surrogateescape")).hexdigest()
+
+
+def _use_dir_fd() -> bool:
+    return sys.platform != "win32" and os.open in os.supports_dir_fd
+
+
+def _open_records(*, create: bool) -> int:
+    """Open the records directory inside the state directory, both checked:
+    real directories owned by this user, the records directory 0700."""
+    parent = _open_dir(state_dir(), create=create)
+    try:
+        if create:
+            try:
+                if _use_dir_fd():
+                    os.mkdir(DIR_NAME, 0o700, dir_fd=parent)
+                else:
+                    os.mkdir(state_dir() / DIR_NAME, 0o700)
+            except FileExistsError:
+                pass
+        flags = os.O_RDONLY | _DIRECTORY | _NOFOLLOW
+        if _use_dir_fd():
+            fd = os.open(DIR_NAME, flags, dir_fd=parent)
+        else:
+            fd = os.open(state_dir() / DIR_NAME, flags)
+    finally:
+        os.close(parent)
+    try:
+        st = os.fstat(fd)
+        if not stat.S_ISDIR(st.st_mode):
+            raise StateDirError(errno.ENOTDIR, "not a directory", DIR_NAME)
+        if sys.platform != "win32":
+            if st.st_uid != os.geteuid():
+                raise StateDirError(errno.EPERM, "owned by another user", DIR_NAME)
+            if stat.S_IMODE(st.st_mode) & 0o077:
+                if not create:
+                    raise StateDirError(errno.EPERM, "not private", DIR_NAME)
+                os.fchmod(fd, 0o700)
+    except BaseException:
+        os.close(fd)
+        raise
+    return fd
+
+
+def _valid(data: Any, real_path: str) -> CloneRecord | None:
+    if not isinstance(data, dict) or data.get("path") != real_path:
+        return None
+    template, url, commit = data.get("template"), data.get("url"), data.get("commit")
+    phases = data.get("phases")
+    if not (isinstance(template, str) and isinstance(url, str)):
+        return None
+    if not isinstance(commit, str) or not _COMMIT.fullmatch(commit):
+        return None
+    if not isinstance(phases, list) or not all(isinstance(p, str) for p in phases):
+        return None
+    dev, ino = data.get("dev"), data.get("ino")
+    if not isinstance(dev, int) or isinstance(dev, bool):
+        return None
+    if not isinstance(ino, int) or isinstance(ino, bool):
+        return None
+    return CloneRecord(real_path, template, url, commit, list(phases), dev, ino)
+
+
+def load(real_path: str) -> CloneRecord | None:
+    """The record for ``real_path``, or None when there is no usable one.
+
+    Only an owner-only regular file with one link, opened without following
+    a link or blocking on a FIFO, in the checked records directory.
+    """
+    try:
+        dir_fd = _open_records(create=False)
+    except OSError:
+        return None
+    name = _name(real_path) + ".json"
+    try:
+        flags = os.O_RDONLY | _NOFOLLOW | _NONBLOCK | _BINARY
+        if _use_dir_fd():
+            fd = os.open(name, flags, dir_fd=dir_fd)
+        else:
+            fd = os.open(state_dir() / DIR_NAME / name, flags)
+    except OSError:
+        os.close(dir_fd)
+        return None
+    try:
+        st = os.fstat(fd)
+        if not stat.S_ISREG(st.st_mode) or st.st_size > _MAX_BYTES:
+            return None
+        if sys.platform != "win32" and (
+            st.st_uid != os.geteuid()
+            or stat.S_IMODE(st.st_mode) & 0o077
+            or st.st_nlink != 1
+        ):
+            return None
+        raw = os.read(fd, _MAX_BYTES + 1)
+    except OSError:
+        return None
+    finally:
+        os.close(fd)
+        os.close(dir_fd)
+    try:
+        return _valid(json.loads(raw.decode("utf-8")), real_path)
+    except ValueError:
+        return None
+
+
+def save(record: CloneRecord) -> None:
+    """Write ``record`` atomically: a new 0600 file (``O_CREAT|O_EXCL|
+    O_NOFOLLOW``) next to the target, then renamed over it. Raises OSError
+    (StateDirError for a state directory dg can't use)."""
+    data = (json.dumps(asdict(record), indent=2) + "\n").encode("utf-8")
+    name = _name(record.path) + ".json"
+    tmp = f".{name}.{secrets.token_hex(8)}.tmp"
+    dir_fd = _open_records(create=True)
+    base = state_dir() / DIR_NAME
+    try:
+        flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL | _NOFOLLOW | _BINARY
+        if _use_dir_fd():
+            fd = os.open(tmp, flags, 0o600, dir_fd=dir_fd)
+        else:
+            fd = os.open(base / tmp, flags, 0o600)
+        try:
+            try:
+                if hasattr(os, "fchmod"):
+                    os.fchmod(fd, 0o600)
+                view = memoryview(data)
+                while view:
+                    view = view[os.write(fd, view) :]
+                os.fsync(fd)
+            finally:
+                os.close(fd)
+            if _use_dir_fd():
+                os.replace(tmp, name, src_dir_fd=dir_fd, dst_dir_fd=dir_fd)
+            else:
+                os.replace(base / tmp, base / name)
+        except BaseException:
+            try:
+                if _use_dir_fd():
+                    os.unlink(tmp, dir_fd=dir_fd)
+                else:
+                    os.unlink(base / tmp)
+            except OSError:
+                pass
+            raise
+    finally:
+        os.close(dir_fd)