Repository navigation
wip: M2 backend - #1
Merged
Merged
Conversation
Each area appends to its own sources_<area>.cmake so parallel M2 work does not edit the same list.
Interrupted M2 work from six areas (pk and pq crypto, SSH transport, auth, connect, apps backend) plus the finished interop fixtures. Builds warning-free on x64-modern; 31/34 tests pass. State per area and next steps are in docs/dev/handoff-backend.md.
Covers the backend parts of M1-M7, the boundary with the frontend track and the contracts both depend on.
- tests/vectors/refimpl/rsa_keys.txt was never generated from pk_ref.py (seed 8017); rsa_reference_keys, cng_diff_rsa and bench_pk_throughput failed on the missing file. Regenerated with the documented command. - ssh_conn_config_validation still expected SD_E_UNSUPPORTED for jump chains, proxies and agent identities; update it to the implemented M2 contract (SD_OK with cleanup, INVALID_PARAM for a proxy without a host and for agent identities without fingerprint or agent source). - Auth tests written before cert auto-discovery (<file>-cert.pub, 01 6.2) and the RSA retry ladder (01 4.5): ssh_auth_rsa_algorithms now walks the certificate first (rsa-sha2-512/256, ssh-rsa with -cert-v01 names) before the plain key, and ssh_auth_agent_keys expects the rsa-sha2-256 retry of a refused signature before the next identity. - Prompts are raised synchronously while the fake server delivers a message (the test thread plays the I/O thread); pump() after the srv_* call waited for a second prompt. Add pump_prompts(f, n) and use it for the password-change, kbdint and passphrase flows. - ssh_keyfmt_truncation_and_corruption: p8_ed25519 stores only the seed, so a corrupted seed legitimately loads as a different key; only assert key stability for formats that store the public part. - The fake pipe agent closed its only listener before creating the next instance, so a reconnecting client raced ERROR_FILE_NOT_FOUND or ERROR_PIPE_BUSY; create the next listener before serving and re-check the stop flag before recreating (ERROR_NO_DATA teardown path).
sd-test-ssh-interop-m2 consumes the run-matrix.ps1 row recipe (SD_IT_*): - interop.ssh_matrix_connect (srv_all): connect + exec over the generic recipe - direct servers, 2/3-hop jumps, HTTP CONNECT (plain and Basic) and SOCKS5 (plain and RFC 1929) proxies, user certificates + host certificates on cert, TOTP keyboard-interactive on otp and the demo row (SOCKS5-auth -> OTP bastion -> target, PPK v3, ML-KEM). Asserts the negotiated KEX, the legacy mask, strict KEX and that the fixture known_hosts decides every hop without prompts. - interop.ssh_matrix_kex (server rows): walks KEX x host-key with disabled lists (ML-KEM, curve25519, ECDH, rsa-sha2-512; the legacy DH/GEX set; 3des-cbc; the host-cert host-key algorithm). - interop.ssh_matrix_pq (ossh10pq): no ML-KEM offer -> SD_E_KEX_FAILED. - interop.ssh_matrix_update (rotate): hostkeys-00 announcement, apply, rotation to B only, reconnect on the learned key. TOTP is computed in-process from SD_IT_OTP_SECRET (RFC 6238, sd_hmac SHA-1); the secret is never logged. run-matrix.ps1 no longer aborts at the first failing row (PS 5.1 native-stderr quirk, same guard as ci-local.ps1). Matrix green on x64-modern with every fixture up except squid/dante.
- interop.ssh_matrix_auth_keys (ossh10): the M2 fixture keys in every format authenticate - OpenSSH v1 plain and encrypted, PPK v3/v2, and the puttygen-written encrypted PPK v3 Argon2id / PPK v2 AES with the fixture passphrase (the PPK cross-check the notes called for; our stdlib fixtures stay for the unit tests). - interop.ssh_matrix_auth_certs (cert): valid user certificates for ed25519, RSA and ECDSA authenticate through TrustedUserCAKeys; expired, wrong-principal and revoked (KRL) certificates are rejected with auth_failed. mx_connect gained key/cert overrides for these tests. The modern row only trusts the M1 keys, so auth_keys runs on ossh10 only.
sd_sftp_options.connections (1..4) opens secondary members after the
primary is READY when authentication was non-interactive (2FA rule):
members are their own sd_ssh_conn with SD_SSH_CFG_POOL_MEMBER and
prompt-cancelling callbacks, connected one at a time; each opens one
bulk SFTP channel and runs INIT/VERSION + REALPATH(".") over the
length-framed wire format. The admit/drop decision runs on the primary
I/O thread and compares the member's host-key fingerprint and
realpath(".") against the primary's; a mismatch drops the member and
stops admissions for the generation. EV_POOL events,
server_info.connections and the conn[1..3] part of session_stats()
report the pool. Striping, per-member loss semantics and the admission
stagger are the next increments; the header states the scope.
Tests against the modern fixture: m2_pool_admits_members (3 admitted)
and m2_pool_interactive_stays_at_one (passphrase prompt keeps it at 1).
- The XP import floor lists lacked the four APIs the M2 live known-hosts sources pull (RegQueryInfoKeyW, GetSecurityDescriptorOwner, GetVolumePathNameW) plus InitializeCriticalSection; all documented as present on XP SP2 x64 / SP3 x86. Added to both bootstrap lists. - InterlockedAdd64 does not exist in the 32-bit CRT intrinsics: the pool member counters use the portable sd_atomic_* helpers. - sd-test-ssh_auth used GetTickCount64 (Vista+) in its pump loops, which broke the x86 flavours: switch to sd_tick_ms(). - test_interop_m2: non-constant aggregate initializers (C4204) became errors under the x86 warning level; assign the response arrays. All five local presets build warning-free again; x64-modern Release unit suite green.
The analysis preset had not been run since the M2 area work landed; it now builds and its unit suite passes. Findings fixed (library code): - bn.c montmul: initialize the accumulator with memset so the analyzer tracks it (the zeroing loop lost its dataflow through the mac/adc out-parameters). - profile_m2.c host-spec parse: initialize the host pointer at declaration and add _Analysis_assume_ where the analyzer's colon aliasing is spurious. - secret.c list_add: typed allocation and a per-element copy so the analyzer's size model for names stays per entry; capacity invariant asserted. Test code: annotate tu_hex (_Outptr_result_buffer_(return), matched at the definition), fix genuinely ignored returns (ReadFile, tu_hex), direct tu_hex calls where the hb() struct return loses the SAL state, static for the big harness structs (sd_profile locals, h2, fake conns), a fixed 16-byte tag buffer in the wycheproof GCM test, bounds checks for the fake pipe agent listener, a length-bounded environment scan in test_interop_m2, and an 8 KB packet buffer in the auth flow harness (auth messages are a few KB; the 70 KB embedded buffer blew every test frame past the analysis stack budget). ci-local.ps1 excludes the 'matrix' label: the srv_* interop tests need the per-row WSL fixtures and are run by tests/interop/run-matrix.ps1.
Records what this session completed (unit fixes, the matrix interop set, auth interop, pool increment 1 with the 2FA rule, the flavour and analysis repairs), what remains (striping/barriers/member loss with the settled NCH/stream design, the vfs-side policy, system-proxy tests, sd_mount + CLI), the environment state (fixtures running, puttygen fixtures present, run-row.sh helper), and that ci-local -Clean must be rerun as the first action of the next session.
Two real defects: - A striped request whose member sub-open failed (a server STATUS error, or start_sub_open running out of memory) was parked forever on sub_state != SST_OPEN, so every unit routed to that member stalled until its deadline instead of falling back to the primary. stripe_unit_member() now resolves the member per unit: SST_FAILED (or an OPEN that could not be created) routes the unit to the primary at once, SST_OPENING still waits, and the next barrier recomputes the table without the refused member. - flush() treated a write error on a member channel as session-fatal. In the window between the member thread marking the stream dropped and the posted member_stream_closed running, the session still shows the channel SS_OPEN, so one lost member turned into a full disconnect. Member channels now leave their bytes queued for the gen-guarded drop report; only channels 0 and 1 lose the session. Tests: a refused sub-open now falls back (and the route table drops the member at the barrier); striped WRITE routing and byte placement through each member's sub-handle is pinned via the fake's WRITE mirror; the three emit_piece limits park the pump when a member's accept window is shut (a bounded byte budget on the fake's member stream) and the write resumes exactly once when it reopens; a member dying with its fsync barrier wire in flight counts that wire as done; a write failing on the member's stream inside the drop window keeps the session READY; and the member-loss write test now answers the surviving channels' held wires, pinning the late-ack discard path of fbf8b17. The fake transport gained fail_member_writes and limit_member_writes for the two drop-window and window scenarios.
A striped WRITE whose pieces settled with a server FAILURE queued the PH_PR_STATVFS refinement probe into Q_STRIPE, but pump_striped_req keyed on r->op alone: the OP_WRITE took the data path regardless of the phase and parked once emit_off reached chunk_cutoff, so the probe never went out and the request stalled until its deadline, completing SD_E_TIMEOUT instead of the refined result (a plain write emits the probe through schedule_chan). pump_striped_req now checks the phase first: a data op that is not in PH_MAIN is emitted through emit() on the handle's own channel, under the same channel guards as the other wires. The fake server learned a canned statvfs@openssh.com answer (bavail selectable) and per-stream WRITE failures. The new test pins the refined SD_E_NO_SPACE for a full filesystem, the unconverted SD_E_FAILURE when space is available, and that SD_SFTP_REQ_NO_REFINE skips the probe; the trace shows the single probe on the handle's channel.
…11 menu, Network locations, logs) diag --check gains read-only rows through the diag_env seam: - Dokany version from the dokan2.dll version resource, and the dokany-deny lists of the cached manifests (per user and per machine); a deny-listed version is a warning with the reason. - Windows 11 menu package registration (per-user AppModel repository). - Network location entries compared with the user-scope profiles that want one (out of sync is a warning). - Setup log and service log file counts. The support bundle adds service-side logs from the machine logs folder and the newest setup log (read only, anonymized like logs/). Pure rules (versions, deny parsing, token comparison) are in cli_diag_rules.c with unit tests. Strings are in all six catalogs. Port checks and a clock skew measurement are not in the spec; the notes say so.
# Conflicts: # tests/apps/CMakeLists.txt
The x64 setup payloads now carry the x86-sse2 shell extension in x86\ beside the x64 files (04 6.2). The per-user setup writes its shell rows in the view of the installed flavour, and registers the x86 copy in the 32-bit view only where HKCU\Software\Classes is split by view (x64 XP and Server 2003). Later systems share one HKCU tree, so the 64-bit CLSIDs are kept. Uninstall takes the view from the installed flavour and removes empty keys with RegDeleteKeyExW resolved at run time. Registration rows carry a view (SD_SETUP_REGVIEW_*); the SHELL32 rows are skipped without an x86 install folder. Tests cover the 32-bit rows through the seam, an x64 install and uninstall with the x86 copy, and the gate. The packaged copy gets the importcheck and isacheck rules of the MSI copy.
…story, host key extras
- hostkey_source_insecure is a notice of its own (switch notify_hostkeys_source): it names the
ignored OpenSSH or PuTTY source, is always recorded in the event list and opens Settings on
the Security page when clicked.
- mount.state reason shows in the reconnected balloon ("reconnected after: ...") and in the
connecting and authenticating entries of the event list; known codes have localized text,
unknown codes are shown as they are. A mounted event with a reason is a reconnect.
- The event list is filled from events.recent when the agent has the feature, at the time each
event happened, skipping what the UI already saw live (ui_evhist_model).
- The host key manager details show the MD5 fingerprint, source, added date and comment of the
known_hosts line.
- 19 new strings in six locales; tests in tests/ui and --test-windows checks.
The CLI now manages the Network location entries of the profiles without an agent, as 03 section 6.1 and 8.7 specify. The entry code in explorer_data.c is shared by the agent and the CLI. - add and remove set mount.network_location through the profile save (config.lock) and sync the one entry; repair holds config.lock while it reconciles all entries. - All three refuse under policy DisableNetworkLocations and change nothing. add and remove refuse scope = service profiles. - Only folders with our [ShellDrive] profile= marker are deleted. - The CLI leaves the target of an existing entry alone; the agent re-points mounted entries on its next profile change or start. - Tests: parser test cli_parse_location_verbs and smoke test smoke_location_verbs (temporary NetHood via SHELLDRIVE_TEST_NETHOOD). - Strings added to all six CLI catalogs.
The service's Event Log messages are user-visible, so they are now localized into en, es (voseo), fr, pt-BR, it and de. The svc PO catalog carries the ten event entries with an sdl10n:event comment giving the message id, severity and symbolic name. sdl10n gains --out-mc, which writes the Event Log table as UTF-16LE with one Language= block per language and id. The build compiles it with mc.exe -u into one table per language (LCIDs 0x409, 0x2C0A, 0x40C, 0x416, 0x410, 0x407). The generated table replaces the hand-written svc_events.mc. Event entries must exist in every locale even with --allow-incomplete, and translations must keep the %N placeholders of en in the same order. Tests cover the new catalog rules (sdl10n fixtures), the generated table (six languages per id) and FormatMessage resolution of every id in every language against the built shelldrive-host.exe.
A lookup of a name the vfs has no entry for fails with SD_VFS_E_DISCONNECTED without waiting offline_wait. Explorer, Office and the WebDAV redirector probe many names that do not exist, so a wait per probe would stall the shell for the whole outage. Known entries keep the wait-free behaviour. 02 section 5.2 and 7.6 state the rule and its reason; the open notes in m1-notes and handoff-backend are closed. Unit test vfs_offline_unknown_lookup_fails_at_once pins the behaviour with a 5 s offline_wait and a 1 s bound.
04 section 12.3 defines the release gates record: a line-oriented ASCII file (LF or CRLF) with a release-gates 1 header, version, commit, channel, one gate line per required gate (pass, fail or waived), and a final approved line. A per-channel table says which gates are required on beta and stable and which waivers are allowed (lab, beta only). sd-release gates check FILE --version V --commit C validates the record: exit 2 for a malformed record, 3 for a refusal (missing or failed gate, waiver not allowed, wrong version or commit, not approved). sign now requires --gates FILE --version V --commit C for every kind and refuses unless the check passes; a manifest or payload body must name the record's channel, and a manifest its version. release.yml runs the check in job check-gates before the signing sessions, with no secrets. tests/release/gates_flow.cmake covers valid, refused and malformed records and the sign refusals; release_flow.cmake signs with a test record.
FormatMessage and the Event Viewer match the exact LANGID, so a single es-AR block left es-ES, es-MX and other Spanish LANGIDs in English. sdl10n --out-mc now writes each language's one text under every sublanguage LANGID that winnt.h defines for its primary language, plus the primary-neutral LANGID: 58 LanguageNames entries (English 17, Spanish 22, French 7, Portuguese 3, Italian 3, German 6). This mirrors the UI, which selects the text by primary language. mc.exe names the .bin files MSG<LCID>, so the CMake outputs no longer list them. unit.svc parses the LanguageNames list from the generated .mc, checks that the required LANGIDs are present, and resolves every id in every LANGID through FormatMessage on the built shelldrive-host.exe.
…nglish; add CODEOWNERS (@LibreStatic)
Windows 11 needs generation 2 with a virtual TPM, and 04 section 11.4 asks for Server 2022 with Secure Boot. New-LabVM.ps1 reads Generation (default 1) and Tpm per VM: Secure Boot on with the MicrosoftWindows template, local key protector plus Enable-VMTPM when asked, SCSI DVD and synthetic NIC on generation 2, DVD-first boot order for fresh installs on both generations. Generation 1 behavior for XP/7 is unchanged.
Initial lab-results entry: generation-2 harness commit, all six first-batch VMs created and verified (still empty disks, no runs yet), and the maintainer steps that unblock the smoke and security runs.
QEMU 11.1.0 present, empty P3 qcow2 created, payload ISO path validated against the current build tree. No guest OS yet, so no runs; added the maintainer install checklist to lab-results.md.
Win7 x86/x64 installed with bases and image backups. SdLabSmoke task fires nothing (fix staged, needs reinstall); Win10/11/Server never start setup (ISOs verified, cause open). Detail in host-only D:\sdlab\handoff-lab.md.
New-LabVM.ps1 enabled CompatibilityForOlderOperatingSystemsEnabled on every generation 1 VM, but only pre-Windows 7 guests (XP) need the reduced CPUID set. The setting is now off unless the VM sets CompatOlderOs = $true (generation 1 only); the example config sets it for XP alone.
CI failed repro.x64-modern and repro.x86-ia32 (the reproducible-build checks carry the repro label, which ci-local excludes). Two causes: - sd-test-svc and sd-test-update compiled absolute fixture paths ($<TARGET_FILE:...> compile definitions), so the binaries depended on the build directory. They now compile only file names, resolved at run time next to the test executable by the new sd_test_exe_dir_path; the generated svc_events.mc is addressed by a CMake-computed relative path (04 12.2: no absolute build paths in outputs). - the payload CAB stored each member's build-clock file time, and its hash travels embedded in shelldrive-setup.exe. Members are staged first with a fixed epoch (SD_PAYLOAD_EPOCH, SOURCE_DATE_EPOCH honored) by tools/sd-cab-stage.ps1, per 04 12.2. Verified: check-repro x64-modern (77 files) and x86-ia32 (82 files, including shelldrive-setup.exe) byte-identical across two builds; unit.svc and unit.update pass with the run-time resolution.
unit.webdav and unit.ssh_auth fail on the windows-2022 runner while passing locally, and a failing Test step skips the artifact upload, so the run shows nothing but Failed. --output-on-failure puts the failing tests' output in the job log.
The windows-2022 runners differ from a dev machine in three ways that broke otherwise green tests; each fix keeps the product semantics. - unit.ssh_auth: the fake agent pipe was created with default security, and on the runners (elevated administrator) Windows owns such objects as the Administrators GROUP, which the client's user-pipe owner check (agent_pipe.c verify_pipe, TM-R08) rightly refuses: every agent connect returned access_denied, and the auth flow then crashed (0xc0000005) on the denied-agent path. The fake now sets the pipe owner to its token user, like a real SSH_AUTH_SOCK pipe; the product check is unchanged. - webdav_token_only_selftest resolved localhost with WinHTTP, which prefers ::1 on the runners while the test server listens on IPv4 loopback; the probe now targets 127.0.0.1 (it never leaves the host). - repro.x86-ia32 ran inside every tree, but the x64-modern job skips the v141_xp toolset setup, so its configure failed with MSB8020; a repro check now registers only for its own tree, shipping flavours only. check-repro.ps1 hashes with the SHA256 .NET class: Get-FileHash does not resolve in the ctest child environment on the runners. Verified locally: check-repro x86-ia32 (82 files byte-identical), unit.ssh_auth/unit.webdav pass, ci-local green on all five presets.
The self-test fails on the CI runners in about five seconds with a generic error, which reads as one 5 s WinHTTP timeout, but the failing step and the Win32 error were not observable: the product log has no unit-test sink. Each probe step now records its step number and GetLastError, a test seam (wd_selftest_last_diag) hands them to the unit test, and the test prints them on failure. Timeouts go from 5 s to 15 s so a slow runner is not misread as a hang.
… levels The WinHTTP autologon policy turned out not to be portable in either direction: level PROXY_ONLY (3) is newer than Server 2022 winhttp, which rejects it with ERROR_INVALID_PARAMETER (the CI failure), and LEVEL_HIGH still negotiated with the default credentials on Windows 11 26100 - the request ring added to the test server caught the plain path answering 207 after a Negotiate retry with authed=1. The probe now disables the request authentication machinery outright (WINHTTP_OPTION_DISABLE_FEATURE with WINHTTP_DISABLE_AUTHENTICATION, present since WinHTTP 5.1): no credential is ever attached, the server 401 comes back verbatim, and TM-R19 holds by construction. The diagnostic scaffolding stays, opt-in: the probe records its step and Win32 error, the server keeps a request ring (method, target, status, auth state; never enabled outside tests - production never records targets, they carry the mount token), and the test prints both on failure. This is what localized the bug in one run each.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Milestone-2 backend work, integrated from the r7/r8/r21 sub-branches plus lab bring-up. Headline areas (232 commits; more inside):
l10n/svc).release-gates.txtdefined and enforced bysd-release sign.tests/lab,docs/dev/lab-results.md): generation-2 VM support with Secure Boot and vTPM; per-VMCompatOlderOsopt-in; recorded VM creation, the host C-state crash incident and its BIOS fix, the P1 smoke fix (schtasksXML registration), the P2 root cause (host virtual-DVD reads fail beyond 4 GiB) and the Win10 LTSC split-media install with smoke PASS.Reviewer notes
tools\ci-local.ps1green on all five presets before push (expected WOW64 dokan skips on the x86 flavours).-IncludeTestson fresh lab bases (Win7 x86, Win10 LTSC) shares a failure pattern (config/crypto/ipc/ssh/svc/update/webdav exit 1, shellext 3) recorded indocs/dev/lab-results.mdas an open triage item - environment gaps vs genuine bugs not yet determined.wip/r7-arm32,wip/r8-tokdav,wip/r21-evloc,wip/r21-gates.