Repository navigation
Feat: credential and auth model (tokens, delagation) #30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Alexander-Auer
wants to merge
73
commits into
DIRACGrid:main
Choose a base branch
from
Alexander-Auer:stack/credential-models
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+930
−58
Open
Changes from all commits
Commits
Show all changes
73 commits
Select commit
Hold shift + click to select a range
ca745e4
feat: Models the core contract needs
Alexander-Auer da8fb42
Feat: Protocols and Exceptions added for the core contract
Alexander-Auer 58a5886
Feat: Private protocols for the BatchBackend
Alexander-Auer 135879a
feat: a fake backend to run unit tests against
Alexander-Auer c97a24d
fix: some code cleanup in the helper functions
Alexander-Auer a5cda76
feat: setup for the unit test suite
Alexander-Auer db0683c
chore: style fix
Alexander-Auer a3eef9a
chore: styling, unit test fixture
Alexander-Auer f9dbb4c
chore: single letter var `h` refering to hanlde changed to `handle`
Alexander-Auer 9fbd6f0
fix: unused parameter in `test_injected_failure_is_reported_per_job()…
Alexander-Auer 1890fb6
chore: Making room in local fork for my documentation so i can easily…
Alexander-Auer cad5cea
fix: placed in the wrong folder :/
Alexander-Auer 835c7aa
chore: added testplan
Alexander-Auer 9332673
feat: Models the core contract needs
Alexander-Auer c77a73c
Feat: Protocols and Exceptions added for the core contract
Alexander-Auer 3171383
Feat: Private protocols for the BatchBackend
Alexander-Auer 0ecafb7
feat: a fake backend to run unit tests against
Alexander-Auer 4c054e5
fix: some code cleanup in the helper functions
Alexander-Auer d311a91
feat: setup for the unit test suite
Alexander-Auer 9831cab
chore: style fix
Alexander-Auer cf68d07
chore: styling, unit test fixture
Alexander-Auer bd78383
chore: single letter var `h` refering to hanlde changed to `handle`
Alexander-Auer 1f185a0
fix: unused parameter in `test_injected_failure_is_reported_per_job()…
Alexander-Auer 8f33853
Merge branch 'feat/core-contract' of github.com:Alexander-Auer/interc…
Alexander-Auer deeb558
fix: made the OutputSpec optional as it is not suppoerted by all and …
Alexander-Auer 5d18685
Feat: Core contract issue #16 (#1)
Alexander-Auer 51d56e8
fix: These documents should not exist in this way in the main repository
Alexander-Auer bb1427c
fix: None check for the outputs in submissionspec
Alexander-Auer aec9426
chore: trailing white spaces
Alexander-Auer d8d5413
chore: fixed pre commit locally
Alexander-Auer 7be1814
chore: removed pdf, as they just take up space and no one will use th…
Alexander-Auer a916473
fix: JobHandle key is now a combination of the handle's fields instea…
Alexander-Auer 1e46227
Merge branch 'main' into feat/core-contract
Alexander-Auer 6c13e44
feat: backend configuration and a closed registry
Alexander-Auer dcf68e0
test: expand unit test coverage to every logical case
Alexander-Auer b6946d6
feat: credential models and provider-based supply
Alexander-Auer f046db0
Merge branch 'main' into feat/core-contract
Alexander-Auer ceb3c21
test: trim the unit tests to logical cases that can happen
Alexander-Auer 9858175
Merge branch 'feat/core-contract' into stack/registry
Alexander-Auer ef717e2
test: trim the registry and configuration tests to logical cases
Alexander-Auer a2c106d
Merge branch 'stack/registry' into stack/test-expansion
Alexander-Auer 838578e
test: trim the expanded tests to logical cases that can happen
Alexander-Auer ce476b9
Merge branch 'stack/test-expansion' into stack/credential-models
Alexander-Auer 1931408
test: trim the credential tests to logical cases that can happen
Alexander-Auer 893ab2d
docs: follow the Diátaxis structure for the user documentation
Alexander-Auer 7b9ca48
fix: address the review of the core contract
Alexander-Auer aac93fe
Merge branch 'feat/core-contract' into stack/registry
Alexander-Auer 2cfc4dc
fix: no references to the ADRs in the registry and configuration
Alexander-Auer 951de16
Merge branch 'stack/registry' into stack/test-expansion
Alexander-Auer 168bab7
Merge branch 'stack/test-expansion' into stack/credential-models
Alexander-Auer a1166ea
fix: no references to the ADRs in the credential code
Alexander-Auer cd4d127
Merge DIRACGrid main into stack/registry (the core contract, squash-m…
Alexander-Auer 7380539
fix: apply the core contract review to the registry
Alexander-Auer d2a00f9
refactor: rename registry.backend() to get_backend()
Alexander-Auer 055c31e
refactor!: one public configuration model per backend kind
Alexander-Auer b8fb65f
docs: generate the configuration reference from the models
Alexander-Auer 4e6758f
Merge branch 'stack/registry' into stack/test-expansion
Alexander-Auer 46dc501
test: remove leftovers of the flat configuration model
Alexander-Auer b068c5e
Merge branch 'stack/test-expansion' into stack/credential-models
Alexander-Auer 3923887
fix: Rename FakeSettings to FakeConfig to stay in line with the revie…
Alexander-Auer a10537a
Merge remote-tracking branch 'origin/stack/registry' into stack/test-…
Alexander-Auer 24df600
test: keep the existing model tests where they are
Alexander-Auer dcde224
Merge DIRACGrid main into stack/test-expansion (the registry, squash-…
Alexander-Auer 1514a5b
test: a configuration that is not a mapping is refused
Alexander-Auer 0b79099
Merge branch 'stack/test-expansion' into stack/credential-models
Alexander-Auer d419ccf
Merge DIRACGrid main into stack/credential-models (the test expansion…
Alexander-Auer e83c067
fix: credential errors never repeat the value they were given
Alexander-Auer 904138f
fix: a provider's error text stays out of the AuthenticationError
Alexander-Auer ca1fe28
docs: document the credential fields in the API reference
Alexander-Auer f104906
docs: Google-style docstrings for the credential functions
Alexander-Auer 031b670
fix: Removed the x509 proxy as it will not be used anymore, and also …
Alexander-Auer cedffad
Merge branch 'main' into stack/credential-models
Alexander-Auer 11e435b
fix: Added a small credentialcache so each scope can get its own toke…
Alexander-Auer File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,192 @@ | ||
| """What a backend asks its credential provider for, when, and what it accepts back. | ||
|
|
||
| ``credential_requirements`` turns a configuration into what to ask for. This is | ||
| also the one place freshness is decided. A backend holds a ``CredentialCache``, | ||
| and before each operation asks it for that operation's credential: the provider | ||
| is asked only when there is none yet or it is about to expire. Where credentials | ||
| come from, and how they are renewed, stays the consumer's business, inside its | ||
| provider. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| from datetime import UTC, datetime, timedelta | ||
|
|
||
| from intercede.auth.models import ( | ||
| BearerToken, | ||
| Credential, | ||
| CredentialRequirements, | ||
| SSHPassword, | ||
| SSHPrivateKey, | ||
| ) | ||
| from intercede.auth.protocols import CredentialProvider | ||
| from intercede.config import ( | ||
| ARCConfig, | ||
| BackendConfig, | ||
| BatchSSHConfig, | ||
| HTCondorCEConfig, | ||
| ) | ||
| from intercede.exceptions import AuthenticationError, InterCEdeError | ||
|
|
||
| # What each configured SSH auth mode asks the credential provider for. | ||
| # "agent" is absent on purpose: an SSH agent holds the key, so nothing is asked. | ||
| AUTH_KINDS: dict[str, type[Credential]] = { | ||
| "key": SSHPrivateKey, | ||
| "password": SSHPassword, | ||
| } | ||
|
|
||
| # Ask again this long before a credential expires, so it cannot run out in the | ||
| # middle of an operation. A library default; whether a provider should be able | ||
| # to change it is still open. | ||
| DEFAULT_MARGIN = timedelta(minutes=5) | ||
|
|
||
|
|
||
| def credential_requirements( | ||
| config: BackendConfig, scopes: frozenset[str] = frozenset() | ||
| ) -> CredentialRequirements | None: | ||
| """Say what a backend must ask its provider for. | ||
|
|
||
| Args: | ||
| config: The backend's validated configuration. | ||
| scopes: The scopes of the operation about to run, for token-based | ||
| backends; ignored by the others. | ||
|
|
||
| Returns: | ||
| What to ask for, or None when the backend needs no credential: it runs | ||
| on this host, or an SSH agent holds the key. | ||
| """ | ||
| if isinstance(config, ARCConfig | HTCondorCEConfig): | ||
| # The endpoint is the resource, so a token is minted for this CE's audience. | ||
| return CredentialRequirements( | ||
| resource=config.endpoint, | ||
| kind=BearerToken, | ||
| scopes=scopes, | ||
| ) | ||
| if isinstance(config, BatchSSHConfig) and config.auth != "agent": | ||
| # The resource is the login, ``user@host:port``: a provider finds the key | ||
| # or password by it, and the same host can take different users. | ||
| return CredentialRequirements( | ||
| resource=f"{config.user}@{config.host}:{config.port}", | ||
| kind=AUTH_KINDS[config.auth], | ||
| ) | ||
| return None | ||
|
|
||
|
|
||
| def is_stale( | ||
| credential: Credential | None, | ||
| requirements: CredentialRequirements, | ||
| *, | ||
| margin: timedelta = DEFAULT_MARGIN, | ||
| now: datetime | None = None, | ||
| ) -> bool: | ||
| """Tell whether ``credential`` cannot serve ``requirements`` for ``margin`` more. | ||
|
|
||
| A credential that never expires is never stale. Only the kind and the expiry | ||
| are checked, not the scopes, since a token is opaque to interCEde: ``credential`` | ||
| must have been given for these same requirements, as ``CredentialCache`` ensures. | ||
|
|
||
| Args: | ||
| credential: The one the backend holds, or None before its first request. | ||
| requirements: What the backend needs for the operation about to run. | ||
| margin: How long the credential must stay valid from now. | ||
| now: The current time; tests pass a fixed one. | ||
|
|
||
| Returns: | ||
| True when there is no credential yet, when it is of another kind than | ||
| required, or when it expires within ``margin``. | ||
| """ | ||
| if not isinstance(credential, requirements.kind): | ||
| return True | ||
| if credential.expires_at is None: | ||
| return False | ||
| return credential.expires_at - margin <= (now or datetime.now(UTC)) | ||
|
|
||
|
|
||
| async def refresh_if_stale( | ||
| current: Credential | None, | ||
| requirements: CredentialRequirements, | ||
| provider: CredentialProvider, | ||
| *, | ||
| margin: timedelta = DEFAULT_MARGIN, | ||
| now: datetime | None = None, | ||
| ) -> Credential: | ||
| """Return ``current`` while it is fresh, otherwise a new one from ``provider``. | ||
|
|
||
| Args: | ||
| current: The credential the backend holds, or None before its first request. | ||
| requirements: What the backend needs for the operation about to run. | ||
| provider: The consumer's provider, asked only when ``current`` is stale. | ||
| margin: How long the credential must stay valid from now. | ||
| now: The current time; tests pass a fixed one. | ||
|
|
||
| Returns: | ||
| The credential to authenticate with: ``current``, or the provider's new one. | ||
|
|
||
| Raises: | ||
| AuthenticationError: The provider failed, returned something other than | ||
| the required kind, or returned a credential that has already expired. | ||
| Always a failure of the whole operation, never a per-job outcome. | ||
| """ | ||
| if not is_stale(current, requirements, margin=margin, now=now): | ||
| assert current is not None # is_stale() is True for None | ||
| return current | ||
|
|
||
| try: | ||
| fresh = await provider.get(requirements) | ||
| except InterCEdeError: | ||
| raise # already typed, e.g. an AuthenticationError from the provider | ||
| except Exception as exc: | ||
| # Only the type: the provider's message may hold a token or a secret URL. | ||
| # The exception itself stays available as __cause__. | ||
| raise AuthenticationError( | ||
| f"the credential provider failed for {requirements.resource} " | ||
| f"with {type(exc).__name__}" | ||
| ) from exc | ||
|
|
||
| if not isinstance(fresh, requirements.kind): | ||
| raise AuthenticationError( | ||
| f"the credential provider returned a {type(fresh).__name__} " | ||
| f"for {requirements.resource}, not a {requirements.kind.__name__}" | ||
| ) | ||
| # Expiring within the margin is accepted (it is asked again next time); | ||
| # already expired is not, since it cannot authenticate anything. | ||
| if is_stale(fresh, requirements, margin=timedelta(0), now=now): | ||
| raise AuthenticationError( | ||
| f"the credential provider returned an expired credential " | ||
| f"for {requirements.resource}" | ||
| ) | ||
| return fresh | ||
|
|
||
|
|
||
| class CredentialCache: | ||
| """Keeps one credential per requirements for one backend, so each scope gets its own token. | ||
|
|
||
| The provider is asked again only when that credential is missing or about to expire. | ||
| """ | ||
|
|
||
| def __init__(self, provider: CredentialProvider) -> None: | ||
| self._provider = provider | ||
| self._held: dict[CredentialRequirements, Credential] = {} | ||
|
|
||
| async def fresh(self, requirements: CredentialRequirements) -> Credential: | ||
| """Return the credential for these requirements, asking the provider if needed. | ||
|
|
||
| Args: | ||
| requirements: What the backend needs for the operation about to run. | ||
|
|
||
| Returns: | ||
| The credential to authenticate with. | ||
|
|
||
| Raises: | ||
| AuthenticationError: The provider failed, returned something other than | ||
| the required kind, or returned a credential that has already expired. | ||
| """ | ||
| credential = await refresh_if_stale( | ||
| self._held.get(requirements), requirements, self._provider | ||
| ) | ||
| self._held[requirements] = credential | ||
| return credential | ||
|
|
||
| def clear(self) -> None: | ||
| """Forget every credential, as a backend does when it closes.""" | ||
| self._held.clear() | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| """Backend credentials: typed, declared, supplied by a provider. | ||
|
|
||
| A consumer implements a ``CredentialProvider``, which returns the one credential | ||
| a backend asks for. Configuration never holds a secret: it only says how a backend | ||
| logs in, and the provider supplies the credential at run time. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| from intercede.auth.models import ( | ||
| BearerToken, | ||
| Credential, | ||
| CredentialRequirements, | ||
| SSHPassword, | ||
| SSHPrivateKey, | ||
| ) | ||
| from intercede.auth.protocols import CredentialProvider | ||
| from intercede.auth.providers import StaticCredentials | ||
|
|
||
| __all__ = [ | ||
| "BearerToken", | ||
| "Credential", | ||
| "CredentialProvider", | ||
| "CredentialRequirements", | ||
| "SSHPassword", | ||
| "SSHPrivateKey", | ||
| "StaticCredentials", | ||
| ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| """The public credential models a consumer of interCEde can use. | ||
|
|
||
| Credentials are immutable values. Secret material is held in pydantic's secret | ||
| types, so a credential cannot be printed or logged by accident: ``repr``, | ||
| ``str`` and JSON all show ``**********``, and the content is only readable on | ||
| purpose, with ``get_secret_value()``. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| from typing import Self | ||
|
|
||
| from pydantic import ( | ||
| AwareDatetime, | ||
| ConfigDict, | ||
| SecretBytes, | ||
| SecretStr, | ||
| model_validator, | ||
| ) | ||
|
|
||
| from intercede.models import _Model | ||
|
|
||
| __all__ = [ | ||
| "BearerToken", | ||
| "Credential", | ||
| "CredentialRequirements", | ||
| "SSHPassword", | ||
| "SSHPrivateKey", | ||
| ] | ||
|
|
||
|
|
||
| # -------------------------------------------------------------------------- | ||
| # Credential values | ||
| # -------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| class Credential(_Model): | ||
| """Base of every credential: the one field the freshness check needs. | ||
|
|
||
| Not used on its own: a provider returns one of the concrete kinds below. | ||
| """ | ||
|
|
||
| # Added to _Model's settings: a value given by mistake may be the secret | ||
| # itself, so no error message repeats one. | ||
| model_config = ConfigDict(hide_input_in_errors=True, use_attribute_docstrings=True) | ||
|
|
||
| expires_at: AwareDatetime | None = None | ||
| """When the credential stops working, with a timezone, so it can be compared | ||
| with the current time. None means it never expires, as for SSH secrets.""" | ||
|
|
||
|
|
||
| class BearerToken(Credential): | ||
| """A token for the channel to a CE. Opaque to interCEde.""" | ||
|
|
||
| value: SecretStr | ||
| """The token itself.""" | ||
|
|
||
|
|
||
| class SSHPrivateKey(Credential): | ||
| """An SSH private key: the key itself, never a path on some host.""" | ||
|
|
||
| pem: SecretBytes | ||
| """The private key, PEM-encoded.""" | ||
|
|
||
| passphrase: SecretStr | None = None | ||
| """The passphrase the key is encrypted with, or None if it is not.""" | ||
|
|
||
|
|
||
| class SSHPassword(Credential): | ||
| """A password for an SSH login.""" | ||
|
|
||
| value: SecretStr | ||
| """The password itself.""" | ||
|
|
||
|
|
||
| # -------------------------------------------------------------------------- | ||
| # What a backend asks for | ||
| # -------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| class CredentialRequirements(_Model): | ||
| """What a backend asks its provider for. | ||
|
|
||
| The backend computes it from its own configuration and keeps it to itself: | ||
| a consumer only meets it as the argument to its provider's ``get``. | ||
| """ | ||
|
|
||
| model_config = ConfigDict(use_attribute_docstrings=True) | ||
|
|
||
| resource: str | ||
| """Which backend instance is asking: the endpoint it talks to, the same | ||
| string as ``JobHandle.resource``. For a token this is also the audience, | ||
| in the form that CE expects.""" | ||
|
|
||
| kind: type[Credential] | ||
| """The kind of credential needed: a backend authenticates with exactly one.""" | ||
|
|
||
| scopes: frozenset[str] = frozenset() | ||
| """For a token, the scopes of the operation about to run; empty for the | ||
| other kinds.""" | ||
|
|
||
| @model_validator(mode="after") | ||
| def _concrete_kind(self) -> Self: | ||
| if self.kind is Credential: | ||
| raise ValueError("kind must be a concrete kind, not the Credential base") | ||
| return self |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can have different tokens for a same backend.
For instance, a token to submit a job with scope
compute.create, and another one to get the status with scopecompute.read.I guess that needs to be taken into account here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed with a small
CredentialCachein_backends/credentials.py.Why it exists:
is_staleonly looks at a credential's kind and expiry, not its scopes, because tokens are opaque to interCEde. A backend that kept a single token would treat one fetched forcompute.readas fresh forcompute.create, and submit with it without asking the provider.What it does: it keeps one credential per
CredentialRequirements, so each set of scopes gets its own token. Before an operation, a backend callsawait cache.fresh(requirements), and the provider is only asked if that credential is missing or about to expire. A provider may mint a token on every call and backends poll often, so this keeps those calls down.What it doesn't do: it isn't a store of secrets. It holds credentials in memory only, for one backend, until that backend closes. Where they come from and how they are renewed stays with the provider.
No backend in this PR uses it yet. The HTCondor-CE backend (#35) keeps its own dict per scope today, and I plan to switch that and the SSH transport to this.