Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 24 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,10 +61,11 @@ aai.settings.api_key = "your-key"
- `aai.TTSError` — Raised on any Text-to-Speech refusal: `.status_code`, `.error_code` (stable API code; `biometric_terms_not_accepted` 403 until the account accepts the biometric data terms in the dashboard, `auth_unavailable` 503 retryable; on a session these arrive as close 3006 and 1011, with `status_code` None, the close code in `.body["close_code"]` and `.retryable` from the API's Error frame), `.param`, `.request_id`, `.retryable`, `.retry_after`, `.errors`, `.body` (the whole envelope)
- `assemblyai.streaming.v3.RealTimeTranscriber` — Real-time streaming with event-based API (threaded)
- `assemblyai.streaming.v3.AsyncRealTimeTranscriber` — Asyncio-native counterpart; same options/events
- `aai.LLMGateway` — LLM Gateway client. Resources: `models.list()`, `chat.completions.create()` (incl. `stream=True`), `understanding.create()`/`.validate()`
- `aai.LLMGateway` — LLM Gateway client. Resources: `models.list()`, `chat.completions.create()` (incl. `stream=True`), `understanding.create()`/`.validate()`, `decisions.create()`
- `aai.AsyncLLMGateway` — Asyncio counterpart of `LLMGateway`. Same resources, every API call a coroutine; owns an HTTP pool (`async with` or `await aclose()`)
- `assemblyai.llm_gateway.v1` — Canonical module for both gateways, matching the gateway's `/v1` API. The top-level `aai.*` names re-export it
- `aai.LLMGatewayMessageParam` — `TypedDict` for a chat message; pass plain dicts (`{"role": "user", "content": "..."}`)
- `aai.LLMGatewayDecisionQuestionParam` — `TypedDict` for a decisions question; pass plain dicts (`{"type": "predicate", "name": "...", "instructions": "..."}`)

## Common patterns

Expand Down Expand Up @@ -574,6 +575,28 @@ gateway.understanding.validate(request={"translation": {"target_languages": ["es
when the request is valid and raises `aai.LLMGatewayError` with `.errors` populated otherwise;
it costs the same auth/balance as a real call, so it is not a free dry run.

**Decisions**: answers `predicate`, `choice`, and `score` questions about an input (a string or a
list of input item dicts) in one request. Questions are plain dicts (`aai.LLMGatewayDecisionQuestionParam`);
each answer carries its question's `name`. `prompt_cache_key`, `prompt_cache_options`, `provider`,
`fallbacks`, `fallback_config`, and `model_region` are named, and any other keyword argument is forwarded
into the body. Only some models serve decisions, and `models.list()` does not show decisions-only models.
```python
result = gateway.decisions.create(
model="gpt-6-luna",
input="Agent: Is there anything else? Caller: No, that fixed it, thanks!",
questions=[
{"type": "predicate", "name": "resolved", "instructions": "Was the caller's issue resolved?"},
{"type": "choice", "name": "sentiment", "instructions": "Caller sentiment at the end of the call",
"choices": [{"value": "positive", "description": "Satisfied"}, {"value": "negative", "description": "Unhappy"}]},
{"type": "score", "name": "politeness", "instructions": "How polite was the agent?",
"levels": [{"label": "rude", "description": "Dismissive"}, {"label": "polite", "description": "Courteous"}]},
],
)
for answer in result.answers:
print(answer.name, answer.probability, answer.choice, answer.score)
```
`probabilities[].value` is the choice value (str) for a choice question and the level index (int) for a score question.

**Errors**: every failure raises `aai.LLMGatewayError` (an `aai.AssemblyAIError`) with
`.status_code`, `.request_id`, and `.errors` (validation messages, when the server sent any).
For a stream, only a failure before the first byte raises — a mid-stream failure truncates the
Expand Down
48 changes: 47 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1558,7 +1558,7 @@ for result in transcript.auto_highlights.results:

### **LLM Gateway Examples**

`aai.LLMGateway` is a client for AssemblyAI's LLM Gateway — model listing, OpenAI-shaped chat completions routed to OpenAI/Claude/Gemini/Bedrock models, and speech understanding over an existing transcript. `aai.AsyncLLMGateway` is the asyncio counterpart, with the same resources and every API call a coroutine.
`aai.LLMGateway` is a client for AssemblyAI's LLM Gateway — model listing, OpenAI-shaped chat completions routed to OpenAI/Claude/Gemini/Bedrock models, speech understanding over an existing transcript, and decisions (predicate, choice, and score questions about an input). `aai.AsyncLLMGateway` is the asyncio counterpart, with the same resources and every API call a coroutine.

[Read more about the LLM Gateway.](https://www.assemblyai.com/docs/llm-gateway/quickstart)

Expand Down Expand Up @@ -1689,6 +1689,52 @@ except aai.LLMGatewayError as error:

</details>

<details>
<summary>Ask decisions questions about an input</summary>

`decisions.create()` answers `predicate`, `choice`, and `score` questions about an input in one request. Each answer carries its question's `name`.

```python
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

gateway = aai.LLMGateway()

result = gateway.decisions.create(
model="gpt-6-luna",
input="Agent: Is there anything else? Caller: No, that fixed it, thanks!",
questions=[
{"type": "predicate", "name": "resolved", "instructions": "Was the caller's issue resolved?"},
{
"type": "choice",
"name": "sentiment",
"instructions": "Caller sentiment at the end of the call",
"choices": [
{"value": "positive", "description": "Satisfied"},
{"value": "negative", "description": "Unhappy"},
],
},
{
"type": "score",
"name": "politeness",
"instructions": "How polite was the agent?",
"levels": [
{"label": "rude", "description": "Dismissive"},
{"label": "polite", "description": "Courteous"},
],
},
],
)

for answer in result.answers:
print(answer.name, answer.probability, answer.choice, answer.score)
```

`input` can also be a list of input item dicts. Only some models serve decisions; an unsupported model raises `aai.LLMGatewayError`.

</details>

<details>
<summary>Use it from asyncio (`AsyncLLMGateway`)</summary>

Expand Down
6 changes: 6 additions & 0 deletions assemblyai/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@
LLMGatewayChunkChoice,
LLMGatewayChunkDelta,
LLMGatewayCompletionChunk,
LLMGatewayDecisionAnswer,
LLMGatewayDecisionQuestionParam,
LLMGatewayDecisionResponse,
LLMGatewayMessageParam,
LLMGatewayModel,
LLMGatewayModelList,
Expand Down Expand Up @@ -187,6 +190,9 @@ def __getattr__(name: str):
"LLMGatewayChunkChoice",
"LLMGatewayChunkDelta",
"LLMGatewayCompletionChunk",
"LLMGatewayDecisionAnswer",
"LLMGatewayDecisionQuestionParam",
"LLMGatewayDecisionResponse",
"LLMGatewayError",
"LLMGatewayMessageParam",
"LLMGatewayModel",
Expand Down
7 changes: 6 additions & 1 deletion assemblyai/llm_gateway/v1/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,16 @@
LLMGatewayChunkChoice,
LLMGatewayChunkDelta,
LLMGatewayCompletionChunk,
LLMGatewayDecisionAnswer,
LLMGatewayDecisionResponse,
LLMGatewayModel,
LLMGatewayModelList,
LLMGatewayResponseMessage,
LLMGatewayToolCall,
LLMGatewayUnderstandingResponse,
LLMGatewayUsage,
)
from .params import LLMGatewayMessageParam
from .params import LLMGatewayDecisionQuestionParam, LLMGatewayMessageParam

__all__ = [
"AsyncLLMGateway",
Expand All @@ -33,6 +35,9 @@
"LLMGatewayChunkChoice",
"LLMGatewayChunkDelta",
"LLMGatewayCompletionChunk",
"LLMGatewayDecisionAnswer",
"LLMGatewayDecisionQuestionParam",
"LLMGatewayDecisionResponse",
"LLMGatewayError",
"LLMGatewayMessageParam",
"LLMGatewayModel",
Expand Down
20 changes: 18 additions & 2 deletions assemblyai/llm_gateway/v1/_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
calls themselves.
"""

from typing import Any, Dict, List, Optional
from typing import Any, Dict, List, Optional, Union

from .params import LLMGatewayMessageParam
from .params import LLMGatewayDecisionQuestionParam, LLMGatewayMessageParam


class _BaseResource:
Expand Down Expand Up @@ -73,3 +73,19 @@ def validate_body(
body["transcript_id"] = transcript_id

return body


def decision_body(
*,
model: str,
input: Union[str, List[Dict[str, Any]]],
questions: List[LLMGatewayDecisionQuestionParam],
extra: Dict[str, Any],
) -> Dict[str, Any]:
"""Builds the request body of `POST /v1/decisions`."""
return {
"model": model,
"input": input,
"questions": questions,
**extra,
}
25 changes: 25 additions & 0 deletions assemblyai/llm_gateway/v1/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
ENDPOINT_CHAT_COMPLETIONS = "/v1/chat/completions"
ENDPOINT_UNDERSTANDING = "/v1/understanding"
ENDPOINT_UNDERSTANDING_VALIDATE = "/v1/understanding/validate"
ENDPOINT_DECISIONS = "/v1/decisions"


def _error_from_response(response: httpx.Response) -> types.LLMGatewayError:
Expand Down Expand Up @@ -186,3 +187,27 @@ def validate_understanding(

if response.status_code != httpx.codes.OK:
raise _error_from_response(response)


def create_decision(
client: httpx.Client,
*,
base_url: str,
timeout: float,
body: Dict[str, Any],
) -> models.LLMGatewayDecisionResponse:
"""
Answers a set of decision questions about an input.

Raises: `LLMGatewayError` on any non-200 response.
"""
response = client.post(
base_url.rstrip("/") + ENDPOINT_DECISIONS,
json=body,
timeout=timeout,
)

if response.status_code != httpx.codes.OK:
raise _error_from_response(response)

return models.LLMGatewayDecisionResponse.parse_obj(response.json())
25 changes: 25 additions & 0 deletions assemblyai/llm_gateway/v1/async_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
from . import models
from .api import (
ENDPOINT_CHAT_COMPLETIONS,
ENDPOINT_DECISIONS,
ENDPOINT_MODELS,
ENDPOINT_UNDERSTANDING,
ENDPOINT_UNDERSTANDING_VALIDATE,
Expand Down Expand Up @@ -152,3 +153,27 @@ async def validate_understanding(

if response.status_code != httpx.codes.OK:
raise _error_from_response(response)


async def create_decision(
client: httpx.AsyncClient,
*,
base_url: str,
timeout: float,
body: Dict[str, Any],
) -> models.LLMGatewayDecisionResponse:
"""
Answers a set of decision questions about an input.

Raises: `LLMGatewayError` on any non-200 response.
"""
response = await client.post(
base_url.rstrip("/") + ENDPOINT_DECISIONS,
json=body,
timeout=timeout,
)

if response.status_code != httpx.codes.OK:
raise _error_from_response(response)

return models.LLMGatewayDecisionResponse.parse_obj(response.json())
55 changes: 54 additions & 1 deletion assemblyai/llm_gateway/v1/async_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

from ... import async_client as _async_client
from . import _base, async_api, models
from .params import LLMGatewayMessageParam
from .params import LLMGatewayDecisionQuestionParam, LLMGatewayMessageParam


class AsyncModelsResource(_base._BaseResource):
Expand Down Expand Up @@ -220,6 +220,58 @@ async def validate(
)


class AsyncDecisionsResource(_base._BaseResource):
"""The `decisions` resource of `AsyncLLMGateway`."""

# Named keyword annotations below are for IDE/mypy discoverability only. They add no
# runtime validation: unnamed fields still flow through **kwargs untouched, and the
# gateway remains the sole source of truth for what a request accepts.
async def create(
self,
*,
model: str,
input: Union[str, List[Dict[str, Any]]],
questions: List[LLMGatewayDecisionQuestionParam],
prompt_cache_key: Optional[str] = None,
prompt_cache_options: Optional[Dict[str, Any]] = None,
provider: Optional[str] = None,
fallbacks: Optional[List[Dict[str, Any]]] = None,
fallback_config: Optional[Dict[str, Any]] = None,
model_region: Optional[str] = None,
**kwargs: Any,
) -> models.LLMGatewayDecisionResponse:
"""
Answers a set of predicate, choice, or score questions about an input.

`input` is a string or a list of input item dicts. `questions` are
plain dicts (see `LLMGatewayDecisionQuestionParam`), e.g.
`[{"type": "choice", "name": "sentiment", "instructions": "...",
"choices": [{"value": "positive", "description": "..."}, ...]}]`.
Each answer in the response carries its question's `name`. Extra
keyword arguments are forwarded directly into the request body.

Raises: `LLMGatewayError` if the request fails.
"""
named = {
"prompt_cache_key": prompt_cache_key,
"prompt_cache_options": prompt_cache_options,
"provider": provider,
"fallbacks": fallbacks,
"fallback_config": fallback_config,
"model_region": model_region,
}
extra = {k: v for k, v in named.items() if v is not None}
extra.update(kwargs)

return await async_api.create_decision(
self._gateway.client.http_client,
body=_base.decision_body(
model=model, input=input, questions=questions, extra=extra
),
**self._http_options,
)


class AsyncLLMGateway:
"""
The asyncio counterpart of `LLMGateway`.
Expand Down Expand Up @@ -275,6 +327,7 @@ def __init__(
self.models = AsyncModelsResource(self)
self.chat = AsyncChatResource(self)
self.understanding = AsyncUnderstandingResource(self)
self.decisions = AsyncDecisionsResource(self)

@property
def client(self) -> _async_client.AsyncClient:
Expand Down
Loading
Loading