Skip to content

feat(MultimodalChatWithTools): Expose fileIds of attached files to the LLM - #464

Merged
marcelklehr merged 1 commit into
mainfrom
feat/expose-file-ids-to-model
Nov 11, 2026
Merged

marcelklehr merged 1 commit into
mainfrom
feat/expose-file-ids-to-model

Conversation

@marcelklehr

@marcelklehr marcelklehr commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

in order for the model to call tools with the fileIds.

this is part of a pack of PRs:

🤖 AI (if applicable)

  • The content of this PR was partly or fully generated using AI (N/A)

…e LLM

Assisted-by: ClaudeCode:claude-opus-5-5
Signed-off-by: Marcel Klehr <mklehr@gmx.net>
@WSHAPER

WSHAPER commented Nov 10, 2026

Copy link
Copy Markdown

Nice! A few notes from reading the diff…

The permission boundary looks right (user folder only, appdata files stay unlabeled). Two small things:

  1. The filename goes into the prompt unescaped. A file shared by someone else can carry a crafted name, so wrapping it in json_encode or at least escaping the quotes would close that.

  2. For history parts without ocp_task_id, isInUserFolder repeats the lookup that buildFileContentFromId just did. Since the history replays every turn, that is N extra filecache queries per turn for N files. buildFileContentFromId already knows which branch it took, so it could pass the answer along.

One thing worth a line in the label or the system prompt (which I'm naively assuming): nothing tells the model the id is for tool calls only, so replies like "your file (file ID: 42)" would be likely.

@marcelklehr
marcelklehr marked this pull request as ready for review November 11, 2026 07:23

@kyteinsky kyteinsky left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

Comment on lines +141 to +153
/**
* Only files in the user's folder can be used by tools, files in appdata (e.g. task outputs) cannot
*/
private function isInUserFolder(File $file, ?string $userId): bool {
if ($userId === null) {
return false;
}
try {
return $this->rootFolder->getUserFolder($userId)->getFirstNodeById($file->getId()) !== null;
} catch (NotPermittedException $e) {
return false;
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe it's enough to check the file path here if it's in appdata or not

@marcelklehr

Copy link
Copy Markdown
Member Author

The filename goes into the prompt unescaped. A file shared by someone else can carry a crafted name, so wrapping it in json_encode or at least escaping the quotes would close that.

I think that's acceptable, there are strict rules for file names in nextcloud.

For history parts without ocp_task_id, isInUserFolder repeats the lookup that buildFileContentFromId just did. Since the history replays every turn, that is N extra filecache queries per turn for N files. buildFileContentFromId already knows which branch it took, so it could pass the answer along.

I think that's acceptable, as there is a very small cap on the number of attachments per chat session.

@marcelklehr
marcelklehr merged commit 6bde64c into main Nov 11, 2026
25 checks passed
@marcelklehr
marcelklehr deleted the feat/expose-file-ids-to-model branch November 11, 2026 07:41
@kyteinsky

Copy link
Copy Markdown
Contributor

nothing tells the model the id is for tool calls only

it's a normal nextcloud file id so I suppose it can be used for any purpose by the agent
did you have anything on mind where the file id should not be used?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants