Repository navigation
Conversation
…hat template
Render layer (xtuner/v1/data_proto/messages/glm53_chat.py):
- Render contiguous tool blocks in the preceding assistant's tool_calls order
when the official can_sort rule holds (all result ids present, unique within
the block, matching a unique tool_call id); otherwise keep message order.
- Render structured {"output": ...} tool results through visible_text and
expand nested tool_reference lists into <tool_response><tools>...</tools>
</tool_response>, instead of Python-stringifying them.
- Render assistant/tool media with the GLM-5.3 placeholder markers via local
_assistant_content_and_reasoning/_render_tool_response (the shared glm52
helpers degrade media to <reminder> text, which GLM-5.3 never emits).
- Append the training <|endoftext|> only after the final assistant message;
the official template never emits EOS mid-conversation, so assistant->system
and assistant->assistant boundaries no longer get one (the slowspeed golden
drops the truncated render's trailing EOS accordingly and un-appends the
prefix EOS for consecutive assistants).
Data layer:
- collect_image_video_paths_and_extra skips plain-string content parts instead
of crashing on c["type"] (strings are text per the official template).
- BaseMLLMTokenizeFunction gains a collect_media hook; Glm53VLTokenizeFunction
overrides it to reject image/video in non-user/pretrain roles with
NotImplementedError in both cache and runtime (the base __call__ no longer
downgrades NotImplementedError into a cache drop), instead of silently
training samples whose placeholder markers are never expanded.
Tests: 11 render-contract cases + 3 can_sort fallback cases from the 2026-10-15
template-consistency failure report (text equality with the checkpoint's
official chat_template.jinja plus fast/slow parity), boundary EOS/loss
semantics, tool-result supervision, media-role rejection and string content
parts. 65 passed in the GLM53 dataset suites; glm52/qwen3_vl/qwen35_vl
regression 32 passed.
Note on length caches: rendered text changes for tool blocks, structured tool
outputs and mid-conversation EOS, so cached lengths computed by older code can
be stale for such samples; delete the old length cache once, as before.
The dataset cache builder runs tokenize_fn in worker processes (jsonl.py tokenize_worker) with no exception handling: any escaping exception kills the worker and the main process blocks forever on the result queue. Route every per-sample dirty-data condition through the existing damaged-sample protocol (log warning + num_tokens=0 in cache, filtered before packing; an upstream inert placeholder at runtime): - BaseMLLMTokenizeFunction.__call__: mixed image+video samples now log a warning and are discarded in cache instead of raising NotImplementedError (which escaped even in cache state); the media-role rejection from collect_media overrides (e.g. GLM-5.3 non-user/pretrain media) takes the same route via _discard_damaged_sample. - Image/video cache geometry/metadata calculation is wrapped: RuntimeError/KeyError/ValueError/IndexError -> warning + drop. AssertionError is deliberately NOT caught, so config errors (e.g. enable_thinking=False) still surface instead of silently emptying the dataset. - collect_image_video_paths_and_extra: per-sample asserts (image_wh / video_wh shape or count mismatch) become RuntimeError so they hit the existing cache drop channel instead of killing the worker. - glm53 video cache: missing origin_video_length/origin_fps metadata raises RuntimeError (dropped with a warning) instead of an assert that escaped the cache worker. Tests: mixed-media and media-role contract tests updated to the discard semantics (cache warns + drops with num_tokens=0, runtime raises for the inert placeholder, media is never loaded); new TestGlm53DirtyDataDoesNotHalt covers bad image_wh, missing video metadata, and that config errors still raise in cache. GLM53 dataset suites 64 passed; glm52/qwen3_vl/qwen35_vl regression 32 passed; ruff check/format clean.
…ignment - docformatter --wrap-descriptions 119 rewraps (no behavior change) - _is_tool_reference_list becomes a TypeGuard so the narrowed value can be passed to _tool_references_to_response (refs typed as list[Mapping]) - collect_media accumulates role names into a set[str] via str(role) so sorted() sees a homogeneous comparable type
This was referenced Oct 16, 2026
Merged
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
接续 #2180。2026-10-15 的 GLM-5.3-Flash 训推 chat template 一致性对比（SFT v2 样例集 vs checkpoint 内官方
chat_template.jinja，SHA-25634d5ee66…）报了 11 个不一致 + 5 个异常。本 PR 修复其中剩余的渲染层与数据层差异（M01 的enable_thinking已由 #2180 显式拒绝）：tool_calls调用序重排output被 Python 字符串化visible_text渲染可见内容tool_reference未展开<tool_response><tools>…</tools></tool_response><reminder>降级文本或字符串化TypeErrorvisible_text一致）NotImplementedErrorE03–E05 音频属对比脚本覆盖限制，渲染层占位符本就一致，不改代码。
改动
渲染层
xtuner/v1/data_proto/messages/glm53_chat.py（GLM-5.2 官方模板没有排序/嵌套引用/结构化 output 逻辑，故全部实现留在 glm53 本地，不改共享 glm52 helper）：_render_tool_block：连续 tool 消息块按官方 can_sort 规则重排或保持原序_render_tool_response/_tool_references_to_response：结构化 output 与嵌套引用渲染_assistant_content_and_reasoning：assistant 媒体渲染占位标记而非 reminder<|endoftext|>仅追加在最终 assistant 后；slowspeed golden 对连续 assistant 前缀与 system/assistant 边界的 EOS 做对称处理数据层：
base_mllm_tokenize_fn.py：collector 跳过纯字符串 content 块；新增collect_media钩子glm53_vl_tokenize_fn.py：overridecollect_media，非 user/pretrain 角色的图片/视频按脏样本处理（"完整接通或显式拒绝"中选择了拒绝接通；接通工具链媒体是更大的独立工作）脏数据不中断（commit 2）
数据集 cache 构建在 worker 子进程中运行 tokenize_fn（
jsonl.py的tokenize_worker没有异常捕获）：任何逃逸异常会杀死 worker，主进程永远阻塞在结果队列上——整条训练启动挂住。本 PR 把所有单条脏数据路径统一到既有损坏样本协议（cache：log warning +num_tokens=0，打包前过滤；runtime：抛出由上游换成无 loss 占位样本）：NotImplementedError）RuntimeError/KeyError/ValueError/IndexError→ 警告+丢弃；AssertionError故意不捕获，配置错误（如enable_thinking=False）必须继续暴露，不能静默掏空数据集image_wh/video_wh形状或数量不匹配）改为RuntimeError，进入既有 cache 丢弃通道origin_video_length/origin_fps元数据：RuntimeError（丢弃）替代 assert（挂住）已知残余风险：
tokenize_worker本身仍无异常转发，未覆盖到的异常类型（如 OSError）在 cache 构建期仍可能挂住；作为后续工作另行评估。测试
共享环境 pt29_glm2（transformers 5.17.0，
GLM_5_3_FLASH_PATH指向共享盘四层 checkpoint，其 chat_template.jinja 与 25B 字节一致）：test_glm53_chat.py/test_glm53_vl_tokenize_fn.py/test_glm53_vl_collator.py：64 passed（11 个报告渲染样例 + 3 个 can_sort 回退样例逐字符对齐官方 jinja 且 fast/slow parity；边界 EOS/loss、tool 结果不监督；混合媒体与媒体角色的丢弃语义；字符串 content 块；TestGlm53DirtyDataDoesNotHalt覆盖坏 image_wh、缺视频元数据、配置错误不吞没）test_glm52_openai_tokenize_fn.py+test_qwen3_vl_tokenize_fn.py+test_qwen35_vl_tokenize_fn.py32 passedNote on length caches：tool 块、结构化 output、中间 EOS 的渲染文本变化会影响既有长度缓存中此类样本的 num_tokens，请按 #2180 的同样做法手动删除一次旧长度缓存。