[remote-controlled-man]

Summary

Found through source analysis and reproduced with real CPU spawn workers through the public inference methods. If inference raises, the worker exits (the reranker also swallows the exception) while the parent waits indefinitely for one result per chunk.

Share the worker/error protocol and result collector across embedding, reranking, ICL query encoding and M3 scoring. Return traceback text in a picklable failure message. An independent process-health monitor also interrupts a partial queue-frame read if a worker exits: terminate the remaining senders and close the parent's unused send handle so the read receives EOF. Stop/discard the failed pool before raising to the caller. Owned pools are recreated on the next call, so outstanding results cannot contaminate it. Successful result ordering and formats are preserved.

The regression coverage uses real processes/queues, controlled failures and abrupt exits, successful and interrupted 32 MiB transfers with a live peer, all four worker entry points, and a locally initialized tiny XLM-R model. Its genuine embedding-layer IndexError reaches the caller, then subsequent multi-process embeddings match single-device embeddings.

Related: #1615 / #1616. The single-pair normalization proposed in #1616 is independent of this shared error propagation change.

Validation

Final head: b90bf6975db30d72f53e24a7509c7dd797d3b1f5. Python 3.12.14, PyTorch 2.15.0, Transformers 5.22.2, macOS CPU.

The following test selection ran with task-local Hugging Face/UV caches and pytest cache/temp overrides: 25 passed in 92.89s.

OMP_NUM_THREADS=1 PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B -m pytest \
  tests/test_multiprocess_failures.py tests/test_imports_v5.py \
  tests/test_finetune_trainer_compat.py tests/test_reranker_tokenizer_compat.py \
  tests/test_tokenizer_pad_compat.py -q
git diff --check 897baa67c0ffddbfccf4e6776ff55310d67809d4 HEAD

The regression cases reproduce failure propagation and recovery; successful transfers provide controls. An independent AI review also reproduced and checked the partial-transfer/live-peer failure and the pool/thread cleanup.

CPU validation only. The full pretrained-model suite and GPU/multi-GPU execution are not covered. This detects exceptions and exited workers; it does not impose a timeout on a worker that remains alive.

AI assistance

OpenAI Codex assisted with investigation, implementation, tests and the description. An independent AI reviewer checked the frozen changes and verification evidence. No human code review is claimed.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.