Skip to content

Fix quadratic processing time in fenced_code preprocessor (up to ~10x faster) - #1660

Open
deadstrobe5 wants to merge 2 commits into
Python-Markdown:masterfrom
deadstrobe5:fenced-code-linear
Open

deadstrobe5 wants to merge 2 commits into
Python-Markdown:masterfrom
deadstrobe5:fenced-code-linear

Conversation

@deadstrobe5

@deadstrobe5 deadstrobe5 commented Oct 16, 2026 •

Copy link
Copy Markdown

The FencedBlockPreprocessor rebuilt the whole document text after every fenced code block it stashed (text = f'{text[:m.start()]}\n{placeholder}\n{text[m.end():]}'), so its cost grew with document size × number of blocks. This collects the output in pieces and joins them once.

Matching is unchanged: the search now resumes at m.end() in the original text instead of just after the placeholder in the rewritten text. The two differ only by the inserted \n, and since the fence pattern ends in [ ]*$, text[m.end()] is always \n or the end of the text, so no fence can begin at that position either way.

Benchmark

The project's own docs/*.md concatenated (250 KB, ~220 fenced blocks), and the same text ×4 (1 MB). Python 3.12, Apple M5, 7 alternating processes per side:

Case Before After
preprocessor alone, 250 KB 10.6 ms 1.95 ms (5.4x)
preprocessor alone, 1 MB 79.4 ms 8.2 ms (9.7x)
markdown(text, extensions=['extra']), 250 KB 81.9 ms 73.8 ms (~10%)
markdown(text, extensions=['extra']), 1 MB 371 ms 304 ms (~18%)
extra on a 10 KB document 2.40 ms 2.37 ms (unchanged)

Checks

  • Output is byte-identical to the current code (same SHA-256) for 169 files (all tests/**/*.txt fixtures and docs/**/*.md) × 4 extension sets, plus 12,000 randomized documents with fence edge cases (unterminated fences, invalid attribute lists, missing trailing newline, fences inside blockquotes and lists) under fenced_code, fenced_code + attr_list, and extra.
  • python -m unittest discover tests: 1127 OK; flake8 clean on the changed file.
  • Changelog entry added under Unreleased / Fixed.

Re-measured on CPython 3.14.7 (7 alternating processes per side, median): the docs concatenated with the extra extensions went from 89.3 ms to 78.7 ms (1.14x).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant