[SizzleUnrlsd]

Summary

Closes #278.

Since #272, the reading ended a comment holding template tags at its first tag, and read what follows as markup. In the rendered page, the comment may also:

• end at a later tag;
• run to its close, when its tags render nothing.

The markup after that point is then read from HTML data. In <!--{% if debug %}{% endif %}<a title="--><{{ name }}">x</a>, the comment runs to the --> in the title. The variable after it names a tag: Django 5.1.15 renders <!--<a title="--><img src=x onerror=...>, and Chrome runs the handler. Nothing was reported, where 0.23.0 reported it. The regression is in no release.

Rule

• The kept reading. The reading resumed after the first tag is kept.
• The points of resumption. That reading must be in HTML data right after each later tag of the comment, and right after its close, where the other readings resume. There, every other reading reads what it reads, which it has already accepted.
• The prudent refusal. Where the kept reading is inside a tag, an end tag, a script or style block or another comment at such a point, the text is refused.
• Inserted markups. An inserted markup (templates: a block or include inside a parent's script inserts unchecked markup there #273) anywhere inside such a comment is refused, since the comment may run past it.
• Comments left open. Such a comment left open does not end in HTML data, since it may hide what follows.

Why the cost is linear

The reading only moves forward, and every part of the text is looked at once:

• Constructs: each is matched where it starts, and the reading resumes after it, as in templates: constructs left open, and template tags inside comments, hide cross-site scripting; unclosed ones make the inspection quadratic #272.
• Closes: each is searched once for all the comments opening before it.
• First tags: each is searched from its comment's opening to that tag, and the reading then resumes after it.
• Later tags: each is listed once, beyond the furthest point already listed.
• Inserted markups: their positions are listed once, and a comment looks one up by bisection.
• The points of resumption come in two sequences that never go back: the later tags as they are listed, and the closes as their comments are read. Each sequence is a queue, and each point enters and leaves it once.

Shape, 3,000,000 characters	main	This change
< repeated	1.18 s	1.48 s
<!--{% if x %} repeated	0.16 s	0.55 s
<p>x</p> repeated	0.44 s	0.53 s
the same, then one -->	0.18 s	0.62 s
one comment holding 3,000,000 tags, left open	0.00 s	0.33 s
one comment holding 1,500,000 tags, closed	0.00 s	0.17 s
<!--{% if x %}<a repeated	0.17 s	0.21 s, refused

Commit	Change
fix(templates): count a comment holding template tags left open as not ending in HTML data	such a comment hides what follows
fix(templates): require the reading in HTML data wherever another continuation of a comment resumes	the points of resumption, inserted markups anywhere in the comment
docs: state how a comment holding template tags is read	docs/usage.md, the module's docstring

Results

Bench of generated texts. 1,708,598 texts were checked, every text of up to five pieces and 200,000 random ones. The reading was compared with main's, and with a reference reading every continuation, which is quadratic:

• it accepts no text the reference refuses;
• it accepts no text main refuses;
• 2,047 texts main accepted are refused, rightly;
• 719 are refused prudently, though the reference accepts them, as with <!--{% if %}<a title="--><!--.

Corpus. On the local corpus, 614 templates are established before and after, and none changes. The remote battery could not be reached.

Reproductions.

• this issue's reproduction is reported;
• D and G of templates: constructs left open, and template tags inside comments, hide cross-site scripting; unclosed ones make the inspection quadratic #272 stay reported;
• the controls stay established.

Tests

The new file tests/test_template_comment_continuations.py holds 7 tests, 15 cases. No existing test changes.

• Regressions:
  • this issue's reproduction, end to end;
  • a later tag inside an attribute of the comment;
  • an inserted markup after the first tag;
  • a comment holding tags left open;
  • four controls.
• Comparison with the reference: the reference reading every continuation is written in the test. On every text of up to four pieces, and on 30,000 random ones, the reading accepts no text it refuses.
• Cost: five shapes multiplying the points of resumption, at 3,000,000 characters, each in a subprocess bounded in time.

Before the fix.

• On main: the left-open test fails, and so do the 5 tests of the second commit.
• On the first commit: those 5 still fail.
• Already passing: the controls and the cost tests pass on main.

Mutations. Each of these makes at least one test fail:

• a point of resumption allowed inside a start tag;
• closes not taken for points;
• later tags not taken for points;
• inserted markups checked only before the first tag;
• a comment holding tags left open counted as ending in data;
• later tags listed again for each comment, caught by the cost tests of templates: constructs left open, and template tags inside comments, hide cross-site scripting; unclosed ones make the inspection quadratic #272.

Verification

Python 3.12, on main at 0da4cc8:

• ruff check . and mypy: clean on each commit;
• unit suite: 2,628, 2,642 and 2,642 passed, commit after commit;
• regression: 19 of 19, snapshots unchanged;
• Enterprise plugins: 978 passed.

This change and the fix of #279 touch different parts of taint/templates.py. Applied in either order, they give the same tree, on which ruff and mypy are clean, the unit suite passes (2,646) and the regression too (19 of 19).