[SCM] PostGIS branch master updated. 3.7.0beta2-44-ga87600d95
git at osgeo.org
git at osgeo.org
Sat Aug 15 09:37:21 PDT 2026
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "PostGIS".
The branch, master has been updated
via a87600d95611a7c22487f0fa01155bd2fe17f6a2 (commit)
via 43e13fe2235e36d5710b83566e33754ed3ae4105 (commit)
from 89cd60efddcd85323dd9ed9ec33cd3a87dbf1091 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
commit a87600d95611a7c22487f0fa01155bd2fe17f6a2
Merge: 89cd60efd 43e13fe22
Author: Darafei Praliaskouski <komzpa at gmail.com>
Date: Sat Aug 15 09:37:19 2026 -0700
Merge pull request 'docs: allow DOCTYPE without entities in docbook QA parser' (!735) from Komzpa/postgis:fix/docbook-doctype-selfbuild-20260815 into master
`utils/docs/xml_tree.py` rejected any document containing the literal byte sequence `<!DOCTYPE` before parsing. That guard was added in [PR 719](https://gitea.osgeo.org/postgis/postgis/pulls/719) to close an XXE hole, but it also rejects our own `doc/postgis-out.xml`: xmllint always emits a DocBook DTD DOCTYPE for it, and that DOCTYPE's internal subset carries dozens of already-resolved `SYSTEM` and literal entity declarations left over from assembling `doc/*.xml` into one file at build time. Every `check-docbook-markup`, `lint-html`, `ref-index`, and `postgis_exampletest.py` invocation on master therefore failed on the project's own generated output (first broken pipeline #7002 on Woodpecker). The same byte scan also false-positives on documentation body content that happens to contain the text `<!DOCTYPE`, such as the X3D example in `reference_output.xml`, which sits inside a CDATA section and is not a real prolog DOCTYPE at all.
This replaces the byte scan with an entity-aware guard. A lightweight expat pass collects the names of literal-value ("internal") general entities the DOCTYPE declares, without an `ExternalEntityRefHandler` and without parameter-entity parsing, so nothing external is fetched during this pass either. If the document body actually references one of those internal entity names, parsing is rejected before the real tree-building parse runs; a harmless DOCTYPE whose declarations are unreferenced, which is exactly the shape our own generated corpus has, now parses normally. External entities were already never resolved via the existing `feature_external_ges`/`feature_external_pes` settings; `parse()` now also installs an `EntityResolver` that returns empty content for any external identifier, so the DocBook DTD and any external general entity stay unresolved even if a given libxml2/expat build ignores the feature flags.
Two regression tests added alongside the existing one: a DOCTYPE shaped like `postgis-out.xml`'s, with declared but unreferenced entities, must parse; a DOCTYPE that declares and references an entity must still be rejected.
Validated by regenerating `postgis-out.xml` exactly as `doc/Makefile.in` does (`xmllint --noent`) and running the `Makefile:815` invocation directly, plus `postgis_exampletest.py --report` and `docbook_qa.py ref-index` against the same file; all pass, and a synthetic XXE document does not leak file content. `utils/docs/xml_tree.py` is master-only (absent on `stable-3.6`), so no NEWS entry and no backpatch.
References https://gitea.osgeo.org/postgis/postgis/pulls/719
Reviewed-on: https://gitea.osgeo.org/postgis/postgis/pulls/735
commit 43e13fe2235e36d5710b83566e33754ed3ae4105
Author: Darafei Praliaskouski <me at komzpa.net>
Date: Sat Aug 15 20:16:15 2026 +0400
docs: allow DOCTYPE without entities in docbook QA parser
utils/docs/xml_tree.py rejected any document containing the literal byte
sequence "<!DOCTYPE" before parsing. That guard was added to close an XXE
hole, but it also rejects our own doc/postgis-out.xml: xmllint always emits a
DocBook DTD DOCTYPE for it, and that DOCTYPE's internal subset carries dozens
of already-resolved SYSTEM and literal entity declarations left over from
assembling doc/*.xml into one file at build time. Every check-docbook-markup,
lint-html, ref-index, and postgis_exampletest.py invocation on master
therefore failed on the project's own generated output. The same byte scan
also false-positives on documentation body content that happens to contain
the text "<!DOCTYPE", such as the X3D example in reference_output.xml, which
is protected by a CDATA section and is not a real prolog DOCTYPE at all.
Replace the byte scan with an entity-aware guard. A lightweight expat pass
collects the names of literal-value ("internal") general entities the
DOCTYPE declares, without an ExternalEntityRefHandler and without enabling
parameter-entity parsing, so nothing external is ever fetched during this
pass either. If the document body actually references one of those internal
entity names, parsing is rejected before the real tree-building parse runs;
a harmless DOCTYPE whose declarations are unreferenced, which is exactly the
shape our own generated corpus has, now parses normally. External entities
were already never resolved via the existing feature_external_ges and
feature_external_pes settings; parse() now also installs an EntityResolver
that returns empty content for any external SYSTEM or PUBLIC identifier, so
the DocBook DTD and any external general entity stay unresolved even if a
given libxml2/expat build ignores the feature flags.
Added two regression tests to utils/docs/tests/test_docbook_qa.py alongside
the existing one: a DOCTYPE shaped like postgis-out.xml's, with declared but
unreferenced entities, must parse; a DOCTYPE that declares and references an
entity must still be rejected.
References https://gitea.osgeo.org/postgis/postgis/pulls/719
diff --git a/utils/docs/tests/test_docbook_qa.py b/utils/docs/tests/test_docbook_qa.py
index aa550da77..2f4b07aa1 100644
--- a/utils/docs/tests/test_docbook_qa.py
+++ b/utils/docs/tests/test_docbook_qa.py
@@ -68,6 +68,27 @@ class DocBookSourceLintTest(unittest.TestCase):
with self.assertRaises(Exception):
parse_xml(path)
+ def test_xml_tree_allows_doctype_without_entity_references(self):
+ # postgis-out.xml itself carries a DOCTYPE: xmllint emits the DocBook DTD's
+ # public identifier plus dozens of unreferenced SYSTEM/literal entity
+ # declarations left over from assembling doc/*.xml at build time. None of
+ # that is an attack when nothing in the document body asks the parser to
+ # expand any of it, so a harmless DOCTYPE like this one must still parse.
+ path = write_tmp(
+ ".xml",
+ '<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML 5.0//EN" '
+ '"http://docbook.org/xml/5.0/dtd/docbook.dtd" ['
+ '<!ENTITY last_release_version "3.6.0">'
+ '<!ENTITY introduction SYSTEM "introduction.xml">'
+ ']>'
+ + DOCBOOK_OPEN
+ + '<para>no custom entity is referenced here</para>'
+ + DOCBOOK_CLOSE,
+ )
+
+ tree = parse_xml(path)
+ self.assertEqual("book", tree.tree.getroot().tag.rsplit("}", 1)[-1])
+
def test_mixed_programlisting_dash_run_markers_and_sql_comments(self):
for output in ("left | right\n----|----\n1 | 2", "----RESULT output ---\n1"):
with self.subTest(output=output):
diff --git a/utils/docs/xml_tree.py b/utils/docs/xml_tree.py
index 042a71be6..502dcbecb 100644
--- a/utils/docs/xml_tree.py
+++ b/utils/docs/xml_tree.py
@@ -2,11 +2,14 @@
from __future__ import annotations
+import re
from dataclasses import dataclass
from pathlib import Path
import xml.etree.ElementTree as ET
+import xml.parsers.expat
import xml.sax
-from xml.sax.handler import ContentHandler, feature_external_ges, feature_external_pes, feature_namespaces
+from xml.sax.handler import ContentHandler, EntityResolver, feature_external_ges, feature_external_pes, feature_namespaces
+from xml.sax.xmlreader import InputSource
@dataclass
@@ -75,21 +78,62 @@ class _TreeBuilder(ContentHandler):
current.text = (current.text or "") + content
-def _reject_doctype(path):
+_GENERAL_ENTITY_REFERENCE_RE = re.compile(rb"&([A-Za-z_][\w.-]*);")
+
+
+def _internal_general_entity_names(path):
+ """Return the names of literal-value ("internal") general entities the
+ document's DOCTYPE declares, without resolving or fetching anything external.
+
+ Our own generated postgis-out.xml carries a DOCTYPE whose internal subset
+ declares dozens of SYSTEM entities; xmllint uses those only to assemble
+ doc/*.xml into one file at build time, so by the time this module sees the
+ output they are unreferenced text in the DOCTYPE. They are external, so
+ ExternalEntityRefHandler is left unset here and they are never resolved or
+ counted. Only an entity with a literal replacement value can inject
+ attacker-controlled text or drive an entity-expansion blowup if the document
+ goes on to reference it, so only those names are worth tracking.
+ """
+ names = set()
+
+ def entity_decl(name, is_parameter_entity, value, _base, _system_id, _public_id, _notation_name):
+ if value is not None and not is_parameter_entity:
+ names.add(name)
+
+ parser = xml.parsers.expat.ParserCreate()
+ parser.EntityDeclHandler = entity_decl
with Path(path).open("rb") as handle:
- previous = b""
- while True:
- chunk = handle.read(8192)
- if not chunk:
- return
- haystack = previous + chunk.upper()
- if b"<!DOCTYPE" in haystack:
- raise xml.sax.SAXException("DOCTYPE declarations are not supported")
- previous = haystack[-8:]
+ parser.ParseFile(handle)
+ return names
+
+
+def _reject_internal_entity_references(path):
+ entity_names = _internal_general_entity_names(path)
+ if not entity_names:
+ return
+ data = Path(path).read_bytes()
+ for match in _GENERAL_ENTITY_REFERENCE_RE.finditer(data):
+ name = match.group(1).decode("ascii", "replace")
+ if name in entity_names:
+ raise xml.sax.SAXException(f"entity references are not supported: &{name};")
+
+
+class _EmptyEntityResolver(EntityResolver):
+ """Refuse to fetch an external entity's replacement content.
+
+ feature_external_ges/feature_external_pes below already tell a conformant
+ parser not to resolve external entities, but that support is best-effort
+ (see the try/except immediately below). Overriding resolveEntity makes the
+ refusal unconditional: any external SYSTEM/PUBLIC identifier, including the
+ DocBook DTD itself, resolves to an empty document instead of a real fetch.
+ """
+
+ def resolveEntity(self, _public_id, _system_id):
+ return InputSource()
def parse(path):
- _reject_doctype(path)
+ _reject_internal_entity_references(path)
parser = xml.sax.make_parser()
parser.setFeature(feature_namespaces, True)
for feature in (feature_external_ges, feature_external_pes):
@@ -97,6 +141,7 @@ def parse(path):
parser.setFeature(feature, False)
except (xml.sax.SAXNotRecognizedException, xml.sax.SAXNotSupportedException):
pass
+ parser.setEntityResolver(_EmptyEntityResolver())
builder = _TreeBuilder()
parser.setContentHandler(builder)
parser.parse(str(Path(path)))
-----------------------------------------------------------------------
Summary of changes:
utils/docs/tests/test_docbook_qa.py | 21 +++++++++++
utils/docs/xml_tree.py | 69 ++++++++++++++++++++++++++++++-------
2 files changed, 78 insertions(+), 12 deletions(-)
hooks/post-receive
--
PostGIS
More information about the postgis-tickets
mailing list