<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Theo,</p>
<p>you indeed spotted an hardcoded limit in the the GeoJSONSeq
driver. Addressed per <a class="moz-txt-link-freetext" href="https://github.com/OSGeo/gdal/pull/6177">https://github.com/OSGeo/gdal/pull/6177</a></p>
<p>Even<br>
</p>
<div class="moz-cite-prefix">Le 09/08/2022 à 05:20, Theo Linnemann a
écrit :<br>
</div>
<blockquote type="cite"
cite="mid:CAC7rD7hZ_DPyEMOqv_hxSvWw8ocssaU7NW4bbj-EGXXNYmTEHw@mail.gmail.com">
<div dir="auto">
<div>
<div class="gmail_quote">
<blockquote class="gmail_quote">
<div dir="ltr">Hi There,
<div><br>
</div>
<div>I'm working with a set of fairly large
geojsonl files, one for each state and territory in
the United States. I'm using ogr2ogr to ingest this
data into PostGIS, generally without issue, to speed
up an expensive geospatial join. </div>
<div><br>
</div>
<div>However, one of my files, specifically
the Commonwealth of Pennsylvania, is about 6.5 GB and
has been throwing an odd error that I can't seem to
find much information on. "ERROR 6: Too large
feature". Below is the command and output from running
ogrinfo with the debug flag set to on, which returns
the same error.</div>
<div><br>
</div>
<div>I'm just hoping for some guidance in terms of what
if anything I can do to sidestep the issue. </div>
<div><br>
</div>
<div>So far, I've taken a look at what seems to be the <a
href="https://github.com/OSGeo/gdal/blob/030ff40cf8340273bcc797e90c938cc32d14a34f/ogr/ogrsf_frmts/geojson/ogrgeojsonseqdriver.cpp#L410"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">GDAL source code that
generates this error</a>, but it's unclear to me if
it's possible to adjust the size constraint that is
specified in the condition that throws the error or if
the constraint is arbitrary or indicative of a larger
problem with the file. I've already tried using the
-gt flag with various sizes as well, but none seem to
resolve the issue. I ran into an issue previously that
required me to set <span>OGR_GEOJSON_MAX_OBJ_SIZE=2048MB</span> but
that doesn't seem to have any effect here for some
reason. Perhaps that only effects the GeoJSON driver
and not the GeoJSONSeq driver? For what it's worth
this is running on a node with ~350GB of RAM, so
generally memory shouldn't be an issue. </div>
<div><br>
</div>
<div>ogrinfo --version<br>
GDAL 3.5.0, released 2022/05/10<br>
</div>
<div>
<div>------------------------------------------------------</div>
<div><br>
</div>
</div>
<div>ogrinfo --build
</div>
<div>PAM_ENABLED=YES<br>
OGR_ENABLED=YES<br>
GEOS_ENABLED=YES<br>
GEOS_VERSION=3.10.2-CAPI-1.16.0<br>
PROJ_BUILD_VERSION=9.0.0<br>
PROJ_RUNTIME_VERSION=9.0.0<br>
</div>
<div>------------------------------------------------------</div>
<div><br>
</div>
<div>ogrinfo crs4326_fema_42_2022-07-22.geojsonl --debug
ON<br>
GeoJSONSeq: First pass: 0.06 %<br>
GeoJSONSeq: First pass: 0.12 %<br>
GeoJSONSeq: First pass: 0.18 %<br>
</div>
<div>...</div>
<div>GeoJSONSeq: First pass: 92.08 %<br>
GeoJSONSeq: First pass: 92.14 %<br>
GeoJSONSeq: First pass: 92.20 %<br>
ERROR 6: Too large feature<br>
GDAL: GDALOpen(crs4326_fema_42_2022-07-22.geojsonl,
this=0x564966fae8b0) succeeds as GeoJSONSeq.<br>
INFO: Open of `crs4326_fema_42_2022-07-22.geojsonl'<br>
using driver `GeoJSONSeq' successful.<br>
OGR: GetLayerCount() = 1<br>
<br>
1: crs4326_fema_42_2022-07-22 (Multi Polygon)<br>
GDAL: GDALClose(crs4326_fema_42_2022-07-22.geojsonl,
this=0x564966fae8b0)<br>
GDAL: In GDALDestroy - unloading GDAL shared library.<br>
</div>
<div><br>
</div>
<div>Thank you for any help you're able to offer, it's
much appreciated. </div>
<div><br>
</div>
<div>TL</div>
<div><br>
</div>
<div>
<div dir="ltr" data-smartmail="gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr"><a
href="https://www.theolinnemann.com/"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">Theo Linnemann</a></div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
gdal-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>
<a class="moz-txt-link-freetext" href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</body>
</html>