[gdal-dev] Guidance for GeoJSONSeq Error 6: Too Large Feature

[Theo Linnemann]

>
> Hi There,
>
> I'm working with a set of fairly large geojsonl files, one for each state
> and territory in the United States. I'm using ogr2ogr to ingest this data
> into PostGIS, generally without issue, to speed up an expensive geospatial
> join.
>
> However, one of my files, specifically the Commonwealth of Pennsylvania,
> is about 6.5 GB and has been throwing an odd error that I can't seem to
> find much information on. "ERROR 6: Too large feature". Below is the
> command and output from running ogrinfo with the debug flag set to on,
> which returns the same error.
>
> I'm just hoping for some guidance in terms of what if anything I can do to
> sidestep the issue.
>
> So far, I've taken a look at what seems to be the GDAL source code that
> generates this error
> <https://github.com/OSGeo/gdal/blob/030ff40cf8340273bcc797e90c938cc32d14a34f/ogr/ogrsf_frmts/geojson/ogrgeojsonseqdriver.cpp#L410>,
> but it's unclear to me if it's possible to adjust the size constraint that
> is specified in the condition that throws the error or if the constraint is
> arbitrary or indicative of a larger problem with the file. I've already
> tried using the -gt flag with various sizes as well, but none seem to
> resolve the issue. I ran into an issue previously that required me to set
> OGR_GEOJSON_MAX_OBJ_SIZE=2048MB but that doesn't seem to have any effect
> here for some reason. Perhaps that only effects the GeoJSON driver and not
> the GeoJSONSeq driver? For what it's worth this is running on a node with
> ~350GB of RAM, so generally memory shouldn't be an issue.
>
> ogrinfo --version
> GDAL 3.5.0, released 2022/05/10
> ------------------------------------------------------
>
> ogrinfo --build
>
> PAM_ENABLED=YES
> OGR_ENABLED=YES
> GEOS_ENABLED=YES
> GEOS_VERSION=3.10.2-CAPI-1.16.0
> PROJ_BUILD_VERSION=9.0.0
> PROJ_RUNTIME_VERSION=9.0.0
> ------------------------------------------------------
>
> ogrinfo crs4326_fema_42_2022-07-22.geojsonl --debug ON
> GeoJSONSeq: First pass: 0.06 %
> GeoJSONSeq: First pass: 0.12 %
> GeoJSONSeq: First pass: 0.18 %
> ...
> GeoJSONSeq: First pass: 92.08 %
> GeoJSONSeq: First pass: 92.14 %
> GeoJSONSeq: First pass: 92.20 %
> ERROR 6: Too large feature
> GDAL: GDALOpen(crs4326_fema_42_2022-07-22.geojsonl, this=0x564966fae8b0)
> succeeds as GeoJSONSeq.
> INFO: Open of `crs4326_fema_42_2022-07-22.geojsonl'
>       using driver `GeoJSONSeq' successful.
> OGR: GetLayerCount() = 1
>
> 1: crs4326_fema_42_2022-07-22 (Multi Polygon)
> GDAL: GDALClose(crs4326_fema_42_2022-07-22.geojsonl, this=0x564966fae8b0)
> GDAL: In GDALDestroy - unloading GDAL shared library.
>
> Thank you for any help you're able to offer, it's much appreciated.
>
> TL
>
> Theo Linnemann <https://www.theolinnemann.com/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220808/a00f1828/attachment.htm>