[gdal-dev] Guidance for GeoJSONSeq Error 6: Too Large Feature

[Theo Linnemann]

Wow, thank you so much for the helpful response and quick fix!

TL


On Tue, Aug 9, 2022, 11:38 AM Even Rouault <even.rouault at spatialys.com>
wrote:

> Theo,
>
> you indeed spotted an hardcoded limit in the the GeoJSONSeq driver.
> Addressed per https://github.com/OSGeo/gdal/pull/6177
>
> Even
> Le 09/08/2022 à 05:20, Theo Linnemann a écrit :
>
> Hi There,
>>
>> I'm working with a set of fairly large geojsonl files, one for each state
>> and territory in the United States. I'm using ogr2ogr to ingest this data
>> into PostGIS, generally without issue, to speed up an expensive geospatial
>> join.
>>
>> However, one of my files, specifically the Commonwealth of Pennsylvania,
>> is about 6.5 GB and has been throwing an odd error that I can't seem to
>> find much information on. "ERROR 6: Too large feature". Below is the
>> command and output from running ogrinfo with the debug flag set to on,
>> which returns the same error.
>>
>> I'm just hoping for some guidance in terms of what if anything I can do
>> to sidestep the issue.
>>
>> So far, I've taken a look at what seems to be the GDAL source code that
>> generates this error
>> <https://github.com/OSGeo/gdal/blob/030ff40cf8340273bcc797e90c938cc32d14a34f/ogr/ogrsf_frmts/geojson/ogrgeojsonseqdriver.cpp#L410>,
>> but it's unclear to me if it's possible to adjust the size constraint that
>> is specified in the condition that throws the error or if the constraint is
>> arbitrary or indicative of a larger problem with the file. I've already
>> tried using the -gt flag with various sizes as well, but none seem to
>> resolve the issue. I ran into an issue previously that required me to set
>> OGR_GEOJSON_MAX_OBJ_SIZE=2048MB but that doesn't seem to have any effect
>> here for some reason. Perhaps that only effects the GeoJSON driver and not
>> the GeoJSONSeq driver? For what it's worth this is running on a node with
>> ~350GB of RAM, so generally memory shouldn't be an issue.
>>
>> ogrinfo --version
>> GDAL 3.5.0, released 2022/05/10
>> ------------------------------------------------------
>>
>> ogrinfo --build
>>
>>
>> PAM_ENABLED=YES
>> OGR_ENABLED=YES
>> GEOS_ENABLED=YES
>> GEOS_VERSION=3.10.2-CAPI-1.16.0
>> PROJ_BUILD_VERSION=9.0.0
>> PROJ_RUNTIME_VERSION=9.0.0
>> ------------------------------------------------------
>>
>> ogrinfo crs4326_fema_42_2022-07-22.geojsonl --debug ON
>> GeoJSONSeq: First pass: 0.06 %
>> GeoJSONSeq: First pass: 0.12 %
>> GeoJSONSeq: First pass: 0.18 %
>> ...
>> GeoJSONSeq: First pass: 92.08 %
>> GeoJSONSeq: First pass: 92.14 %
>> GeoJSONSeq: First pass: 92.20 %
>> ERROR 6: Too large feature
>> GDAL: GDALOpen(crs4326_fema_42_2022-07-22.geojsonl, this=0x564966fae8b0)
>> succeeds as GeoJSONSeq.
>> INFO: Open of `crs4326_fema_42_2022-07-22.geojsonl'
>>       using driver `GeoJSONSeq' successful.
>> OGR: GetLayerCount() = 1
>>
>> 1: crs4326_fema_42_2022-07-22 (Multi Polygon)
>> GDAL: GDALClose(crs4326_fema_42_2022-07-22.geojsonl, this=0x564966fae8b0)
>> GDAL: In GDALDestroy - unloading GDAL shared library.
>>
>> Thank you for any help you're able to offer, it's much appreciated.
>>
>> TL
>>
>> Theo Linnemann <https://www.theolinnemann.com/>
>>
>
> _______________________________________________
> gdal-dev mailing listgdal-dev at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220809/778e3fab/attachment-0001.htm>