[gdal-dev] Guidance for GeoJSONSeq Error 6: Too Large Feature

[Even Rouault]

Theo,

you indeed spotted an hardcoded limit in the the GeoJSONSeq driver. 
Addressed per https://github.com/OSGeo/gdal/pull/6177

Even

Le 09/08/2022 à 05:20, Theo Linnemann a écrit :
>
>     Hi There,
>
>     I'm working with a set of fairly large geojsonl files, one for
>     each state and territory in the United States. I'm using ogr2ogr
>     to ingest this data into PostGIS, generally without issue, to
>     speed up an expensive geospatial join.
>
>     However, one of my files, specifically the Commonwealth of
>     Pennsylvania, is about 6.5 GB and has been throwing an odd error
>     that I can't seem to find much information on. "ERROR 6: Too large
>     feature". Below is the command and output from running ogrinfo
>     with the debug flag set to on, which returns the same error.
>
>     I'm just hoping for some guidance in terms of what if anything I
>     can do to sidestep the issue.
>
>     So far, I've taken a look at what seems to be the GDAL source code
>     that generates this error
>     <https://github.com/OSGeo/gdal/blob/030ff40cf8340273bcc797e90c938cc32d14a34f/ogr/ogrsf_frmts/geojson/ogrgeojsonseqdriver.cpp#L410>,
>     but it's unclear to me if it's possible to adjust the size
>     constraint that is specified in the condition that throws the
>     error or if the constraint is arbitrary or indicative of a larger
>     problem with the file. I've already tried using the -gt flag with
>     various sizes as well, but none seem to resolve the issue. I ran
>     into an issue previously that required me to set
>     OGR_GEOJSON_MAX_OBJ_SIZE=2048MB but that doesn't seem to have any
>     effect here for some reason. Perhaps that only effects the GeoJSON
>     driver and not the GeoJSONSeq driver? For what it's worth this is
>     running on a node with ~350GB of RAM, so generally memory
>     shouldn't be an issue.
>
>     ogrinfo --version
>     GDAL 3.5.0, released 2022/05/10
>     ------------------------------------------------------
>
>     ogrinfo --build
>     PAM_ENABLED=YES
>     OGR_ENABLED=YES
>     GEOS_ENABLED=YES
>     GEOS_VERSION=3.10.2-CAPI-1.16.0
>     PROJ_BUILD_VERSION=9.0.0
>     PROJ_RUNTIME_VERSION=9.0.0
>     ------------------------------------------------------
>
>     ogrinfo crs4326_fema_42_2022-07-22.geojsonl --debug ON
>     GeoJSONSeq: First pass: 0.06 %
>     GeoJSONSeq: First pass: 0.12 %
>     GeoJSONSeq: First pass: 0.18 %
>     ...
>     GeoJSONSeq: First pass: 92.08 %
>     GeoJSONSeq: First pass: 92.14 %
>     GeoJSONSeq: First pass: 92.20 %
>     ERROR 6: Too large feature
>     GDAL: GDALOpen(crs4326_fema_42_2022-07-22.geojsonl,
>     this=0x564966fae8b0) succeeds as GeoJSONSeq.
>     INFO: Open of `crs4326_fema_42_2022-07-22.geojsonl'
>           using driver `GeoJSONSeq' successful.
>     OGR: GetLayerCount() = 1
>
>     1: crs4326_fema_42_2022-07-22 (Multi Polygon)
>     GDAL: GDALClose(crs4326_fema_42_2022-07-22.geojsonl,
>     this=0x564966fae8b0)
>     GDAL: In GDALDestroy - unloading GDAL shared library.
>
>     Thank you for any help you're able to offer, it's much appreciated.
>
>     TL
>
>     Theo Linnemann <https://www.theolinnemann.com/>
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220809/ee34fef9/attachment.htm>