<div dir="auto"><div>Wow, thank you so much for the helpful response and quick fix!<div dir="auto"><br></div><div dir="auto">TL</div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 9, 2022, 11:38 AM Even Rouault <<a href="mailto:even.rouault@spatialys.com">even.rouault@spatialys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div>
    <p>Theo,</p>
    <p>you indeed spotted an hardcoded limit in the the GeoJSONSeq
      driver. Addressed per <a href="https://github.com/OSGeo/gdal/pull/6177" target="_blank" rel="noreferrer">https://github.com/OSGeo/gdal/pull/6177</a></p>
    <p>Even<br>
    </p>
    <div>Le 09/08/2022 à 05:20, Theo Linnemann a
      écrit :<br>
    </div>
    <blockquote type="cite">
      <div dir="auto">
        <div>
          <div class="gmail_quote">
            <blockquote class="gmail_quote">
              <div dir="ltr">Hi There, 
                <div><br>
                </div>
                <div>I'm working with a set of fairly large
                  geojsonl files, one for each state and territory in
                  the United States. I'm using ogr2ogr to ingest this
                  data into PostGIS, generally without issue, to speed
                  up an expensive geospatial join. </div>
                <div><br>
                </div>
                <div>However, one of my files, specifically
                  the Commonwealth of Pennsylvania, is about 6.5 GB and
                  has been throwing an odd error that I can't seem to
                  find much information on. "ERROR 6: Too large
                  feature". Below is the command and output from running
                  ogrinfo with the debug flag set to on, which returns
                  the same error.</div>
                <div><br>
                </div>
                <div>I'm just hoping for some guidance in terms of what
                  if anything I can do to sidestep the issue. </div>
                <div><br>
                </div>
                <div>So far, I've taken a look at what seems to be the <a href="https://github.com/OSGeo/gdal/blob/030ff40cf8340273bcc797e90c938cc32d14a34f/ogr/ogrsf_frmts/geojson/ogrgeojsonseqdriver.cpp#L410" rel="noreferrer noreferrer noreferrer" target="_blank">GDAL source code that
                    generates this error</a>, but it's unclear to me if
                  it's possible to adjust the size constraint that is
                  specified in the condition that throws the error or if
                  the constraint is arbitrary or indicative of a larger
                  problem with the file. I've already tried using the
                  -gt flag with various sizes as well, but none seem to
                  resolve the issue. I ran into an issue previously that
                  required me to set <span>OGR_GEOJSON_MAX_OBJ_SIZE=2048MB</span> but
                  that doesn't seem to have any effect here for some
                  reason. Perhaps that only effects the GeoJSON driver
                  and not the GeoJSONSeq driver? For what it's worth
                  this is running on a node with ~350GB of RAM, so
                  generally memory shouldn't be an issue. </div>
                <div><br>
                </div>
                <div>ogrinfo --version<br>
                  GDAL 3.5.0, released 2022/05/10<br>
                </div>
                <div>
                  <div>------------------------------------------------------</div>
                  <div><br>
                  </div>
                </div>
                <div>ogrinfo --build                                   
                                                                       
                                                                </div>
                <div>PAM_ENABLED=YES<br>
                  OGR_ENABLED=YES<br>
                  GEOS_ENABLED=YES<br>
                  GEOS_VERSION=3.10.2-CAPI-1.16.0<br>
                  PROJ_BUILD_VERSION=9.0.0<br>
                  PROJ_RUNTIME_VERSION=9.0.0<br>
                </div>
                <div>------------------------------------------------------</div>
                <div><br>
                </div>
                <div>ogrinfo crs4326_fema_42_2022-07-22.geojsonl --debug
                  ON<br>
                  GeoJSONSeq: First pass: 0.06 %<br>
                  GeoJSONSeq: First pass: 0.12 %<br>
                  GeoJSONSeq: First pass: 0.18 %<br>
                </div>
                <div>...</div>
                <div>GeoJSONSeq: First pass: 92.08 %<br>
                  GeoJSONSeq: First pass: 92.14 %<br>
                  GeoJSONSeq: First pass: 92.20 %<br>
                  ERROR 6: Too large feature<br>
                  GDAL: GDALOpen(crs4326_fema_42_2022-07-22.geojsonl,
                  this=0x564966fae8b0) succeeds as GeoJSONSeq.<br>
                  INFO: Open of `crs4326_fema_42_2022-07-22.geojsonl'<br>
                        using driver `GeoJSONSeq' successful.<br>
                  OGR: GetLayerCount() = 1<br>
                  <br>
                  1: crs4326_fema_42_2022-07-22 (Multi Polygon)<br>
                  GDAL: GDALClose(crs4326_fema_42_2022-07-22.geojsonl,
                  this=0x564966fae8b0)<br>
                  GDAL: In GDALDestroy - unloading GDAL shared library.<br>
                </div>
                <div><br>
                </div>
                <div>Thank you for any help you're able to offer, it's
                  much appreciated. </div>
                <div><br>
                </div>
                <div>TL</div>
                <div><br>
                </div>
                <div>
                  <div dir="ltr" data-smartmail="gmail_signature">
                    <div dir="ltr">
                      <div dir="ltr">
                        <div dir="ltr"><a href="https://www.theolinnemann.com/" rel="noreferrer noreferrer noreferrer" target="_blank">Theo Linnemann</a></div>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
            </blockquote>
          </div>
        </div>
      </div>
      <br>
      <fieldset></fieldset>
      <pre>_______________________________________________
gdal-dev mailing list
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank" rel="noreferrer">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank" rel="noreferrer">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
    </blockquote>
    <pre cols="72">-- 
<a href="http://www.spatialys.com" target="_blank" rel="noreferrer">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
  </div>

</blockquote></div></div></div>