[gdal-dev] GDAL 3.8.0beta1 available for testing

[Rahkonen Jukka]

Hi,

I made a simple test with ogr2ogr and geopackage to geopackage on Windows. With GDAL 3.8.0dev-3e4dc710a2 (no arrow, old R-Tree) the timing was 36 minutes, with GDAL 3.8.0dev-6bbd2c080a the same conversion took 21 minutes. The gpkg file is 75 GB, with 120 layers.

Could it be that the new R-Tree implementation does not print anything into debug? With older GDAL I see
GPKG: Creating insert/delete feature_count triggers
GPKG: 194505 rows inserted into rtree_virtausnuoli_geom
but with the brand new just
GPKG: Creating insert/delete feature_count triggers
GPKG: Creating insert/delete feature_count triggers

-Jukka Rahkonen-




-----Alkuperäinen viesti-----
Lähettäjä: gdal-dev <gdal-dev-bounces at lists.osgeo.org> Puolesta Even Rouault via gdal-dev
Lähetetty: tiistai 31. lokakuuta 2023 16.33
Vastaanottaja: gdal-dev at lists.osgeo.org
Aihe: [gdal-dev] GDAL 3.8.0beta1 available for testing

Hi,

I've prepared a beta1 of GDAL 3.8.0 to get feedback from earlier testers.

Sorry no updated NEWS.md file yet, but I'd in particular be interesting by testing of ogr2ogr workflows, since they have underwent significant changes in the underlying implementation:

- when the source layer is a layer implementing the ArrowStream API (that is  GeoPackage, FlatGeoBuf, Arrow or Parquet), and when no ogr2ogr options than -of, -where, -spat, -lco, -dsco, -gt, -append/-overwrite/-update are used (and -sql as well for GeoPackage).
When enabling CPL_DEBUG, you'll see a "OGR2OGR: Using WriteArrowBatch()"
trace when that new code path is taken. If specifying other options, the feature-iteration-based traditional implementation is used

- and/or when the output layer is GeoPackage (new layer), due to the revamped much faster spatial index creation.  This enhanced spatial index creation is not ogr2ogr specific and is actually available more generally for CreateLayer() + CreateFeature() or CreateLayer() +
WriteArrowBatch() scenarios.

Point of attention would be when in situations with large files and/or with low RAM.

The ghcr.io/osgeo/gdal:ubuntu-small-latest,
ghcr.io/osgeo/gdal:ubuntu-full-latest,
ghcr.io/osgeo/gdal:alpine-normal-latest Docker images have been refreshed with 3.8.0beta1 (ghcr.io/osgeo/gdal:alpine-small-latest still building at time of writing).

(Note: the GDAL master conda builds mentioned at
https://gdal.org/download.html#gdal-master-conda-builds have been broken for a couple weeks and are thus not usable to test beta1 currently. I'm
investigating)

Source snapshots at:

- https://download.osgeo.org/gdal/3.8.0/gdal-3.8.0beta1.tar.gz

- https://download.osgeo.org/gdal/3.8.0/gdal-3.8.0beta1.tar.xz

- https://download.osgeo.org/gdal/3.8.0/gdal380beta1.zip

Autotest snapshots:

- https://download.osgeo.org/gdal/3.8.0/gdalautotest-3.8.0beta1.tar.gz

- https://download.osgeo.org/gdal/3.8.0/gdalautotest-3.8.0beta1.zip

Even

--
http://www.spatialys.com/
My software is free, but my time generally not.

_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev