[gdal-dev] Error when importing GeoPackage into PostgreSQL database ("Worker thread task has not expected m_iStartShapeId value")

[leo.fuhrmann.lgln at icloud.com]

Thank you for the quick responses. I have written a comment on the pull request you supplied in your last message with an example GeoPackage.

Leo

> On Jan 3, 2024, at 15:04, Even Rouault <even.rouault at spatialys.com> wrote:
> 
> OK, I'm submitting https://github.com/OSGeo/gdal/pull/9018 to disable the multi-threaded mode by default, as I want to have reliable working in the 3.8.3 release ASAP. But I'm still interested in having that issue diagnose, to potentially re-enable multi-threading later. In 3.8.3, multi-threading in the ArrowArray interface of the GPKG driver will be opt-in through the newly added OGR_GPKG_NUM_THREADS configuration option / environment variable.
> 
> Even
> 
> Le 02/01/2024 à 16:42, leo.fuhrmann.lgln at icloud.com <mailto:leo.fuhrmann.lgln at icloud.com> a écrit :
>> Unfortunately, I’m not allowed to share the GeoPackage at the moment. I will get back to you as soon as I have permission or managed to reduce the dataset accordingly.
>> 
>> In the meantime, I found out, that the Error is not raised, if I use e. g. the `-where` or `-sql` statement. Here is a command that doesn’t produce an error, with the logs at the bottom:
>> 
>> ```
>> ogr2ogr --debug ON -f PostgreSQL "PG:dbname=XX host=XX port=XX user=XX password=XX” <path/to/gpkg.gpkg> -where "1" <layer>
>> ```
>> 
>> Logs:
>> 
>> GPKG: GeoPackage v1.2.0
>> GDAL: GDALOpen(<path/to/gpkg.gpkg>, this=0x13a83b400) succeeds as GPKG.
>> PG: Client encoding: 'UTF8'
>> PG: PostGIS schema: 'public'
>> PG: PostgreSQL version string : 'PostgreSQL 16.1 (Debian 16.1-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit'
>> PG: PostGIS version string : '3.4 USE_GEOS=1 USE_PROJ=1 USE_STATS=1'
>> GDAL: GDALOpen(PG:dbname=XX host=XX port=XX user=XX password=XX this=0x13a631ee0) succeeds as PostgreSQL.
>> GPKG: Filter: 1
>> PG: Could not retrieve table oid for <layer>
>> GDALVectorTranslate: Using FID=fid and -preserve_fid
>> PG: Could not retrieve table oid for <layer>
>> OGR2OGR: Using WriteArrowBatch()
>> PG: PQputCopyEnd()
>> GDAL: GDALClose(<path/to/gpkg.gpkg>, this=0x13a83b400)
>> GDAL: GDALClose(PG:dbname=XX host=XX port=XX user=XX password=XX this=0x13a631ee0)
>> GDAL: In GDALDestroy - unloading GDAL shared library.
>> 
>> 
>>> On Jan 2, 2024, at 15:57, Even Rouault <even.rouault at spatialys.com> <mailto:even.rouault at spatialys.com> wrote:
>>> 
>>> Leo,
>>> 
>>> could you provide your GeoPackage file and the layer you're converting ? (privately possibly)
>>> 
>>> Even
>>> 
>>> Le 02/01/2024 à 15:09, Leo Fuhrmann via gdal-dev a écrit :
>>>> Hi,
>>>> 
>>>> I’m using ogr2ogr 3.8.2 to import a (layer of a) GeoPackage into a PostGIS database:
>>>> 
>>>> ```
>>>> ogr2ogr --debug ON -f PostgreSQL "PG:dbname=XX host=XX port=XX user=XX password=XX” <path/to/gpkg.gpkg> <layer>
>>>> ```
>>>> 
>>>> However, I’m seeing an error in the logs that I cannot sort out (see mail subject and/or logs below). Despite the error, the data gets written to the database, yet I’m worried that something is messed up along the way.
>>>> 
>>>> Using the official docker image to execute the command (from a MacBook with M1 Pro) works without an error. Then again, using a cloud service (IBM Code Engine in this case) with the official image also throws the error. I tried with different image versions but found no difference.
>>>> 
>>>> I tried to track down the error and found https://github.com/OSGeo/gdal/blob/1df855eb676e17278c337e9b14774d1bb298df90/ogr/ogrsf_frmts/gpkg/ogrgeopackagetablelayer.cpp#L8352
>>>> 
>>>> The comment suggests that I might have messed with GetNextFeature(), but I’m sure I haven’t.
>>>> 
>>>> Can someone help me to make sense of this error? I’m sorry to say that I can’t share the GeoPackage at the moment.
>>>> 
>>>> Thanks, Leo
>>>> 
>>>> 
>>>> 
>>>> Here are the logs:
>>>> 
>>>> GPKG: GeoPackage v1.2.0
>>>> GDAL: GDALOpen(<path/to/gpkg.gpkg>, this=0x14c053e00) succeeds as GPKG.
>>>> GDAL: GDALDriver::Create(PostgreSQL,PG:dbname=XX host=XX port=XX user=XX password=XX
>>>> PG: Client encoding: 'UTF8'
>>>> PG: PostGIS schema: 'public'
>>>> PG: PostgreSQL version string : 'PostgreSQL 16.1 (Debian 16.1-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit'
>>>> PG: PostGIS version string : '3.4 USE_GEOS=1 USE_PROJ=1 USE_STATS=1'
>>>> PG: Could not retrieve table oid for <layer>
>>>> GDALVectorTranslate: Using FID=fid and -preserve_fid
>>>> PG: Could not retrieve table oid for <layer>
>>>> OGR2OGR: Using WriteArrowBatch()
>>>> GPKG: GeoPackage v1.2.0
>>>> GPKG: GeoPackage v1.2.0
>>>> GPKG: GeoPackage v1.2.0
>>>> ERROR 1: Worker thread task has not expected m_iStartShapeId value
>>>> PG: PQputCopyEnd()
>>>> GDAL: GDALClose(<path/to/gpkg.gpkg>, this=0x14c053e00)
>>>> GDAL: GDALClose(PG:dbname=XX host=XX port=XX user=XX password=XX this=0x14b721fb0)
>>>> GDAL: In GDALDestroy - unloading GDAL shared library.
>>>> 
>>>> 
>>>> _______________________________________________
>>>> gdal-dev mailing list
>>>> gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>
>>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>> -- 
>>> http://www.spatialys.com <http://www.spatialys.com/>
>>> My software is free, but my time generally not.
>> 
> -- 
> http://www.spatialys.com <http://www.spatialys.com/>
> My software is free, but my time generally not.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240104/e77e7a8e/attachment-0001.htm>