[gdal-dev] Error when importing GeoPackage into PostgreSQL database ("Worker thread task has not expected m_iStartShapeId value")

[Even Rouault]

OK, I'm submitting https://github.com/OSGeo/gdal/pull/9018 to disable 
the multi-threaded mode by default, as I want to have reliable working 
in the 3.8.3 release ASAP. But I'm still interested in having that issue 
diagnose, to potentially re-enable multi-threading later. In 3.8.3, 
multi-threading in the ArrowArray interface of the GPKG driver will be 
opt-in through the newly added OGR_GPKG_NUM_THREADS configuration option 
/ environment variable.

Even

Le 02/01/2024 à 16:42, leo.fuhrmann.lgln at icloud.com a écrit :
> Unfortunately, I’m not allowed to share the GeoPackage at the moment. 
> I will get back to you as soon as I have permission or managed to 
> reduce the dataset accordingly.
>
> In the meantime, I found out, that the Error is not raised, if I use 
> e. g. the `-where` or `-sql` statement. Here is a command that doesn’t 
> produce an error, with the logs at the bottom:
>
> ```
> ogr2ogr --debug ON -f PostgreSQL "PG:dbname=XX host=XX port=XX user=XX 
> password=XX” <path/to/gpkg.gpkg> -where "1" <layer>
> ```
>
> Logs:
>
> GPKG: GeoPackage v1.2.0
>
> GDAL: GDALOpen(<path/to/gpkg.gpkg>, this=0x13a83b400) succeeds as GPKG.
>
> PG: Client encoding: 'UTF8'
>
> PG: PostGIS schema: 'public'
>
> PG: PostgreSQL version string : 'PostgreSQL 16.1 (Debian 
> 16.1-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 
> 10.2.1-6) 10.2.1 20210110, 64-bit'
>
> PG: PostGIS version string : '3.4 USE_GEOS=1 USE_PROJ=1 USE_STATS=1'
>
> GDAL: GDALOpen(PG:dbname=XX host=XX port=XX user=XX password=XX 
> this=0x13a631ee0) succeeds as PostgreSQL.
>
> GPKG: Filter: 1
>
> PG: Could not retrieve table oid for <layer>
>
> GDALVectorTranslate: Using FID=fid and -preserve_fid
>
> PG: Could not retrieve table oid for <layer>
>
> OGR2OGR: Using WriteArrowBatch()
>
> PG: PQputCopyEnd()
>
> GDAL: GDALClose(<path/to/gpkg.gpkg>, this=0x13a83b400)
>
> GDAL: GDALClose(PG:dbname=XX host=XX port=XX user=XX password=XX 
> this=0x13a631ee0)
>
> GDAL: In GDALDestroy - unloading GDAL shared library.
>
>
>
>> On Jan 2, 2024, at 15:57, Even Rouault <even.rouault at spatialys.com> 
>> wrote:
>>
>> Leo,
>>
>> could you provide your GeoPackage file and the layer you're 
>> converting ? (privately possibly)
>>
>> Even
>>
>> Le 02/01/2024 à 15:09, Leo Fuhrmann via gdal-dev a écrit :
>>> Hi,
>>>
>>> I’m using ogr2ogr 3.8.2 to import a (layer of a) GeoPackage into a 
>>> PostGIS database:
>>>
>>> ```
>>> ogr2ogr --debug ON -f PostgreSQL "PG:dbname=XX host=XX port=XX 
>>> user=XX password=XX” <path/to/gpkg.gpkg> <layer>
>>> ```
>>>
>>> However, I’m seeing an error in the logs that I cannot sort out (see 
>>> mail subject and/or logs below). Despite the error, the data gets 
>>> written to the database, yet I’m worried that something is messed up 
>>> along the way.
>>>
>>> Using the official docker image to execute the command (from a 
>>> MacBook with M1 Pro) works without an error. Then again, using a 
>>> cloud service (IBM Code Engine in this case) with the official image 
>>> also throws the error. I tried with different image versions but 
>>> found no difference.
>>>
>>> I tried to track down the error and found 
>>> https://github.com/OSGeo/gdal/blob/1df855eb676e17278c337e9b14774d1bb298df90/ogr/ogrsf_frmts/gpkg/ogrgeopackagetablelayer.cpp#L8352
>>>
>>> The comment suggests that I might have messed with GetNextFeature(), 
>>> but I’m sure I haven’t.
>>>
>>> Can someone help me to make sense of this error? I’m sorry to say 
>>> that I can’t share the GeoPackage at the moment.
>>>
>>> Thanks, Leo
>>>
>>>
>>>
>>> Here are the logs:
>>>
>>> GPKG: GeoPackage v1.2.0
>>> GDAL: GDALOpen(<path/to/gpkg.gpkg>, this=0x14c053e00) succeeds as GPKG.
>>> GDAL: GDALDriver::Create(PostgreSQL,PG:dbname=XX host=XX port=XX 
>>> user=XX password=XX
>>> PG: Client encoding: 'UTF8'
>>> PG: PostGIS schema: 'public'
>>> PG: PostgreSQL version string : 'PostgreSQL 16.1 (Debian 
>>> 16.1-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 
>>> 10.2.1-6) 10.2.1 20210110, 64-bit'
>>> PG: PostGIS version string : '3.4 USE_GEOS=1 USE_PROJ=1 USE_STATS=1'
>>> PG: Could not retrieve table oid for <layer>
>>> GDALVectorTranslate: Using FID=fid and -preserve_fid
>>> PG: Could not retrieve table oid for <layer>
>>> OGR2OGR: Using WriteArrowBatch()
>>> GPKG: GeoPackage v1.2.0
>>> GPKG: GeoPackage v1.2.0
>>> GPKG: GeoPackage v1.2.0
>>> ERROR 1: Worker thread task has not expected m_iStartShapeId value
>>> PG: PQputCopyEnd()
>>> GDAL: GDALClose(<path/to/gpkg.gpkg>, this=0x14c053e00)
>>> GDAL: GDALClose(PG:dbname=XX host=XX port=XX user=XX password=XX 
>>> this=0x14b721fb0)
>>> GDAL: In GDALDestroy - unloading GDAL shared library.
>>>
>>> _______________________________________________
>>> gdal-dev mailing list
>>> gdal-dev at lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>> -- 
>> http://www.spatialys.com
>> My software is free, but my time generally not.
>
-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240103/81384d2f/attachment-0001.htm>