[gdal-dev] Issue converting geo parquet data

[Even Rouault]

Hi Michael,

This error comes very deep inside libparquet, actually in libthrift. The 
more natural assumption would be that it would be due to a corrupted 
Parquet file. If you disabled multithreading (GDAL_NUM_THREADS=1) and 
enabled --debug on, perhaps this would happen on the same file ? I was 
wondering also if that couldn't come from RAM exhaustion, but that 
doesn't seem likely (did you monitor the RAM consumption?). It would be 
interesting to see if that would also occur after fetching locally the 
files under overturemaps-us-west-2/release/2024-07-22.0/theme=buildings 
and running the conversion from the local files

Even

Le 30/07/2024 à 20:37, Michael Smith via gdal-dev a écrit :
>
> Hi all,
>
> I was converting the overture maps parquet data to geopackage and got 
> this error using gdal master (via conda).
>
> GDAL_NUM_THREADS=ALL_CPUS CPL_TMPDIR=/data2 ogr2ogr -f gpkg 
> /data/overture_buildings.gpkg 
> /vsis3/overturemaps-us-west-2/release/2024-07-22.0/theme=buildings 
> "theme=buildings"  -progress  -nlt PROMOTE_TO_MULTI -nln buildings 
> -skipfailures
>
> ERROR 1: ReadNext() failed: Couldn't deserialize thrift: 
> TProtocolException: Exceeded size limit
>
> Not sure what size limit it exceeded? I’ve had this happen several 
> times but at very different points in the process.
>
> Ideas?
>
> Mike
>
> -- 
>
> Michael Smith
>
> US Army Corps of Engineers
>
> Remote Sensing/GIS Center
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240730/19d47e57/attachment.htm>