[gdal-dev] Timestamp cast error in GDAL Parquet Directory with sqlite dialect

[Even Rouault]

Michael,

are you sure this is the *same* schema ? From the error messages (which 
comes from libarrow-compute itself), it would seem there's a mix of 
timestamp in microsecond and timestamp in nanosecond, and that the 
algorithm in libarrow tries to homogenize things and an overflow occur. 
If 237718454400000000 is a timestamp in microseconds, that corresponds 
to January 1st 9503...   I doubt the SQLite dialect plays any role in that.

Even

Le 26/07/2025 à 20:56, Michael Smith via gdal-dev a écrit :
>
> I have a collection of parquet files all with the same schema, 
> different stac collections written using geopandas to parquet.
>
> When I query at the cli or in python for a directory of parquert files 
> using sql I get timestamp casting errors
>
> gdal vector info -i PARQUET:s3://mybucket/stac/mds/rasters/ --sql 
> "select * from 'rasters' where st_intersects(geometry, 
> st_geomfromtext('POLYGON ((-68.00948853933728 17.7602787370086, 
> -64.99052950907739 17.7602787370086, -64.99052950907739 
> 18.6509945435268, -68.00948853933728 18.6509945435268, 
> -68.00948853933728 17.7602787370086))'))" --dialect sqlite -f text
>
> INFO: Open of `PARQUET:s3://grid-dev-publiclidar/stac/mds/rasters/'
>
>       using driver `Parquet' successful.
>
> Layer name: SELECT
>
> Geometry: Polygon
>
> ERROR 1: ReadNext() failed: Casting from timestamp[us, tz=UTC] to 
> timestamp[ns, tz=UTC] would result in out of bounds timestamp: 
> 237718454400000000
>
> Feature Count: 34
>
> ERROR 1: ReadNext() failed: Casting from timestamp[us, tz=UTC] to 
> timestamp[ns, tz=UTC] would result in out of bounds timestamp: 
> 237718454400000000
>
> Extent: (-180.000000, -90.000000) - (180.000000, 83.999167)
>
> If I do it file by file for all the parquet in a directory, I don’t 
> get an error.
>
> Is this a bug or a problem with sqlite dialect and parquet?
>
> -- 
>
> Michael Smith
>
> RSGIS Center – ERDC CRREL NH
>
> US Army Corps
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20250726/0da4f728/attachment.htm>