[gdal-dev] Read a /vsigzip/ csv.gz all at once

[Even Rouault]

Hi,

I don't know anything about Julia but I'd suspect that there must be 
something particularly slow in the way it interacts with C. For 
comparison,  "time python3 
swig/python/gdal-utils/osgeo_utils/samples/ogrinfo.py 
/vsigzip//vsicurl/https://bulk.meteostat.net/v2/hourly/2022/08554.csv.gz 
-al > /dev/null" that does essentially your loop, and also prints on 
stdout, runs in 1.5 seconds (compared to native ogrinfo that runs in 0.7 
s). Perhaps you could write a Julia wrapper to get all fields of feature 
at once and return whatever dictionary or equivalent data structure is 
idiomatic (and efficient )in Julia ? Also are you sure your Julia 
wrapper is built with optimization enabled?

Even

Le 24/06/2025 à 16:33, Joaquim Manuel Freire Luís via gdal-dev a écrit :
>
> Hi,
>
> Im trying to read files like
>
> https://bulk.meteostat.net/v2/hourly/2022/08554.csv.gz
>
> in my Julia wrapper. The point is that, although I’m kind off 
> succeeding, the hole operation is very slow.
>
> What I’m doing (code not committed yet so can’t post a link) is to 
> read like this
>
> layer = getlayer(dataset, 0)
>
> for f in layer
>
>                for k = 1: Gdal.nfield(f)
>
> Gdal.getfield(f, k-1)
>
> …
>
> This works but it’s extremely slow because each “getfield” takes about 
> 1e-4 seconds and the file has ~8 k rows, each with 13 fields. That 
> amounts to > 10 sec.
>
> I’ve searched but couldn’t find a way to read the entire file at once 
> (which takes 1e-2 seconds if I read it, locally, with a gzip wrapper) 
> and return it as a single string array that I could parse later.
>
> Is that possible?
>
> Thanks
>
> Joaquim
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20250624/6f02c92f/attachment.htm>