<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle26
{mso-style-type:personal-compose;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1576354578;
mso-list-type:hybrid;
mso-list-template-ids:700073180 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">To finish this. The problem was in the GMT.jl wrapper.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Short history.<o:p></o:p></span></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1"><span style="font-size:11.0pt">Julia 1.9 started to crash with some GMT.jl tests<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1"><span style="font-size:11.0pt">Opened an issue (<a href="https://github.com/JuliaLang/julia/issues/47003">https://github.com/JuliaLang/julia/issues/47003</a>) but got little-to-none
help<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1"><span style="font-size:11.0pt">Found a patch for the situation (that seemed innocuous)<o:p></o:p></span></li></ol>
<p class="MsoListParagraph"><span style="font-size:11.0pt"><a href="https://github.com/GenericMappingTools/GMT.jl/blob/master/src/gdal.jl#L1953">https://github.com/GenericMappingTools/GMT.jl/blob/master/src/gdal.jl#L1953</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">but that patch ended up causing this extreme slowdown. Since Julia doesn’t crash anymore, I removed that patch and now<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="FI" style="font-size:11.0pt">julia> @time gdalread("/vsigzip//vsicurl/https://bulk.meteostat.net/v2/hourly/2022/08554.csv.gz");<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 1.007115 seconds (130.95 k allocations: 3.804 MiB)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks for the discussion that helped a lot figuring out the problem.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Joaquim<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> gdal-dev <gdal-dev-bounces@lists.osgeo.org>
<b>On Behalf Of </b>Joaquim Manuel Freire Luís via gdal-dev<br>
<b>Sent:</b> Tuesday, June 24, 2025 6:19 PM<br>
<b>To:</b> Erik Schnetter <schnetter@gmail.com>; gdal-dev@lists.osgeo.org<br>
<b>Subject:</b> Re: [gdal-dev] Read a /vsigzip/ csv.gz all at once<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Even<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">In case you are spending any time on this, please do not. It this time I’m persuaded that this is a Juia wrapper(s) issue but have no time to investigating it much more right now.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Joaquim Manuel Freire Luís <<a href="mailto:jluis@ualg.pt">jluis@ualg.pt</a>>
<br>
<b>Sent:</b> Tuesday, June 24, 2025 5:52 PM<br>
<b>To:</b> Joaquim Manuel Freire Luís <<a href="mailto:jluis@ualg.pt">jluis@ualg.pt</a>>; Erik Schnetter <<a href="mailto:schnetter@gmail.com">schnetter@gmail.com</a>>;
<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
<b>Subject:</b> RE: [gdal-dev] Read a /vsigzip/ csv.gz all at once<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Erik, BINGO.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Since, in this case, I know that the field type is string, if I replace the calls to getfield(…) to<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">OGR_F_GetFieldAsString(f.ptr, k)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I get these timings (on a local file)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">julia> @time gdalread("/vsigzip/C:/TMP/.meteostat/cache/hourly/2025/08554.csv.gz");<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 0.046272 seconds (56.04 k allocations: 1.751 MiB)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">and, for a remote one<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="FI" style="font-size:11.0pt">julia> @time gdalread("/vsigzip//vsicurl/https://bulk.meteostat.net/v2/hourly/2022/08554.csv.gz");<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> 1.191215 seconds (113.43 k allocations: 3.537 MiB)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> gdal-dev <<a href="mailto:gdal-dev-bounces@lists.osgeo.org">gdal-dev-bounces@lists.osgeo.org</a>>
<b>On Behalf Of </b>Joaquim Manuel Freire Luís via gdal-dev<br>
<b>Sent:</b> Tuesday, June 24, 2025 5:33 PM<br>
<b>To:</b> Erik Schnetter <<a href="mailto:schnetter@gmail.com">schnetter@gmail.com</a>>;
<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
<b>Subject:</b> Re: [gdal-dev] Read a /vsigzip/ csv.gz all at once<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">To complement what Eric said, here’s the ‘getfield’ function (This code was taken from ArchGDAL so we are talking of the same thing)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="https://github.com/GenericMappingTools/GMT.jl/blob/master/src/gdal.jl#L2161">https://github.com/GenericMappingTools/GMT.jl/blob/master/src/gdal.jl#L2161</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> gdal-dev <<a href="mailto:gdal-dev-bounces@lists.osgeo.org">gdal-dev-bounces@lists.osgeo.org</a>>
<b>On Behalf Of </b>Erik Schnetter via gdal-dev<br>
<b>Sent:</b> Tuesday, June 24, 2025 5:21 PM<br>
<b>To:</b> <a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
<b>Subject:</b> Re: [gdal-dev] Read a /vsigzip/ csv.gz all at once<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The Julia wrapper (ArchGDAL.jl) for `getfield` calls `OGR_FD_GetFieldDefn` and several related function (to get the type of the field etc.). Are these possibly expensive operations in GDAL?<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Any C function in GDAL can easily be called from Julia. Which C function would get all fields at once? I assume that e.g. `OGR_F_GetFieldAsDoubleList` would not work; this would be for values that are themselves lists?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The Julia code for `getfield` spends quite a bit of work to find out the type of the field. This includes a bit of reference counting, allocating small structures on the heap, registering finalizers for them etc. This could be avoided by
adding a Julia wrapper that calls `getfield` repeatedly (even from Julia, calling C has no overhead by itself) for a range of integers. This would avoid the additional overhead having to do with handling types, and the Julia/GDAL reference counting. Even,
is that what you had in mind?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-erik<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Jun 24, 2025, at 11:01, Even Rouault via gdal-dev <<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">Hi,<o:p></o:p></span></p>
<p style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">I don't know anything about Julia but I'd suspect that there must be something particularly slow in the way it interacts with C. For comparison, "time python3 swig/python/gdal-utils/osgeo_utils/samples/ogrinfo.py
/vsigzip//vsicurl/<a href="https://bulk.meteostat.net/v2/hourly/2022/08554.csv.gz"><span style="color:#467886">https://bulk.meteostat.net/v2/hourly/2022/08554.csv.gz</span></a> -al > /dev/null" that does essentially your loop, and also prints on stdout, runs
in 1.5 seconds (compared to native ogrinfo that runs in 0.7 s). Perhaps you could write a Julia wrapper to get all fields of feature at once and return whatever dictionary or equivalent data structure is idiomatic (and efficient )in Julia ? Also are you sure
your Julia wrapper is built with optimization enabled?<o:p></o:p></span></p>
<p style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">Even<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">Le 24/06/2025 à 16:33, Joaquim Manuel Freire Luís via gdal-dev a écrit :<o:p></o:p></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<div>
<p class="MsoNormal"><span lang="PT" style="font-size:11.0pt">Hi,</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="PT" style="font-size:11.0pt"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Im trying to read files like<span class="apple-converted-space"> </span><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="https://bulk.meteostat.net/v2/hourly/2022/08554.csv.gz"><span style="color:#467886">https://bulk.meteostat.net/v2/hourly/2022/08554.csv.gz</span></a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">in my Julia wrapper. The point is that, although I’m kind off succeeding, the hole operation is very slow.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">What I’m doing (code not committed yet so can’t post a link) is to read like this<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">layer = getlayer(dataset, 0)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">for f in layer<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> for k = 1: Gdal.nfield(f)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> Gdal.getfield(f, k-1)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">…<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">This works but it’s extremely slow because each “getfield” takes about 1e-4 seconds and the file has ~8 k rows, each with 13 fields. That amounts to > 10 sec.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">I’ve searched but couldn’t find a way to read the entire file at once (which takes 1e-2 seconds if I read it, locally, with a gzip wrapper) and return it as a single string array that I could parse later.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Is that possible?<span class="apple-converted-space"> </span><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Joaquim<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><o:p> </o:p></span></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>gdal-dev mailing list<o:p></o:p></pre>
<pre><a href="mailto:gdal-dev@lists.osgeo.org"><span style="color:#467886">gdal-dev@lists.osgeo.org</span></a><o:p></o:p></pre>
<pre><a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev"><span style="color:#467886">https://lists.osgeo.org/mailman/listinfo/gdal-dev</span></a><o:p></o:p></pre>
</blockquote>
<pre style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;text-align:start;-webkit-text-stroke-width: 0px;word-spacing:0px"><span style="font-size:9.0pt">-- <o:p></o:p></span></pre>
<pre><span style="font-size:9.0pt"><a href="http://www.spatialys.com/"><span style="color:#467886">http://www.spatialys.com</span></a><o:p></o:p></span></pre>
<pre><span style="font-size:9.0pt">My software is free, but my time generally not.<o:p></o:p></span></pre>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">_______________________________________________<br>
gdal-dev mailing list<br>
</span><a href="mailto:gdal-dev@lists.osgeo.org"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#467886">gdal-dev@lists.osgeo.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
</span><a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#467886">https://lists.osgeo.org/mailman/listinfo/gdal-dev</span></a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>