[PROJ] Is PROJ unreasonably slow?
Thomas Knudsen
knudsen.thomas at gmail.com
Fri Mar 13 03:18:20 PDT 2026
> I hope it is not a revenge for my last emails containing some explicit WKT2
@Javier: Definitely not - you are probably one of the friendliest people, I have
met, so I would never ponder staging revenge plans in your general direction :-)
> That is good news! If a projection library, with all the trigonometric and
> complicated functions it is calling, is affected by reading and writing
> doubles in input/output, it means that the library is really efficient
Definitely good news. My first reaction to the initial results, until I
understood how to interpret them, was "this is really bad news". On the other
hand I found it unbelievable, because PROJ and Rust Geodesy largely implement
the same algorithms (and some of them even implemented by me in both cases), and
I wouldn't expect just switching between two compiled languages could make that
much of a difference. Fortunately it didn't!
> Remove compression? well, maybe the Swedish files are small.
I did not suggest ot remove the compression - I just stated that compression
eliminates the possibility of direct access, so choosing one implies the other
> About grids in memory, there is always the trade off between memory economics
> and performance
Absolutely! And my reason for testing the Unigrid 1:1 menory mapping is to leave
the memory economics considerations to the OS, which has a much better general
view of the memory situation. In a sense, any kind of explicit caching/reading
ahead is an attempt of second-guessing the OS, so the Unigrid experiment is a
matter of testing whether the OS catches the hint of "do your thing - I trust
your decisions". As long as the file is mapped read-only, dropping cached pages
is in principle close to zero-cost, so mapping infinitely large files should be
a realistic possibility. But that's the theory. How it plays out practically may
be another thing.
I do, however, remember Poul-Henning Kamp, lead developer of the Varnish web
cache system, expressing that one of the reasons for Varnish's efficiency was
that it did not try to second-guess the OS. So that's what I try (not) to :-)
/Thomas
Den tors. 12. mar. 2026 kl. 20.40 skrev Javier Jimenez Shaw
<j1 at jimenezshaw.com>:
>
> Hi Thomas
>
> It took me some time to read the email. I hope it is not a revenge for my last emails containing some explicit WKT2 ;)
>
> That is good news! If a projection library, with all the trigonometric and complicated functions it is calling, is affected by reading and writing doubles in input/output, it means that the library is really efficient.
> Maybe not in the parsing of the strings, but the rest it is.
> When I started reading the email a while ago, my first reaction was "many people say std streams are not efficient".
>
> About caching grid files, we could probably add an environment variable (read only once) to set the cache size to a value. That could give you some flexibility requesting random points.
> Remove compression? well, maybe the Swedish files are small. The new geoid model for US will not fit in the size limit per file we have in GitHub. We will have to do some tricks to include it (like use int16 instead of float32).
>
> In summary I agree with Even: command line apps are "toys" for testing/playing. Can we improve that performance? Probably. Is it worth it? I don't know. Is your fear that they use it to compare with other libraries, and PROJ is minus-valued? Could be.
> About grids in memory, there is always the trade off between memory economics and performance. We can be more flexible for extreme cases as yours.
>
> Maybe I am missing other points inside the library where we are slow and we can improve. Please, let me know.
>
> Cheers
> Javier.
>
>
> On Wed, 11 Mar 2026 at 18:43, Even Rouault <even.rouault at spatialys.com> wrote:
>>
>> Thomas,
>>
>> I don't know how you manage to write such long essays. Does my (non LLM
>> generated) below summary reflect your findings:
>>
>> - Command line utilities are to be considered "toys" if you process a
>> large amount of data and care about perf. Not that suprising, and I
>> wouldn't expect anyone trying to use PROJ at its max perf capabilities
>> to use them, but rather use the (binary) API. In GDAL, we use at some
>> places the fast_float C++ header library for fast string->double :
>> https://github.com/fastfloat/fast_float . They claim it can be up to
>> about 4x faster than strtod(). We could also vendor in PROJ if needed.
>>
>> - Grid usage could be made much faster if we decompressed everything in
>> memory. There's already a bit of caching of decompressed tiles. But
>> random spatial access pattern will easily defeat it as it is quite
>> modest currently. We could add an option to increase the cache to a
>> specified size and/or increase the default size. I wouldn't trust too
>> much OS virtual memory mapping. Can easily lead to crashing your
>> system (or making it unusable at the point your need to hard reboot) if
>> you map too much memory vs your available RAM.
>>
>> Even
>>
>> Le 11/03/2026 à 09:45, Thomas Knudsen a écrit :
>> > TL;DR: Is PROJ unreasonably slow? The answer is "Mostly no", but with a few
>> > caveats...
>> >
>> > DISCLAIMER: The evidence presented below is weak - timing experiments repeated
>> > only 2-4 times, on just a single computer. But at first sight the speed tests
>> > all hint at the same conclusion: That PROJ could be made significantly faster.
>> >
>> > Closer inspection, however, shows that, while in some corner cases, PROJ really
>> > can be made faster, in the general cases, it already really is quite fast.
>> >
>> ... deleted content.
>> >
>> > But I would highly prefer to leave this kind of code to reside in system
>> > libraries, not in an application library, like PROJ.
>> >
>> > Nevertheless: Hope y'all will consider this (much too) long writeup, and give it
>> > a deep thought, whether rearchitecting PROJ, and to what extent, may be worth
>> > the effort.
>> >
>> > /Thomas Knudsen
>> >
>> >
>> > [1] Rust Geodesy: https://lib.rs/geodesy
>> > https://github.com/busstoptaktik/geodesy
>> > [2] kp: https://github.com/busstoptaktik/geodesy/blob/main/ruminations/003-rumination.md
>> > [3] cs2cs: https://proj.org/en/stable/apps/cs2cs.html
>> > [4] cct: https://proj.org/en/stable/apps/cct.html
>> > [5] The ITRF2014->SWEREF99 transformation:
>> > $ projinfo -o proj --hide-ballpark -s itrf2014 -t sweref99
>> > +proj=pipeline
>> > +step +proj=axisswap +order=2,1
>> > +step +proj=unitconvert +xy_in=deg +xy_out=rad
>> > +step +proj=cart +ellps=GRS80
>> > +step +proj=helmert +x=0 +y=0 +z=0 +rx=0.001785 +ry=0.011151
>> > +rz=-0.01617 +s=0
>> > +dx=0 +dy=0 +dz=0 +drx=8.5e-05 +dry=0.000531 +drz=-0.00077 +ds=0
>> > +t_epoch=2010 +convention=position_vector
>> > +step +inv +proj=deformation +t_epoch=2000 +grids=eur_nkg_nkgrf17vel.tif
>> > +ellps=GRS80
>> > +step +proj=helmert +x=0.03054 +y=0.04606 +z=-0.07944 +rx=0.00141958
>> > +ry=0.00015132 +rz=0.00150337 +s=0.003002
>> > +convention=position_vector
>> > +step +proj=deformation +dt=-0.5 +grids=eur_nkg_nkgrf17vel.tif
>> > +ellps=GRS80
>> > +step +inv +proj=cart +ellps=GRS80
>> > +step +proj=unitconvert +xy_in=rad +xy_out=deg
>> > +step +proj=axisswap +order=2,1
>> > [6] Rumination 012: Unigrids and the UG grid maintenance utility
>> > https://github.com/busstoptaktik/geodesy/blob/main/ruminations/012-rumination.md
>> > [7] Even Rouault om lam0:
>> > https://github.com/OSGeo/PROJ/pull/4667/changes#diff-bfb0c333155a0c8bf863b0a3e76df46cfddf646cd5f13d6313eb8a3cb123f5f1R58
>> > [8] proj_strtod():
>> > https://github.com/OSGeo/PROJ/blob/master/src/apps/proj_strtod.cpp
>> > [9] Update Rust Float-Parsing Algorithms to use the Eisel-Lemire algorithm
>> > https://github.com/rust-lang/rust/pull/86761
>> > [10] Implementing a Fast, Correct Float Parser
>> > https://internals.rust-lang.org/t/implementing-a-fast-correct-float-parser/14670
>> > [11] David Tolnay's dtoa-benchmark: https://github.com/dtolnay/dtoa-benchmark
>> > [12] Victor Zverovich's zmij algorithm: https://github.com/vitaut/zmij/
>> > [13] OxiGDAL - Pure Rust Geospatial Data Abstraction Library:
>> > https://github.com/cool-japan/oxigdal
>> > [14] proj4rs - Rust adaptation of PROJ.4: https://crates.io/crates/proj4rs
>>
>> --
>> http://www.spatialys.com
>> My software is free, but my time generally not.
>>
More information about the PROJ
mailing list