From weston.renoud at qps.nl Fri Aug 14 07:25:51 2026 From: weston.renoud at qps.nl (Weston Renoud) Date: Fri, 14 Aug 2026 14:25:51 +0000 Subject: [PROJ] Flaky tests for new tmerczoned projection Message-ID: Hi all, My PR to implement EPSG:32600 and EPSG:32700 got merged recently (https://github.com/OSGeo/PROJ/pull/4805), and I have noticed (as pointed out by Javier) this is causing intermittent/flaky test failures in the linux_gcc_32bit build. I've added an issue with the details https://github.com/OSGeo/PROJ/issues/4814. When the tests fail the projection is resulting in NaN values in the returned PJ_COORD for both forward and inverse. Specifically, I'll looking for input on how I'm creating and using the Transverse Mercator projection in tmerczoned.cpp. The projection is using Transverse Mercator for the actual projection and then offsetting appropriately for the zone. So tmerc is being constructed in the initialization of tmerczoned, and I believe the initialization is incomplete (but sometimes right), and the source of the flaky failures. I only saw an example of creating a "sub" projection in the in-review PR https://github.com/OSGeo/PROJ/pull/4770 and copied that. It relies heavily on pj_calc_ellipsoid_params, and manually copying forward the ctx and k0. Is there a better approach to this initialization? Any input greatly appreciated, Weston Renoud -------------- next part -------------- An HTML attachment was scrubbed... URL: From j1 at jimenezshaw.com Sat Aug 15 03:11:57 2026 From: j1 at jimenezshaw.com (Javier Jimenez Shaw) Date: Sat, 15 Aug 2026 12:11:57 +0200 Subject: [PROJ] Problems in CI: MacOS build Message-ID: Apparently the compilation in CI changed for MacOS Before it was using Clang 19.1.7, and now Clang 21.1.0 Unfortunately it does not compile with Clang 21.1.0 I have seen two different problems. Sometimes it is complaining about the linker: https://github.com/OSGeo/PROJ/actions/runs/31825301827/job/94847898881 Linking C executable cmTC_51f2b /Users/runner/micromamba/envs/proj/bin/cmake -E cmake_link_script CMakeFiles/cmTC_51f2b.dir/link.txt --verbose=1 ld: -lto_library library filename must be 'libLTO.dylib' clang: error: linker command failed with exit code 1 (use -v to see invocation) Others it is showing two compilation errors: https://github.com/jjimenezshaw/PROJ/actions/runs/31823956660/job/94843554428 In file included from /Users/runner/work/PROJ/PROJ/build_from_dist/proj-9.9.0/shared_build/_deps/googletest-src/googletest/include/gtest/gtest-matchers.h:49: /Users/runner/work/PROJ/PROJ/build_from_dist/proj-9.9.0/shared_build/_deps/googletest-src/googletest/include/gtest/gtest-printers.h:520:35: error: implicit conversion from 'char16_t' to 'char32_t' may change the meaning of the represented code unit [-Werror,-Wcharacter-conversion] 520 | PrintTo(ImplicitCast_(c), os); | ~~~~~~~~~~~~~ ^ 1 error generated. ... [ 30%] Linking CXX shared library ../lib/libproj.dylib Undefined symbols for architecture arm64: "___divdc3", referenced from: complex_horner_iterative_inverse_4d(PJ_COORD&, PJconsts*) in unity_22_cxx.cxx.o ld: symbol(s) not found for architecture arm64 clang++: error: linker command failed with exit code 1 (use -v to see invocation) Any idea how to fix it? I have no idea what is the condition to trigger each scenario. Thanks Javier -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdt at lexort.com Sat Aug 15 05:50:01 2026 From: gdt at lexort.com (Greg Troxel) Date: Sat, 15 Aug 2026 08:50:01 -0400 Subject: [PROJ] Problems in CI: MacOS build In-Reply-To: (Javier Jimenez Shaw via PROJ's message of "Sat, 15 Aug 2026 12:11:57 +0200") References: Message-ID: Javier Jimenez Shaw via PROJ writes: > https://github.com/OSGeo/PROJ/actions/runs/31825301827/job/94847898881 > > Linking C executable cmTC_51f2b > /Users/runner/micromamba/envs/proj/bin/cmake -E cmake_link_script > CMakeFiles/cmTC_51f2b.dir/link.txt --verbose=1 > ld: -lto_library library filename must be 'libLTO.dylib' > clang: error: linker command failed with exit code 1 (use -v to see > invocation) Standard rant about cmake and modern build systems: Actual build lines are often suppressed in favor of eye candy. Nobody reads build output until something goes wrong, so we should turn on enough verbosity to print the commands being executed, always. I see a line to invoke some ld script with cmake and then an ld error message but not the ld invocation. I think this can be done in CMakeFile.txt This error is perhaps reproducible with just a cmakefile that runs this test, out of proj context, and is likely some blend of a cmake bug and an apple bug. Perhaps LTO can be disabled. > https://github.com/jjimenezshaw/PROJ/actions/runs/31823956660/job/94843554428 > > In file included from > /Users/runner/work/PROJ/PROJ/build_from_dist/proj-9.9.0/shared_build/_deps/googletest-src/googletest/include/gtest/gtest-matchers.h:49: > /Users/runner/work/PROJ/PROJ/build_from_dist/proj-9.9.0/shared_build/_deps/googletest-src/googletest/include/gtest/gtest-printers.h:520:35: > error: implicit conversion from 'char16_t' to 'char32_t' may change the > meaning of the represented code unit [-Werror,-Wcharacter-conversion] > 520 | PrintTo(ImplicitCast_(c), os); > | ~~~~~~~~~~~~~ ^ > 1 error generated. I am guessing that char16_t is like char in C, where it is up to the implementation to decide if it is signed or unsigned as a variable, and that char16_t holds unicode, but only 0-65535, and char32_t can hold 0-{2^32-1]. If my guess is right, then values 32768-65535 have implementation-chosen sign extension behavior and if the unicode code points are unsigned (which they probably are), one of those choices is wrong. Perhaps somebody who understands the complexities of unicode better than I do (I was raised ASCII, and excited to have lower case!) can explain if characters are supposed to be signed or not and related issues. > [ 30%] Linking CXX shared library ../lib/libproj.dylib > Undefined symbols for architecture arm64: > "___divdc3", referenced from: > complex_horner_iterative_inverse_4d(PJ_COORD&, PJconsts*) in > unity_22_cxx.cxx.o > ld: symbol(s) not found for architecture arm64 > clang++: error: linker command failed with exit code 1 (use -v to see > invocation)> I think the plan is for compilers to be able to use functions for complicated math, with an implementation strategy that sometimes they are replaced by machine instructions and sometimes by assembler routines. My guess is that this is a situation where compiling the file used such a function but for some reason it isn't in the standard library used at link time. That could be a toolchain bug, and it could be a build system issue if multiple toolchains are in scope somehow and -L options lead to using the wrong standard library. Apparently this is a common woe: https://github.com/llvm/llvm-project/issues/54596 I have kicked off a build of proj master on NetBSD 10 aarch64, using gcc 10. I'll follow up with how that went; it's going to take a while because the machine is slow. From gdt at lexort.com Sat Aug 15 12:07:39 2026 From: gdt at lexort.com (Greg Troxel) Date: Sat, 15 Aug 2026 15:07:39 -0400 Subject: [PROJ] Problems in CI: MacOS build In-Reply-To: (Greg Troxel via PROJ's message of "Sat, 15 Aug 2026 08:50:01 -0400") References: Message-ID: Greg Troxel via PROJ writes: > I have kicked off a build of proj master on NetBSD 10 aarch64, using gcc > 10. I'll follow up with how that went; it's going to take a while > because the machine is slow. No warnings or errors in the build. All tests pass. I should be able to test on macOS x86_64 15 macOS aarch64 16 and will report anything interesting. From j1 at jimenezshaw.com Sat Aug 15 14:05:27 2026 From: j1 at jimenezshaw.com (Javier Jimenez Shaw) Date: Sat, 15 Aug 2026 23:05:27 +0200 Subject: [PROJ] Problems in CI: MacOS build In-Reply-To: References: Message-ID: fixed in https://github.com/OSGeo/PROJ/pull/4816 On Sat, 15 Aug 2026 at 21:07, Greg Troxel via PROJ wrote: > Greg Troxel via PROJ writes: > > > I have kicked off a build of proj master on NetBSD 10 aarch64, using gcc > > 10. I'll follow up with how that went; it's going to take a while > > because the machine is slow. > > No warnings or errors in the build. All tests pass. > > I should be able to test on > > macOS x86_64 15 > macOS aarch64 16 > > and will report anything interesting. > _______________________________________________ > PROJ mailing list > PROJ at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/proj > -------------- next part -------------- An HTML attachment was scrubbed... URL: From j1 at jimenezshaw.com Tue Aug 18 04:35:30 2026 From: j1 at jimenezshaw.com (Javier Jimenez Shaw) Date: Tue, 18 Aug 2026 13:35:30 +0200 Subject: [PROJ] testing PROJ master online Message-ID: Hi PROJ people. (in particular geodetic agencies and users) Release 9.9.0 is coming soon. It includes big changes in EPSG in Europe related to ETRS89 realizations and transformations. If you don't know how to build master to test it, maybe you can test it in this page (it is updated on every merge into master): https://jjimenezshaw.github.io/wasm-proj/master/transform.html Notice the different URL, color and text at the top. Enjoy it! Cheers Javier. . -------------- next part -------------- An HTML attachment was scrubbed... URL: From wwcohen at gmail.com Wed Aug 26 07:59:54 2026 From: wwcohen at gmail.com (Will Cohen) Date: Wed, 26 Aug 2026 10:59:54 -0400 Subject: [PROJ] proj-wasm 0.1.0-alpha9 Message-ID: Hello all, An update on the grid fetching and worker questions from the January proj.js thread. The latest release of proj-wasm (0.1.0-alpha9, https://www.npmjs.com/package/proj-wasm) addresses them as follows: Grid fetching now works in browsers and Node.js without curl in the build. PROJ runs entirely inside workers, and grids come through PROJ's network callback API backed by synchronous HTTP requests, which workers permit. In Node.js, a small fetch worker bridges those synchronous callbacks to Node's async http. The WASM build is now fully single-threaded, with no pthreads. (Pthreads ends up requiring SharedArrayBuffer and COOP/COEP headers, which limits the usability of the WASM build to certain environments only.) Instead, parallelism now comes from a pluggable pool of workers, each running its own module instance, with PROJ contexts pinned to workers. The demo runs this pool with network grids enabled: https://willcohen.github.io/clj-proj/ This is a slightly different path from Javier's https://github.com/jjimenezshaw/wasm-proj, which is exercising the ENABLE_EMSCRIPTEN_FETCH option from https://github.com/OSGeo/PROJ/pull/4627. I still like the idea of trying to use that -- it definitely seems cleaner and less like reinventing the wheel -- but I still want PROJ to be able to nab grids without requiring that webpages be served with a particular set of headers. I've also since moved this worker plumbing into reusable libraries ( https://github.com/willcohen/clj-native and https://github.com/willcohen/worker-router), and proj-wasm can adopt an external pool the caller supplies. This is how backproj (live demo at https://willcohen.github.io/backproj/), which uses proj-wasm to back-project custom CRSes onto MapLibre, coordinates wasmts geometry operations (JTS compiled to wasm) and proj-wasm projection operations in one worker pool. Now that these are settled and the mechanisms to build and hook into PROJ on wasm are abstracted out, I plan to reuse them for a similar wasm build of GDAL, hopefully without re-embedding PROJ inside it. With PROJ, JTS (via wasmts), and eventually GDAL, I'm hoping that most GIS functionality can happen client side in the browser! Will -------------- next part -------------- An HTML attachment was scrubbed... URL: From j1 at jimenezshaw.com Wed Aug 26 12:59:11 2026 From: j1 at jimenezshaw.com (Javier Jimenez Shaw) Date: Wed, 26 Aug 2026 21:59:11 +0200 Subject: [PROJ] proj-wasm 0.1.0-alpha9 In-Reply-To: References: Message-ID: Hi Will On Wed, 26 Aug 2026 at 17:00, Will Cohen via PROJ wrote: > Hello all, > > An update on the grid fetching and worker questions from the January > proj.js thread. The latest release of proj-wasm (0.1.0-alpha9, > https://www.npmjs.com/package/proj-wasm) addresses them as follows: > > Grid fetching now works in browsers and Node.js without curl in the build. > PROJ runs entirely inside workers, and grids come through PROJ's network > callback API backed by synchronous HTTP requests, which workers permit. In > Node.js, a small fetch worker bridges those synchronous callbacks to Node's > async http. > > The WASM build is now fully single-threaded, with no pthreads. (Pthreads > ends up requiring SharedArrayBuffer and COOP/COEP headers, which limits the > usability of the WASM build to certain environments only.) Instead, > parallelism now comes from a pluggable pool of workers, each running its > own module instance, with PROJ contexts pinned to workers. The demo runs > this pool with network grids enabled: > https://willcohen.github.io/clj-proj/ > > This is a slightly different path from Javier's > https://github.com/jjimenezshaw/wasm-proj, which is exercising the > ENABLE_EMSCRIPTEN_FETCH option from > https://github.com/OSGeo/PROJ/pull/4627. I still like the idea of trying > to use that -- it definitely seems cleaner and less like reinventing the > wheel -- but I still want PROJ to be able to nab grids without requiring > that webpages be served with a particular set of headers. > What headers are you talking about? I remember that at the beginning of my research I had to deal with COOP/COEP headers (and we were talking about it), but not when I published wasm-proj months ago (at least I removed that part of the code). I am not using any SharedArrayBuffer to talk with the web worker, just standard serializable messages. Am I missing any detail? Maybe emscripten is doing that internally, but it is only starting the web worker I explicitly start. > I've also since moved this worker plumbing into reusable libraries ( > https://github.com/willcohen/clj-native and > https://github.com/willcohen/worker-router), and proj-wasm can adopt an > external pool the caller supplies. This is how backproj (live demo at > https://willcohen.github.io/backproj/), which uses proj-wasm to > back-project custom CRSes onto MapLibre, coordinates wasmts geometry > operations (JTS compiled to wasm) and proj-wasm projection operations in > one worker pool. > > Now that these are settled and the mechanisms to build and hook into PROJ > on wasm are abstracted out, I plan to reuse them for a similar wasm build > of GDAL, hopefully without re-embedding PROJ inside it. > > With PROJ, JTS (via wasmts), and eventually GDAL, I'm hoping that most GIS > functionality can happen client side in the browser! > > Will > Thank you Javier. > _______________________________________________ > PROJ mailing list > PROJ at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/proj > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wwcohen at gmail.com Wed Aug 26 13:27:19 2026 From: wwcohen at gmail.com (Will Cohen) Date: Wed, 26 Aug 2026 16:27:19 -0400 Subject: [PROJ] proj-wasm 0.1.0-alpha9 In-Reply-To: References: Message-ID: Oh neat. Last time I checked, there was still that service worker, but yeah, it looks like it's totally unnecessary now. I thought that SharedArrayBuffer (which, I THINK, is still being used internally by emscripten when pthreads is turned on) always required those headers, but it looks like you got it to work without, by keeping it in the one worker. I'll need to think a little more about the effect this has on parallelizing with multiple workers, and the memory / throughput comparisons with pthreads. This is cool. On Wed, Aug 26, 2026 at 3:59?PM Javier Jimenez Shaw wrote: > Hi Will > > On Wed, 26 Aug 2026 at 17:00, Will Cohen via PROJ > wrote: > >> Hello all, >> >> An update on the grid fetching and worker questions from the January >> proj.js thread. The latest release of proj-wasm (0.1.0-alpha9, >> https://www.npmjs.com/package/proj-wasm) addresses them as follows: >> >> Grid fetching now works in browsers and Node.js without curl in the >> build. PROJ runs entirely inside workers, and grids come through PROJ's >> network callback API backed by synchronous HTTP requests, which workers >> permit. In Node.js, a small fetch worker bridges those synchronous >> callbacks to Node's async http. >> >> The WASM build is now fully single-threaded, with no pthreads. (Pthreads >> ends up requiring SharedArrayBuffer and COOP/COEP headers, which limits the >> usability of the WASM build to certain environments only.) Instead, >> parallelism now comes from a pluggable pool of workers, each running its >> own module instance, with PROJ contexts pinned to workers. The demo runs >> this pool with network grids enabled: >> https://willcohen.github.io/clj-proj/ >> >> This is a slightly different path from Javier's >> https://github.com/jjimenezshaw/wasm-proj, which is exercising the >> ENABLE_EMSCRIPTEN_FETCH option from >> https://github.com/OSGeo/PROJ/pull/4627. I still like the idea of trying >> to use that -- it definitely seems cleaner and less like reinventing the >> wheel -- but I still want PROJ to be able to nab grids without requiring >> that webpages be served with a particular set of headers. >> > > What headers are you talking about? I remember that at the beginning of my > research I had to deal with COOP/COEP headers (and we were talking about > it), but not when I published wasm-proj months ago (at least I removed that > part of the code). I am not using any SharedArrayBuffer to talk with the > web worker, just standard serializable messages. Am I missing any detail? > Maybe emscripten is doing that internally, but it is only starting the web > worker I explicitly start. > > >> I've also since moved this worker plumbing into reusable libraries ( >> https://github.com/willcohen/clj-native and >> https://github.com/willcohen/worker-router), and proj-wasm can adopt an >> external pool the caller supplies. This is how backproj (live demo at >> https://willcohen.github.io/backproj/), which uses proj-wasm to >> back-project custom CRSes onto MapLibre, coordinates wasmts geometry >> operations (JTS compiled to wasm) and proj-wasm projection operations in >> one worker pool. >> >> Now that these are settled and the mechanisms to build and hook into PROJ >> on wasm are abstracted out, I plan to reuse them for a similar wasm build >> of GDAL, hopefully without re-embedding PROJ inside it. >> >> With PROJ, JTS (via wasmts), and eventually GDAL, I'm hoping that most >> GIS functionality can happen client side in the browser! >> >> Will >> > > Thank you > Javier. > > >> _______________________________________________ >> PROJ mailing list >> PROJ at lists.osgeo.org >> https://lists.osgeo.org/mailman/listinfo/proj >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From weston.renoud at qps.nl Mon Aug 31 09:13:28 2026 From: weston.renoud at qps.nl (Weston Renoud) Date: Mon, 31 Aug 2026 16:13:28 +0000 Subject: [PROJ] Performance when Auxilary DB used Message-ID: Hi all, Even, Javier, and I discussed at FOSS4G EU some topics I could explore to get more familiar with PROJ and one of the ideas was to explore performance when using an auxiliary database. I have poked at the problem and wanted to share what I have found so far. TL&DR I think I may have a found an improvement that reduces the auxillary database overhead by about 50%. The diagnostics I'm using, and the improvement can be found in the branch https://github.com/OSGeo/PROJ/compare/master...wrenoud:PROJ:investigate-aux-db-performance I focused on the usage of an empty auxiliary database with cs2cs, with a simple projection of WGS 84 / UTM zone 1N coordinates. In windows command prompt this looks like: ????set PROJ_AUX_DB=/aux.db ????set PROJ_DEBUG=2 ????cs2cs EPSG:4236 EPSG:32601 test.txt I develop primarily on Windows and Linux for Intel CPUs, so I make use of Intel's Vtune Profiler when I'm looking to evaluate code performance. I instrumented the method `DatabaseContext::Private::run` with begin and end gates to time the SQL queries to try to identify if there are particular queries that perform especially poorly. What I found: Usage of `DatabaseContext::Private::run`: ????without auxiliary database: 43 SQL queries, totaling ~3ms. ????with empty auxiliary database: 221 SQL queries, totaling ~70ms. With the auxiliary database the extra queries and time could be broken down into 3 main categories * Setup - about 20ms (97 queries) of the 70ms when using the auxiliary database is related to initial setup * Duplicates - for queries that don't need cross referencing, they are called directly on each attached database and have comparable performance. * Usage query - about 40-50ms (11 queries) The setup does three main things: ATTACHING each database, querying the table structures, and creating temporary VIEWs to UNION the tables between the attached databases. These are required to support cross referencing from the auxiliary database(s) to items in the primary database, and it is not obvious there is any room for improvements. The duplicates on each attached database are necessary given the additional/auxillary database. Again, it is not obvious there is any room for improvements here. The usage query: of the queries on the UNION'ed VIEWs, the query from `AuthorityFactory::Private::createPropertiesSearchUsages` jumped out given the significant run time. Without an auxiliary database these queries sum to less than 0.1ms, but with an empty auxiliary database they sum to 40-50ms. This query requires a join between the usage, extent and scope tables. ????SELECT extent.description, ???????? extent.south_lat, ???????? extent.north_lat, ???????? extent.west_lon, ???????? extent.east_lon, ???????? scope.scope, ???????? (CASE WHEN scope.scope LIKE '%large scale%' THEN 0 ELSE 1 END) AS score ????FROM usage ???????????? JOIN extent ON usage.extent_auth_name = extent.auth_name AND usage.extent_code = extent.code ???????????? JOIN scope ON usage.scope_auth_name = scope.auth_name AND usage.scope_code = scope.code ????WHERE object_table_name = ? ???? AND object_auth_name = ? ???? AND object_code = ? ???? AND NOT (usage.extent_auth_name = 'PROJ' AND usage.extent_code = 'EXTENT_UNKNOWN') ???? AND NOT (usage.scope_auth_name = 'PROJ' AND usage.scope_code = 'SCOPE_UNKNOWN') ????ORDER BY score, usage.auth_name, usage.code Trying to optimize this query seemed like a good target, so for the next step I tried to learn more about it by making use of `EXPLAIN QUERY PLAN` (https://sqlite.org/eqp.html). Without the auxiliary database, the query execution is very straightforward and fast. It is executed as three searches by index or primary key: ????6,0,43,SEARCH usage USING INDEX idx_usage_object (object_table_name=? AND object_auth_name=? AND object_code=?) ????25,0,39,SEARCH extent USING PRIMARY KEY (auth_name=? AND code=?) ????31,0,34,SEARCH scope USING PRIMARY KEY (auth_name=? AND code=?) ????58,0,0,USE TEMP B-TREE FOR ORDER BY With the auxiliary database the query plan bloats to this due to the JOINs over the UNION'ed tables: ????2,0,0,CO-ROUTINE usage ????3,2,0,COMPOUND QUERY ????4,3,0,LEFT-MOST SUBQUERY ????7,4,43,SEARCH db_0.usage USING INDEX idx_usage_object (object_table_name=? AND object_auth_name=? AND object_code=?) ????37,3,0,UNION ALL ????39,37,216,SCAN db_1.usage ????69,0,0,MATERIALIZE extent ????71,69,0,COMPOUND QUERY ????72,71,0,LEFT-MOST SUBQUERY ????74,72,135,SCAN db_0.extent ????93,71,0,UNION ALL ????95,93,215,SCAN db_1.extent ????117,0,0,MATERIALIZE scope ????119,117,0,COMPOUND QUERY ????120,119,0,LEFT-MOST SUBQUERY ????122,120,97,SCAN db_0.scope ????132,119,0,UNION ALL ????134,132,215,SCAN db_1.scope ????146,0,196,SCAN usage ????167,0,0,BLOOM FILTER ON extent (code=? AND auth_name=?) ????188,0,53,SEARCH extent USING AUTOMATIC COVERING INDEX (code=? AND auth_name=?) ????201,0,0,BLOOM FILTER ON scope (code=? AND auth_name=?) ????213,0,53,SEARCH scope USING AUTOMATIC COVERING INDEX (code=? AND auth_name=?) ????245,0,0,USE TEMP B-TREE FOR ORDER BY Of note are SCAN and MATERIALIZE. When combined, they effectively duplicate the table's contents into temporary tables for the query. Based on this I then explored if creating tables, instead of views for the extent and usage tables would improve the performance. This dropped the total query time (for all 11 calls) to ~3ms, with an overhead of ~4ms to create the tables. That is over 30ms reduction. But again, this is with an empty auxiliary database. I will test further by putting some entries into the auxiliary tables and see if there is a significant difference. I'm also curious if anyone with more sqlite experience might have some ideas based on these findings? Or if there are other concerns or considerations I should be thinking about here? Best Regards, Weston -------------- next part -------------- An HTML attachment was scrubbed... URL: From j1 at jimenezshaw.com Mon Aug 31 09:22:22 2026 From: j1 at jimenezshaw.com (Javier Jimenez Shaw) Date: Mon, 31 Aug 2026 18:22:22 +0200 Subject: [PROJ] Performance when Auxilary DB used In-Reply-To: References: Message-ID: Hi Weston. That looks promising. Thank you. If you want a "big" auxiliary database (with about extra 1900 CRSs) there is one in https://github.com/jjimenezshaw/NSRS-2022-PROJ The file itself is working with PROJ 9.8.1, not with master (there was a change in the schema, that I will update with 9.9.0). Best Javier. On Mon, 31 Aug 2026 at 18:13, Weston Renoud via PROJ wrote: > Hi all, > > Even, Javier, and I discussed at FOSS4G EU some topics I could explore to > get more familiar with PROJ and one of the ideas was to explore performance > when using an auxiliary database. I have poked at the problem and wanted to > share what I have found so far. > > TL&DR I think I may have a found an improvement that reduces the auxillary > database overhead by about 50%. > > The diagnostics I'm using, and the improvement can be found in the branch > https://github.com/OSGeo/PROJ/compare/master...wrenoud:PROJ:investigate-aux-db-performance > > I focused on the usage of an empty auxiliary database with cs2cs, with a > simple projection of WGS 84 / UTM zone 1N coordinates. In windows command > prompt this looks like: > > set PROJ_AUX_DB=/aux.db > set PROJ_DEBUG=2 > cs2cs EPSG:4236 EPSG:32601 test.txt > > I develop primarily on Windows and Linux for Intel CPUs, so I make use of > Intel's Vtune Profiler when I'm looking to evaluate code performance. I > instrumented the method `DatabaseContext::Private::run` with begin and end > gates to time the SQL queries to try to identify if there are particular > queries that perform especially poorly. What I found: > > Usage of `DatabaseContext::Private::run`: > without auxiliary database: 43 SQL queries, totaling ~3ms. > with empty auxiliary database: 221 SQL queries, totaling ~70ms. > > With the auxiliary database the extra queries and time could be broken > down into 3 main categories > * Setup - about 20ms (97 queries) of the 70ms when using the auxiliary > database is related to initial setup > * Duplicates - for queries that don't need cross referencing, they are > called directly on each attached database and have comparable performance. > * Usage query - about 40-50ms (11 queries) > > The setup does three main things: ATTACHING each database, querying the > table structures, and creating temporary VIEWs to UNION the tables between > the attached databases. These are required to support cross referencing > from the auxiliary database(s) to items in the primary database, and it is > not obvious there is any room for improvements. > > The duplicates on each attached database are necessary given the > additional/auxillary database. Again, it is not obvious there is any room > for improvements here. > > The usage query: of the queries on the UNION'ed VIEWs, the query from > `AuthorityFactory::Private::createPropertiesSearchUsages` jumped out given > the significant run time. Without an auxiliary database these queries sum > to less than 0.1ms, but with an empty auxiliary database they sum to > 40-50ms. This query requires a join between the usage, extent and scope > tables. > > SELECT extent.description, > extent.south_lat, > extent.north_lat, > extent.west_lon, > extent.east_lon, > scope.scope, > (CASE WHEN scope.scope LIKE '%large scale%' THEN 0 ELSE 1 END) AS score > FROM usage > JOIN extent ON usage.extent_auth_name = extent.auth_name AND > usage.extent_code = extent.code > JOIN scope ON usage.scope_auth_name = scope.auth_name AND usage.scope_code > = scope.code > WHERE object_table_name = ? > AND object_auth_name = ? > AND object_code = ? > AND NOT (usage.extent_auth_name = 'PROJ' AND usage.extent_code = > 'EXTENT_UNKNOWN') > AND NOT (usage.scope_auth_name = 'PROJ' AND usage.scope_code = > 'SCOPE_UNKNOWN') > ORDER BY score, usage.auth_name, usage.code > > Trying to optimize this query seemed like a good target, so for the next > step I tried to learn more about it by making use of `EXPLAIN QUERY PLAN` ( > https://sqlite.org/eqp.html). > > Without the auxiliary database, the query execution is very > straightforward and fast. It is executed as three searches by index or > primary key: > > 6,0,43,SEARCH usage USING INDEX idx_usage_object (object_table_name=? AND > object_auth_name=? AND object_code=?) > 25,0,39,SEARCH extent USING PRIMARY KEY (auth_name=? AND code=?) > 31,0,34,SEARCH scope USING PRIMARY KEY (auth_name=? AND code=?) > 58,0,0,USE TEMP B-TREE FOR ORDER BY > > With the auxiliary database the query plan bloats to this due to the JOINs > over the UNION'ed tables: > > 2,0,0,CO-ROUTINE usage > 3,2,0,COMPOUND QUERY > 4,3,0,LEFT-MOST SUBQUERY > 7,4,43,SEARCH db_0.usage USING INDEX idx_usage_object (object_table_name=? > AND object_auth_name=? AND object_code=?) > 37,3,0,UNION ALL > 39,37,216,SCAN db_1.usage > 69,0,0,MATERIALIZE extent > 71,69,0,COMPOUND QUERY > 72,71,0,LEFT-MOST SUBQUERY > 74,72,135,SCAN db_0.extent > 93,71,0,UNION ALL > 95,93,215,SCAN db_1.extent > 117,0,0,MATERIALIZE scope > 119,117,0,COMPOUND QUERY > 120,119,0,LEFT-MOST SUBQUERY > 122,120,97,SCAN db_0.scope > 132,119,0,UNION ALL > 134,132,215,SCAN db_1.scope > 146,0,196,SCAN usage > 167,0,0,BLOOM FILTER ON extent (code=? AND auth_name=?) > 188,0,53,SEARCH extent USING AUTOMATIC COVERING INDEX (code=? AND > auth_name=?) > 201,0,0,BLOOM FILTER ON scope (code=? AND auth_name=?) > 213,0,53,SEARCH scope USING AUTOMATIC COVERING INDEX (code=? AND > auth_name=?) > 245,0,0,USE TEMP B-TREE FOR ORDER BY > > Of note are SCAN and MATERIALIZE. When combined, they effectively > duplicate the table's contents into temporary tables for the query. > > Based on this I then explored if creating tables, instead of views for the > extent and usage tables would improve the performance. This dropped the > total query time (for all 11 calls) to ~3ms, with an overhead of ~4ms to > create the tables. That is over 30ms reduction. But again, this is with an > empty auxiliary database. > > I will test further by putting some entries into the auxiliary tables and > see if there is a significant difference. I'm also curious if anyone with > more sqlite experience might have some ideas based on these findings? Or if > there are other concerns or considerations I should be thinking about here? > > Best Regards, > Weston > > _______________________________________________ > PROJ mailing list > PROJ at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/proj > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdt at lexort.com Mon Aug 31 09:53:20 2026 From: gdt at lexort.com (Greg Troxel) Date: Mon, 31 Aug 2026 12:53:20 -0400 Subject: [PROJ] Performance when Auxilary DB used In-Reply-To: (Weston Renoud via PROJ's message of "Mon, 31 Aug 2026 16:13:28 +0000") References: Message-ID: My quick impression is that sqlite is not dealing particularly well with joins over views, and that I can't really fault it as this seems beyond the scope of the main mission. I wonder about using memoization as a main strategy here, where when proj is run with an auxiliary database, it checks the mtime of both databases, and looks for a memoized merged database in ~/.cache/proj, perhaps named with the aux path and both timestamps. If so, use it, and if not, remove old versions, do the merge and write it. I suspect the first run will be a bit slow, but if proj is used a lot, then subsequent runs should be fast and on balance a good result. If proj is only used once with the aux db, then it's unlikely to matter. If someone has a use case where proj is used a large number of times with a different aux db every time, that's interesting, but to me unexpected. From even.rouault at spatialys.com Mon Aug 31 09:43:36 2026 From: even.rouault at spatialys.com (Even Rouault) Date: Mon, 31 Aug 2026 18:43:36 +0200 Subject: [PROJ] Performance when Auxilary DB used In-Reply-To: References: Message-ID: <67609653-6a6c-448d-8e4b-53be4e776299@spatialys.com> Hi Weston, thanks for this interesting analysis. I'm wondering if removing the ORDER BY clause would have a significant effect on the performance, if ordering would be the reason for materializing ? Sorting could be easily done on the C++ side. And the expected number of rows of the query in in 99% of the cases is just 1, and the remaining 1% 2, but SQLite has of course no clue about that. Otherwise you could try to submit the question to the SQLite experts at https://sqlite.org/forum/forum, but to have some hope of them having a look at that, you should provide an easy reproducer using the sqlite3 binary and provide the databases likely stripped down to only contain the needed tables to facilitate their analysis. Cheers, Even Le 31/08/2026 ? 18:13, Weston Renoud via PROJ a ?crit?: > Hi all, > > Even, Javier, and I discussed at FOSS4G EU some topics I could explore > to get more familiar with PROJ and one of the ideas was to explore > performance when using an auxiliary database. I have poked at the > problem and wanted to share what I have found so far. > > TL&DR I think I may have a found an improvement that reduces the > auxillary database overhead by about 50%. > > The diagnostics I'm using, and the improvement can be found in the > branch > https://github.com/OSGeo/PROJ/compare/master...wrenoud:PROJ:investigate-aux-db-performance > > > I focused on the usage of an empty auxiliary database with cs2cs, with > a simple projection of WGS 84 / UTM zone 1N coordinates. In windows > command prompt this looks like: > > ????set PROJ_AUX_DB=/aux.db > ????set PROJ_DEBUG=2 > ????cs2cs EPSG:4236 EPSG:32601 test.txt > > I develop primarily on Windows and Linux for Intel CPUs, so I make use > of Intel's Vtune Profiler when I'm looking to evaluate code > performance. I instrumented the method `DatabaseContext::Private::run` > with begin and end gates to time the SQL queries to try to identify if > there are particular queries that perform especially poorly. What I found: > > Usage of `DatabaseContext::Private::run`: > ????without auxiliary database: 43 SQL queries, totaling ~3ms. > ????with empty auxiliary database: 221 SQL queries, totaling ~70ms. > > With the auxiliary database the extra queries and time could be broken > down into 3 main categories > * Setup - about 20ms (97 queries) of the 70ms?when using the auxiliary > database is related to initial setup > * Duplicates - for queries that don't need cross referencing, they are > called directly on each attached database and have comparable performance. > * Usage query - about 40-50ms (11 queries) > > The setup does three main things: ATTACHING each database, querying > the table structures, and creating temporary VIEWs to UNION the tables > between the attached databases. These are required to support cross > referencing from the auxiliary database(s) to items in the primary > database, and it is not obvious there is any room for improvements. > > The duplicates on each attached database?are necessary given the > additional/auxillary database. Again, it is not obvious there is any > room for improvements here. > > The usage query: of the queries on the UNION'ed VIEWs, the query from > `AuthorityFactory::Private::createPropertiesSearchUsages` jumped out > given the significant run time. Without an auxiliary database these > queries sum to less than 0.1ms, but with an empty auxiliary database > they sum to 40-50ms. This query requires a join between the usage, > extent and scope tables. > > ????SELECT extent.description, > ???????? ? extent.south_lat, > ???????? ? extent.north_lat, > ???????? ? extent.west_lon, > ???????? ? extent.east_lon, > ???????? ? scope.scope, > ???????? ? (CASE WHEN scope.scope LIKE '%large scale%' THEN 0 ELSE 1 > END) AS score > ????FROM usage > ???????????? JOIN extent ON usage.extent_auth_name = extent.auth_name > AND usage.extent_code = extent.code > ???????????? JOIN scope ON usage.scope_auth_name = scope.auth_name AND > usage.scope_code = scope.code > ????WHERE object_table_name = ? > ???? ?AND object_auth_name = ? > ???? ?AND object_code = ? > ???? ?AND NOT (usage.extent_auth_name = 'PROJ' AND usage.extent_code = > 'EXTENT_UNKNOWN') > ???? ?AND NOT (usage.scope_auth_name = 'PROJ' AND usage.scope_code = > 'SCOPE_UNKNOWN') > ????ORDER BY score, usage.auth_name, usage.code > > Trying to optimize this query seemed like a good target, so for the > next step I tried to learn more about it by making use of `EXPLAIN > QUERY PLAN` (https://sqlite.org/eqp.html ). > > Without the auxiliary database, the query execution is very > straightforward and fast. It is executed as three searches by index or > primary key: > > ????6,0,43,SEARCH usage USING INDEX idx_usage_object > (object_table_name=? AND object_auth_name=? AND object_code=?) > ????25,0,39,SEARCH extent USING PRIMARY KEY (auth_name=? AND code=?) > ????31,0,34,SEARCH scope USING PRIMARY KEY (auth_name=? AND code=?) > ????58,0,0,USE TEMP B-TREE FOR ORDER BY > > With the auxiliary database the query plan bloats to this due to the > JOINs over the UNION'ed tables: > > ????2,0,0,CO-ROUTINE usage > ????3,2,0,COMPOUND QUERY > ????4,3,0,LEFT-MOST SUBQUERY > ????7,4,43,SEARCH db_0.usage USING INDEX idx_usage_object > (object_table_name=? AND object_auth_name=? AND object_code=?) > ????37,3,0,UNION ALL > ????39,37,216,SCAN db_1.usage > ????69,0,0,MATERIALIZE extent > ????71,69,0,COMPOUND QUERY > ????72,71,0,LEFT-MOST SUBQUERY > ????74,72,135,SCAN db_0.extent > ????93,71,0,UNION ALL > ????95,93,215,SCAN db_1.extent > ????117,0,0,MATERIALIZE scope > ????119,117,0,COMPOUND QUERY > ????120,119,0,LEFT-MOST SUBQUERY > ????122,120,97,SCAN db_0.scope > ????132,119,0,UNION ALL > ????134,132,215,SCAN db_1.scope > ????146,0,196,SCAN usage > ????167,0,0,BLOOM FILTER ON extent (code=? AND auth_name=?) > ????188,0,53,SEARCH extent USING AUTOMATIC COVERING INDEX (code=? AND > auth_name=?) > ????201,0,0,BLOOM FILTER ON scope (code=? AND auth_name=?) > ????213,0,53,SEARCH scope USING AUTOMATIC COVERING INDEX (code=? AND > auth_name=?) > ????245,0,0,USE TEMP B-TREE FOR ORDER BY > > Of note are SCAN and MATERIALIZE. When combined, they effectively > duplicate the table's contents into temporary tables for the query. > > Based on this I then explored if creating tables, instead of views for > the extent and usage tables would improve the performance. This > dropped the total query time (for all 11 calls) to ~3ms, with an > overhead of ~4ms to create the tables. That is over 30ms reduction. > But again, this is with an empty auxiliary database. > > I will test further by putting some entries into the auxiliary tables > and see if there is a significant difference. I'm also curious if > anyone with more sqlite experience might have some ideas based on > these findings? Or if there are other concerns or considerations I > should be thinking about here? > > Best Regards, > Weston > > > _______________________________________________ > PROJ mailing list > PROJ at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/proj -- http://www.spatialys.com My software is free, but my time generally not. LLMs contribute to global warming and brain rot. Let's guillotine them! "Ah ! ?a ira, ?a ira, ?a ira !" -------------- next part -------------- An HTML attachment was scrubbed... URL: From weston.renoud at qps.nl Mon Aug 31 13:34:27 2026 From: weston.renoud at qps.nl (Weston Renoud) Date: Mon, 31 Aug 2026 20:34:27 +0000 Subject: [PROJ] Performance when Auxilary DB used In-Reply-To: <67609653-6a6c-448d-8e4b-53be4e776299@spatialys.com> References: <67609653-6a6c-448d-8e4b-53be4e776299@spatialys.com> Message-ID: Hey Even, Removing the ORDER BY clause removed the need to materialize and gives the same query performance as with the temporary tables, but without the need to create any temporary tables which took extra time. Adding back just the "ORDER BY score" still avoided the materialize, it seemed to be the "usage.auth_name, usage.code" that triggered the materialize. Are those necessary, or is "ORDER BY score" sufficient? Cheers, Weston ________________________________ From: Even Rouault Sent: Monday, August 31, 2026 6:43 PM To: Weston Renoud ; proj at lists.osgeo.org Subject: Re: [PROJ] Performance when Auxilary DB used You don't often get email from even.rouault at spatialys.com. Learn why this is important Hi Weston, thanks for this interesting analysis. I'm wondering if removing the ORDER BY clause would have a significant effect on the performance, if ordering would be the reason for materializing ? Sorting could be easily done on the C++ side. And the expected number of rows of the query in in 99% of the cases is just 1, and the remaining 1% 2, but SQLite has of course no clue about that. Otherwise you could try to submit the question to the SQLite experts at https://sqlite.org/forum/forum, but to have some hope of them having a look at that, you should provide an easy reproducer using the sqlite3 binary and provide the databases likely stripped down to only contain the needed tables to facilitate their analysis. Cheers, Even Le 31/08/2026 ? 18:13, Weston Renoud via PROJ a ?crit : Hi all, Even, Javier, and I discussed at FOSS4G EU some topics I could explore to get more familiar with PROJ and one of the ideas was to explore performance when using an auxiliary database. I have poked at the problem and wanted to share what I have found so far. TL&DR I think I may have a found an improvement that reduces the auxillary database overhead by about 50%. The diagnostics I'm using, and the improvement can be found in the branch https://github.com/OSGeo/PROJ/compare/master...wrenoud:PROJ:investigate-aux-db-performance I focused on the usage of an empty auxiliary database with cs2cs, with a simple projection of WGS 84 / UTM zone 1N coordinates. In windows command prompt this looks like: ????set PROJ_AUX_DB=/aux.db ????set PROJ_DEBUG=2 ????cs2cs EPSG:4236 EPSG:32601 test.txt I develop primarily on Windows and Linux for Intel CPUs, so I make use of Intel's Vtune Profiler when I'm looking to evaluate code performance. I instrumented the method `DatabaseContext::Private::run` with begin and end gates to time the SQL queries to try to identify if there are particular queries that perform especially poorly. What I found: Usage of `DatabaseContext::Private::run`: ????without auxiliary database: 43 SQL queries, totaling ~3ms. ????with empty auxiliary database: 221 SQL queries, totaling ~70ms. With the auxiliary database the extra queries and time could be broken down into 3 main categories * Setup - about 20ms (97 queries) of the 70ms when using the auxiliary database is related to initial setup * Duplicates - for queries that don't need cross referencing, they are called directly on each attached database and have comparable performance. * Usage query - about 40-50ms (11 queries) The setup does three main things: ATTACHING each database, querying the table structures, and creating temporary VIEWs to UNION the tables between the attached databases. These are required to support cross referencing from the auxiliary database(s) to items in the primary database, and it is not obvious there is any room for improvements. The duplicates on each attached database are necessary given the additional/auxillary database. Again, it is not obvious there is any room for improvements here. The usage query: of the queries on the UNION'ed VIEWs, the query from `AuthorityFactory::Private::createPropertiesSearchUsages` jumped out given the significant run time. Without an auxiliary database these queries sum to less than 0.1ms, but with an empty auxiliary database they sum to 40-50ms. This query requires a join between the usage, extent and scope tables. ????SELECT extent.description, ???????? extent.south_lat, ???????? extent.north_lat, ???????? extent.west_lon, ???????? extent.east_lon, ???????? scope.scope, ???????? (CASE WHEN scope.scope LIKE '%large scale%' THEN 0 ELSE 1 END) AS score ????FROM usage ???????????? JOIN extent ON usage.extent_auth_name = extent.auth_name AND usage.extent_code = extent.code ???????????? JOIN scope ON usage.scope_auth_name = scope.auth_name AND usage.scope_code = scope.code ????WHERE object_table_name = ? ???? AND object_auth_name = ? ???? AND object_code = ? ???? AND NOT (usage.extent_auth_name = 'PROJ' AND usage.extent_code = 'EXTENT_UNKNOWN') ???? AND NOT (usage.scope_auth_name = 'PROJ' AND usage.scope_code = 'SCOPE_UNKNOWN') ????ORDER BY score, usage.auth_name, usage.code Trying to optimize this query seemed like a good target, so for the next step I tried to learn more about it by making use of `EXPLAIN QUERY PLAN` (https://sqlite.org/eqp.html). Without the auxiliary database, the query execution is very straightforward and fast. It is executed as three searches by index or primary key: ????6,0,43,SEARCH usage USING INDEX idx_usage_object (object_table_name=? AND object_auth_name=? AND object_code=?) ????25,0,39,SEARCH extent USING PRIMARY KEY (auth_name=? AND code=?) ????31,0,34,SEARCH scope USING PRIMARY KEY (auth_name=? AND code=?) ????58,0,0,USE TEMP B-TREE FOR ORDER BY With the auxiliary database the query plan bloats to this due to the JOINs over the UNION'ed tables: ????2,0,0,CO-ROUTINE usage ????3,2,0,COMPOUND QUERY ????4,3,0,LEFT-MOST SUBQUERY ????7,4,43,SEARCH db_0.usage USING INDEX idx_usage_object (object_table_name=? AND object_auth_name=? AND object_code=?) ????37,3,0,UNION ALL ????39,37,216,SCAN db_1.usage ????69,0,0,MATERIALIZE extent ????71,69,0,COMPOUND QUERY ????72,71,0,LEFT-MOST SUBQUERY ????74,72,135,SCAN db_0.extent ????93,71,0,UNION ALL ????95,93,215,SCAN db_1.extent ????117,0,0,MATERIALIZE scope ????119,117,0,COMPOUND QUERY ????120,119,0,LEFT-MOST SUBQUERY ????122,120,97,SCAN db_0.scope ????132,119,0,UNION ALL ????134,132,215,SCAN db_1.scope ????146,0,196,SCAN usage ????167,0,0,BLOOM FILTER ON extent (code=? AND auth_name=?) ????188,0,53,SEARCH extent USING AUTOMATIC COVERING INDEX (code=? AND auth_name=?) ????201,0,0,BLOOM FILTER ON scope (code=? AND auth_name=?) ????213,0,53,SEARCH scope USING AUTOMATIC COVERING INDEX (code=? AND auth_name=?) ????245,0,0,USE TEMP B-TREE FOR ORDER BY Of note are SCAN and MATERIALIZE. When combined, they effectively duplicate the table's contents into temporary tables for the query. Based on this I then explored if creating tables, instead of views for the extent and usage tables would improve the performance. This dropped the total query time (for all 11 calls) to ~3ms, with an overhead of ~4ms to create the tables. That is over 30ms reduction. But again, this is with an empty auxiliary database. I will test further by putting some entries into the auxiliary tables and see if there is a significant difference. I'm also curious if anyone with more sqlite experience might have some ideas based on these findings? Or if there are other concerns or considerations I should be thinking about here? Best Regards, Weston _______________________________________________ PROJ mailing list PROJ at lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/proj -- http://www.spatialys.com My software is free, but my time generally not. LLMs contribute to global warming and brain rot. Let's guillotine them! "Ah ! ?a ira, ?a ira, ?a ira !" -------------- next part -------------- An HTML attachment was scrubbed... URL: From even.rouault at spatialys.com Mon Aug 31 13:43:42 2026 From: even.rouault at spatialys.com (Even Rouault) Date: Mon, 31 Aug 2026 22:43:42 +0200 Subject: [PROJ] Performance when Auxilary DB used In-Reply-To: References: <67609653-6a6c-448d-8e4b-53be4e776299@spatialys.com> Message-ID: Le 31/08/2026 ? 22:34, Weston Renoud a ?crit?: > Hey Even, > > Removing the ORDER BY clause removed the need to materialize and gives > the same query performance as with the temporary tables, but without > the need to create any temporary tables which took extra time. Adding > back just the "ORDER BY score" still avoided the materialize, it > seemed to be the "usage.auth_name, usage.code" that triggered the > materialize. Are those necessary, or is "ORDER BY score" sufficient? They are only necessary for reproducibility of results among SQLite versions, in case of equality of score.? There is no guarantee otherwise that different SQLite versions would return the same ordered result set. So they would need to be replicated on the C++ side ( std::sort() with an appropriate sort lambda) -- http://www.spatialys.com My software is free, but my time generally not. LLMs contribute to global warming and brain rot. Let's guillotine them! "Ah ! ?a ira, ?a ira, ?a ira !" -------------- next part -------------- An HTML attachment was scrubbed... URL: