[MetaCRS] CS-MAP patch: Cache NTv2 grid files in their entirety

[Paul Nalos]

Hi Hugues,

Would you consider reviewing the attached patch to cause CS-MAP to cache
NTv2 grid files fully in memory on first use? We've run into an issue where
not caching these files results in a substantial performance penalty.

This patch is not sufficient to satisfy the RFC 5 reentrancy requirements,
because grids are loaded when transforming the first point, not at
initialization time.

Details of our investigation in the context of our product, FME (
www.safe.com), follow below.

---

After upgrading to CS-Map 13.10 (r2144) from 12.02 (r1964), FME was
observed to slow down by two orders of magnitude while reprojecting ~44M
points involving a DHDN BeTa2007 NTv2 datum transformation.  This slowdown
only occurred on Windows; Linux remained performant.

Profiling indicated that:

* all time was being spent within fseek/fread calls
* we appeared to be doing an appropriate amount of work for each point
* whereas previous versions of FME registered almost no disk I/O during
this operation, the current FME shows massive disk I/O (as observed in
Process Explorer)

Tweaking setvbuf cache sizes had no observable impact on performance.  Diff
inspection of the upgrade did not find any changes obviously related to the
degraded performance.  Extensive Google searches did not result in any
recommendations of how to encourage Windows to return to its previous
caching behavior.

Erratic performance seemingly tied to file system caching has been observed
within FME before, with Japanese grids (CScalcTokyoToJgd2k).

One thing not explicitly tested that might have been interesting: Can a
slowdown of similar magnitude be observed in the CS-Map testbed?  This may
have indicated whether the way FME interacts with CS-Map, or the way FME is
managing other resources, is a factor.

Testing of patch:

* Logs of the Test and TestCpp programs were collected with and without the
NTv2 caching patch (random seed of '1' used in all runs).  Diffs indicate
no change in behavior.
* FME's own testsuite did not detect behavior changes during NTv2
reprojection

Regards,

Paul

––––––––––––––––––––––––––––––––

Paul Nalos | Database Team Lead

Safe Software Inc.
T 604.501.9985 x 301
paul.nalos at safe.com | www.safe.com

––––––––––––––––––––––––––––––––
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/metacrs/attachments/20130712/d7afd303/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: csmap_ntv2_grid_caching_tabs.patch
Type: application/octet-stream
Size: 13611 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/metacrs/attachments/20130712/d7afd303/attachment.obj>