[MetaCRS] Re: RFC #5 -- Multi-thread upgrade

[Paul Nalos]

Hi Hugues, MetaCRS,

> This has been discussed quite heavily and the good thing of this RFC is that it leaves the long term door open for large grid files like in Canada.

I agree. I think this flexibility (now and for later) is a very good approach.

> While for a single user the memory hit would not be bad, for other situations like in a server environment e.g. MapGuide that is using CS-Map, that could become a problem.

I haven't spent much time with MapGuide, but I took a look at the
source and architecture document, and my understanding is that there
is a single server process with multiple worker threads. Calls to
CS-MAP are protected by a critical section. (I hope that's efficient
given there is one call per point when a datum shift is in play.) If
this is the case, there would only be one copy of each grid in memory,
which sounds Ok to me. However, if I've understood incorrectly, it may
be that there are multiple MapGuide processes on a machine, which
could make this a more significant issue.

References:

http://trac.osgeo.org/mapguide/wiki/MapGuideArchitecture
http://svn.osgeo.org/mapguide/trunk/MgDev/Common/CoordinateSystem/CoordSysTransform.cpp
http://svn.osgeo.org/mapguide/trunk/MgDevMgDev/Common/CoordinateSystem/MentorUtil.cpp
http://trac.osgeo.org/mapguide/ticket/1463

Our FME Server product is based on process-level concurrency, and so
would be affected by the "many copies of each grid in memory" issue.

Overall, I don't dispute that the memory / performance trade-off is
real, or that different stakeholders may have conflicting requirements
or desires.

My sense is that all of these issues could be overcome through
additional effort and engineering (e.g., make the full-grid-caching
v.s. non-reentrant choice configurable, use shared memory for multiple
processes to store one copy of each grid, or devise a larger-grain
thread-safe caching mechanism), but support the current proposal: The
implementors will make the choice, for everyone, on a grid-by-grid
basis.

> You mention Lidar.
> What's the plan there? Is it that you'd be implementing a multithreaded transformation breaking up a point cloud into smaller clouds to speed up the process?

We don't have a detailed design yet. What I do know is that when we
reproject rasters and point clouds, we have huge numbers of
coordinates to reproject at once. These could be divided into groups
and handled by different threads.

Regards,

Paul

On Fri, Jul 15, 2011 at 2:38 AM, Hugues Wisniewski
<hugues.wisniewski at autodesk.com> wrote:
>
> Hi Paul,
>
> Thanks for your input
>
> >I am comfortable with the memory and performance cost of fully caching all grid files in use
>
> This has been discussed quite heavily and the good thing of this RFC is that it leaves the long term door open for large grid files like in Canada.
> While for a single user the memory hit would not be bad, for other situations like in a server environment e.g. MapGuide that is using CS-Map, that could become a problem. That's why the current proposal is to have not all grid file transformations treated the same way at first.
>
> You mention Lidar.
> What's the plan there? Is it that you'd be implementing a multithreaded transformation breaking up a point cloud into smaller clouds to speed up the process?
>
> Thanks
>
> Hugues
>
> -----Original Message-----
> From: metacrs-bounces at lists.osgeo.org [mailto:metacrs-bounces at lists.osgeo.org] On Behalf Of Paul Nalos
> Sent: Thursday, July 14, 2011 7:01 PM
> To: metacrs at lists.osgeo.org
> Subject: [MetaCRS] Re: RFC #5 -- Multi-thread upgrade
>
> On behalf of Safe Software, I'd like to express our "+1" for this
> proposal. (We're not part of the PSC.)
>
> These improvements would be a significant benefit to us and our users,
> and I'm comfortable with the trade-offs involved.
>
> We'd plan to integrate the new functionality into our product (FME)
> and test environment, with a focus on raster and LiDAR reprojection,
> where the performance gains would be most pronounced.
>
> I have one minor comment on the proposal. Quoting narrowly:
>
> > Unless the file is buffered in its entirity, the object becomes non-reentrant.
>
> I'm convinced there are other options for providing partial buffering
> while maintaining reentrancy (involving some kind of locking or
> critical section), but am comfortable with the memory and performance
> cost of fully caching all grid files in use.
>
> Regards,
>
> --------------------------------
>
> Paul Nalos | Database Team Lead
>
> Safe Software Inc.
> T 604.501.9985 x 301
> paul.nalos at safe.com | www.safe.com
>
> --------------------------------
>
> > I have posted an RFC to http://trac.geo.org/csmap concerning an upgrade to CS-MAP with regard to its ability to function in a multi-threaded environment:
> >
> >                http://trac.osgeo.org/csmap/wiki/CsMapRfc5
> >
> > Please review and comment as soon as is convenient for you.
> >
> > Thanks a lot.
> >
> > Norm