[fdo-internals] Re: [mapguide-internals] Please test MGOS 1.2.0 Beta 2

Frank Warmerdam warmerdam at pobox.com
Wed May 2 10:08:09 EDT 2007 

Andy Morsell wrote:
> Thanks for the information Frank.  Yes, I will still contribute that money
> to the project(s) in some way or another.
> 
> What more can you (or Autodesk) tell us about the GDAL contract you are
> working on?  Are they doing away with the ATIL provider, or is there going
> to be convergence of the best of both?  Is their intention to release
> anything you develop back to the GDAL and FDO projects?
> 
> Thanks for any information that you can provide.

Andy,

I am including the technical statement of work, which I don't believe is
confidential in any way.  Of course if I'm wrong, I'll no doubt burn in
h***.

I have completed Phase 1, and included a few items from Phase 2, but
haven't otherwise started Phase 2 in earnest which will presumably
live only in FDO 3.3.  Yes, I really am 15 months late and counting
on Phase 2 (and many months late on Phase 1).  Autodesk has been very
understanding, bless their souls.

It is my understanding that Autodesk has no desire to replace their
ATIL provider with this GDAL one, though it is our objective to keep
them very compatible.  I *hope* that some day they will find some of
the formats in the GDAL provider interesting enough that they will
also offer the GDAL provider as a supported provider for MG Enterprise
and Map, in addition to the ATIL one.  But I think it still requires
some "proving out" before they might be willing to do that.

This proposal does not cover any sort of tile indexing capability,
but it is my intention to support the new "bounds in xml config"
stuff being done for the ATIL provider and likely also some sort
of automated "keep a tileindex in memory as long as the provider
is loaded" sort of mechanism on a pro-bono basis.

One thing that has been holding me back on this a bit, is that it
seems like something that doesn't belong in FDO 3.2 (major feature
improvements wouldn't seem to belong in a stable branch), but
doing the work in 3.3 won't help any mapguide users for ... a
long time?  The disconnect between MapGuide's released (or even
development version) and the development version of FDO just sucks
all the enthusiasm out of me for new work.  It seems like it will
be at least many months, if not years before it is actually in
peoples hands.


	Proposal to Implement GDAL Based FDO Raster Provider
	====================================================

Summary
-------

A GDAL based raster provider will be implemented for FDO providing
read access to all GDAL supported geospatial raster formats in Tux
and other FDO based applications.


Features
--------

In order to provide a relatively near term raster capability in the
open source Tux, the implementation of a GDAL raster provider is being
split into two phases.


Phase 1:

  * Support for "default" schema derived when a single file, or directory is
    identified in the connection string.

  * Support for efficient downsampling.

  * Support for establishing the schema and overrides from a configuration file.
    The same format used for the existing ATIL based raster driver will be
    used.

  * Support for the Clip operation.  This will be handled directly by the
    provider, without creating a VRT dataset, ensuring efficient subwindow
    clipping.  Where a clip windows does not fall on pixel boundaries, the
    clip window will be implicitly rounded out to the nearest pixel boundary.

  * Support for reading paletted image palettes as Auxilary Metadata using the
    same approach as the ATIL raster provider.

  * Support will be provided for RGBA or RGB images to be treated as single
    band RGBA images.

  * Other multispectral images will be treated as multi-band greyscale, or
    "data" images.

  * An override mechanism will be provided via the configuration file to
    treat 3 or 4 band 8bit images as RGBA even if they are not internally
    identified as RGBA.

  * Full data model information will be returned via the FdoRasterDataModel
    interface.


Phase 2:

  * Support for rotation of non-north up images.  These will be rotated by
    creation of an on-the-fly "VRT" dataset to perform the warping.  There
    will be a non-trivial performance penalty for this operation.

  * Support for the MOSAIC operation.  This will be implemented by creation of
    an on-the-fly "VRT" mosaic.  There will be a non-trivial performance penalty
    for this operation.

  * Support for changing the datamodel (ie. bit depth, pixel type, data
    organization).  The one exception is that color model changes will not
    be supported.

  * Support for reading the coordinate system from the raster file and
    performing any needed translation to Autodesk-WKT.

  * Support for reading GDAL band and dataset metadata as Auxilary Properties.

  * We *could* easily implement on the fly reprojection in Phase 2 (as part of
    the VRT layering) if desired, though it isn't clear that the ATIL provider
    currently implements that.



Derivation from ATIL Raster Provider
------------------------------------

It is intended that the GDAL raster provider infrastructure will be cloned
from that already implmented for the ATIL based provider.  That is the "Rdf"
code tree will be copied and modified but the ATIL Schema Interface and
ATIL itself will not be used in the construction of the GDAL provider.

This approach will accelerate the implementation of the GDAL provider,
ensure it remains configuration file compatible with the ATIL provider and
ensure correct implementation of some generic needs.


Technical Issues
----------------

  * The raster provider will be thread safe.  Due to limitations in GDAL,
    a given GDALDataset will only be readable on one thread at a time.  The
    provider will implement a mutex to enforce this restriction.  However,
    distict GDALDatasets may be safely concurrently accessed in different
    threads.

  * All errors will be issued via C++ exceptions in the provider, including
    translation of GDAL error return codes into C++ exceptions.

  * It is anticipated that support support will be required from Autodesk
    engineers.  Providing updated source code, answering interpretation
    questions about technical issues, code reviews, etc


Build Infrastructure
--------------------

I shall attempt to duplicate and adapt the appropriate Linux build scripts for
the raster provider.  Construction of an appropriate Win32 Visual Studio 2005
solution will be left to Autodesk personeel.

It is anticipated that GDAL itself will not be part of the provider, and will
live in the thirdparty tree (like Xerces, Xalan, etc).  So normal GDAL build
mechanisms will be used for GDAL.


Unit Tests
----------

A set of unit tests will be implemented, derived from the ATIL provider
unit tests.


Documentation
-------------

I shall provide support and input on development of appropriate end user
documentation for the provider, up to 16 hours of effort.


Intellectual Property
---------------------

The GDAL based FDO raster provider copyright will be assigned to Autodesk,
or the Foundation as appropriate and the code will be placed under the same
open source license as FDO will be placed under.  (LGPL?)

The licensing of GDAL will be remain unchanged, even if some improvements are
made in it in support of the FDO provider.


Delivery
--------

Phase 1 will be completed and delivered by December 31st, 2005.

Phase 2 will be completed and delivered by February 10th, 2006.

Ongoing integration support will be provided after that time, without
limitation, within the scope of the above proposal features.

Interim versions will be made accessable before phase deliveries to facilitate
code review, and other feedback on implementation.


Code Repository / Bug Feedback
------------------------------

I would like to see this provider being the first component actually living
in a public subversion repository (possibly before the "live" FDO and Tux
code can be transferred there).  Also, I would like to see defect tracking done
via a public bugzilla.  My intent is that this project would give us the
opportunity to start exercising public collaberation tools and approaches.

-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org

More information about the fdo-internals mailing list