[Gdal-dev] ArcSDE Raster Driver for GDAL

[Howard Butler]

Folks,

I have received a commitment for funding the development of a GDAL  
driver for ESRI ArcSDE rasters.   I am providing the technical  
proposal to the PSC to ask for permission to implement and include it  
in GDAL, and I ask for detailed technical feedback on it if anyone  
has any.

Thanks

Howard

	ESRI ArcSDE Raster Driver for GDAL
========================================

Purpose
=======

To implement a new GDAL driver for reading ESRI ArcSDE Raster data.


Technical Details
=================

Overall
-------

The ESRI ArcSDE (SDE) Raster driver will connect through the SDE C  
API SDK to
read raster data from SDE servers, and it will require linkage to the
C API SDK (available for Linux, Windows, and Solaris ) as well as a
license to use the SDK (often an SDE license itself, or permission/ 
agreement
from ESRI to redistribute).

OGR already provides vector read support for ArcSDE.  Configuration
infrastructure is already in place on both the Linux and Windows  
platform
for the vector side.  This infrastructure will also be utilized for the
raster side.

Observations
------------

* This driver implementation will only provide *read* support for ArcSDE
   Rasters.  Storing rasters in ArcSDE with the GDAL library or GDAL  
tools
   will not be implemented, but it should be possible to implement it  
at a
   future time.

* Due to library/linking limitations, support for ArcSDE Rasters on
   Solaris will require that GDAL with ArcSDE raster (and vector)  
support
   be compiled with the Sun Pro compiler.

* "Direct connect", or direct database connection and data extraction
   *will not* be provided by this driver.  Usage of this driver will  
require
   linkage to the ArcSDE C API SDK.

* ArcSDE supports n-band imagery, and the driver will take
   advantage of this feature.

* ArcSDE supports the concept of "raster catalogs," where multiple  
rasters
   are mosaic'd into a common layer with a common color table and  
combined
   extent.  The ArcSDE raster driver will attempt to play nice with  
these
   layers, but there may be unforeseen complications.

* ArcSDE internal colormaps will be supported.

Performance Considerations
--------------------------

* ArcSDE connections typically take 1-2 seconds for authentication  
and setup
   via the ArcSDE C API.  This penalty will be taken upon each  
GDALOpen call.

* For applications like MapServer, it will be important to support
   "fast-large-reads," and this will probably mean overriding the  
RasterIO
   machinery within GDAL.  This usage model is probably most  
prevalent in
   pure data format transformation operations as well.

* The "lots-of-small-reads" usage model could be problematic to make  
performant
   due to network overhead and database overhead (ArcSDE makes
   database joins for each request which will be unavoidable).

Specification Features
----------------------

The ArcSDE GDAL raster driver will attempt to take advantage of most
of the features that ArcSDE provides with respect to rasters.  These  
items
include:

*  Proper NODATA support

*  Reasonable attempts to access data stored in all the various
    formats (LZW, JP2K, etc) ArcSDE provides will be made.

*  Properly configured and stored ArcSDE Raster Catalogs *should* be
    supported, given that all data in the catalog are stored in a  
similar
    format.

*  GDAL configuration options will be provided to adjust operations  
of the
    ArcSDE raster driver to accommodate certain performance  
considerations.

*  "Plugin" support for the ArcSDE raster driver will be provided in  
a manner
    similar to the ArcSDE vector driver.  This will optionally allow  
a user
    to dynamically load the driver at runtime.  Additionally, this  
will allow
    users of MS4W to take advantage of the driver if they have the  
ArcSDE
    C API SDK.

*  The driver will take advantage of internally stored overviews of the
    data.  Externally generating overviews using GDAL and storing and  
reading
    them via disk will not be supported initially.

Connection Specification
------------------------

A connection to an ArcSDE raster layer will be specified in a similar  
fashion
to a vector layer:

-- SDE:sdemachine,port:5151,database,username,password,layername

A practical and convenient solution for multiple raster "layers"  
would need to
be identified to query the entire database to report information on  
all of the
raster layers available.  One approach might be to follow the SUBDATASET
metadata model that HDF uses, although in databases with lots of  
layers, this
could perform poorly.


Spatial referencing
--------------------

OSR (contained in the GDAL source tree) provides extensive functionality
for transforming back and forth between ESRI-encoded SRS WKT.  This
functionality will be utilized to support coordinate systems that are
properly described within ArcSDE raster layers.  It will continue to be
possible to override SRS information manually in a similar fashion to
other GDAL drivers.


Testing
-------

ArcSDE versions 9.1 and 9.2 will be tested.
Accommodations to use earlier versions of ArcSDE will be made where  
appropriate
(and identified), but testing of the driver will not happen on those  
versions
of either the SDK or the server itself.  Both 32-bit Linux and Windows
environments will be tested.  Initially, 64-bit platforms will not be  
tested,
but any issues that are identified will be corrected.

RGB, RGBA, and multi-band imagery will be used as test cases.  A raster
catalog of typical color ortho imagery will be used to test the "catalog
layer" case. Usage with GDAL VRT will be tested.


Documentation
-------------

Standard "user" documentation will be written for the ArcSDE raster  
driver
in HTML format.

Intellectual Property
---------------------

The ArcSDE raster driver will become a regular part of the GDAL library,
and its source code and documentation will be released under the normal
GDAL open source license.