[gdal-dev] Re: GFC

[Frank Warmerdam]

Liujian Qian wrote:
> Frank,
> 
> I just sent out an invitation letter about joining my newly created
> gfc-dev list. Hope to see you on board ;)

L.J.,

I am there. :-) Because I think the following discussion will be of
substantial interest to GDAL and GFC developers, and am crossposting
to both lists. It isn't my intention that this would always occur, but
for some core data model issues it may be helpful to discuss in both
forums. Certainly with the limited subscriptions currently in place I 
don't think the cost is high. 

> It seems that GDAL is kicking off nicely? Has your CVS server up and
> running?

Yes, Daniel has successfully downloaded source, and updated the master with
portability fixes. 

> I was wondering the other day whether a uniform call interface for
> raster-type data importer is possible, something like:
> 
> class GGridImporter
> {
> public:
> virtual int open (const char* ) = 0;
> virtual void fillGridMeta (GGridMeta& meta) = 0;
> virtual void customHeader (char*& cust_hdr, int& len) = 0;
> virtual int numColumns () = 0;
> virtual int numRows () = 0;
> virtual GCellType baseType () = 0;
> virtual GCellValue ignoreValue () = 0;
> virtual void scanline (int row, char* buf) = 0;
> virtual int close () = 0;
> };
> 
> Then from GGridImporter you can derive any number of custom importers
> that follow this public interface, such as DEMImporter, or
> ArcGridImporter. Now suppose my general grid class GGrid has the following
> two methods:
> 
> GGrid::installImporter(GGridImporter* importer);
> GGrid::importIt();
> 
> where GGrid::importIt() uses only the public interface defined by the
> above GGridImporter to read in all the external data.
> 
> Now if one of my GGrid object (my_grid) decides to
> import some data from a USGS DEM file, It just needs to do the following:
> 
> my_grid->installImporter(new DemImporter);
> my_grid->importIt();
> 
> Later on if the same GGrid object wants to discard the DEM data but
> make use of some Arc/Info grid data, it can do:
> 
> my_grid->installImporter(new ArcGridImporter);
> my_grid->importIt();

The above approach is good, and fairly analygous to the approach I am 
planning within GDAL. I will briefly explain that:

GDALDriver:
This is the base class for each of the format drivers. The 
driver itself is mostly just responsible for reporting some 
general information about the format (or gateway). For instance
the kind of information you would need to populate dialogs allowing
dataset creation before an instance of the format exists. 

Some GDALDrivers will actually be drivers for another underlying 
multi-format API (such as an OGDIDriver) and so I want to ensure 
that would driver can represent a variety of underlying formats
effectively. I am not sure what all this will entail. 

The GDALDriver derived driver class is also responsible for 
dataset opening, and creation. Once the dataset is instantiated
all further data access is the responsibility of the dataset
(GDALDataset). 

In theory one GDALDriver might be responsible for handling a
more than one GDALDataset derived class, though it isn't clear
why this would be needed.

GDALDriverManager: 
This singleton class maintains a list of GDALDriver's. 
In addition to the ability to register a driver with it
there are (or will be) methods to manipulate the order in
which they are tried on unknown datasets. 

The GDALOpen() function essentially passes the dataset to each 
GDALDriver derived driver in turn till one recognises it. 

GDALDataset:
Particular format classes are derived from this base class, which
is a close analog to your GGridImporter. It contains a variety of
virtual methods, some with default implementations and others that
are pure. The derived class of course fills these in. 

It is my intention to use the GDALDataset baseclass for both raster
and vector formats. In part this comes from my grounding in PCIDSK
format which held both raster and vector information. This really
doesn't seem to happen with many other formats, so I don't know if
keeping this commonality is important. 

Comments on GGridImporter:

Multiple Bands:
Do you intend that applications would instantiate a GGridImporter for
each band (channel) of multi-band raster files? Would there be a class
to to relate the different bands? In fact a GDALDataset represents the
entire dataset, and a GDALRasterBand is instantiated for each band. At
the least you might consider extending the GGridImporter to allow 
reporting of the number of bands, and access to a requested band. 

Blocking:
In GDALDataset I have provided three core methods to access raster data.
The simple ReadBlock(), and WriteBlock() methods are intended to simply
read, or write a block of raster data in the ``optimal'' blocking
organization of the underlying dataset. The idea is that when a band is
initially accessed it would return information on it's optimal blocking. 
For many formats the block size would be one scanline. For tiled 
formats the optimal block size would be the tile size. Within GDAL I
am assuming a uniform block size. A format like TIFF might report a
whole strip (usually a number of scanlines amounting to no more than 8K
of data) as the natural block size. 

In order to facilitate efficient access to modern tiled formats, I think
this could be important.

RasterIO:
The third raster access method on the GDALRasterBand is the RasterIO()
method. This is a complex call that allows for rectangular window
resampling, and alteration of datatypes on the fly. The caller expresses
the size of the dataset window, the size of their buffer, and the datatype
of their buffer. The method takes care of resampling, and type conversion.

This method will have a default implementation on the base GDALRasterBand
class, but the intention is that formats with overviews, or that have other
specialized ways of satisfying such requests can override the method to 
to do it more efficiently than the default implementation. 

API Bridges:
Finally, I am thinking about how I would build a single GGridImporter 
interface for GDAL. In effect this wouldn't be very hard, but I think it
would be difficult to make much specialized information about the real
underlying format available. In essence, I would like to be able to report
back some information about the underlying database, such as a short and
long format name that is not fixed by the class (derived from GGridImporter).

> This seems to provide a clean interface for making use of raster data format
> converters that you guys are developing. Comments and suggestions? Is such
> a uniform interface possible? What about vector data formats?

I do think such a clean interface is possible, though certainly the complexity
depends in part on the objectives. It is an objective of GDAL that 
applications be able to operate on any GDAL supported file as if it were
a native file (within the restrictions of the target file format). 

The class you have described is a grid _importer_. Does this mean that 
within your system you are intending to convert data to your own internal
format for further access, or do you intend that the user be able to 
do operations directly with data in other formats? Either is a reasonable
goal, but if you want to be able to do operations directly on foreign 
format data you will need to substantially complicate the methodology. 

> One last thing, in your shapelib 1.2, I found that you use a for loop to
> swap the byte order of shorts/ints/doubles, this seems to be a bit slow
> compared to macros that are based on bit-shifting, particularly on 64-bit
> machines or for the case of shorts/ints (sometimes the macro version is
> 4 or 6 times faster). If you are interested in testing my macro-based
> byte order swaping routine, I can certainly send it to you.

I would be interested in getting these macros for incorporation into the 
CPL (Common Portability Library) (currently in gdal/port). When Shapelib
is brought into the GDAL tree I will update it to use them. I have at least
one client interested in having Shapelib be as efficient as reasonably 
possible. 

PS. It would be nice if the gfc-dev Description contained a clickable
URL for the GFC project web page. 

Best regards,

-----------------------------------+---------------------------------------
Who can give them back their lives | Frank Warmerdam, Software Developer
and all those wasted years? - Rush | URL http://members.home.com/warmerda
| warmerda at h...
------------------------------------------------------------------------
Free Web-based e-mail groups -- http://www.eGroups.com