[gdal-dev] Low Level Library

[Frank Warmerdam]

Folks,

I would like to start a discussion of how a low level support library 
should be organized. In the current GDAL tree I have a ``port'' directory
which I intend to hold:

o portability functions for system dependent functions. 

o convenience functions for common stuff (Tokenize() type stuff for
instance). 

o virtualized APIs for things that different application environments
might want to hook (memory management, file access)

o an application hookable error handling mechanism. 

o low level data objects for geospatial stuff ... perhaps the GDALShape.

Now a rationale for each, and some comments:

Portability Functions
---------------------

This would be simple stuff like strcasecmp() vs. stricmp() ... which is
standard? It would also include more sophisticated capabilities like 
getting a function pointer in a DLL/shared library (IMPGetSymbol()), and
so forth. This aspect would also have an include file (gdal_port.h) for
providing stuff like NULL, MAX(), ABS() and a byte order macro. 

We have a decent example in the PCI libraries for what we would want in 
this. 

One thing I am not too sure about it how to keep track of what system 
we are on. At PCI we defined a short list of well known SYSTEM_xx codes
for different operating system configurations, with a few broad defines
(SYSTEM_DF (dos filesystem), SYSTEM_UX (Unix)). However, I want GDAL to
be easily built on many configurations, including those I didn't plan in
advance. 

Does anyone know much about how ``configure'' based GNU products keep
track of stuff internally? I think this approach would be taken for GDAL
as well. 

Although, I also want to ensure that GDAL is easily built into commercial 
packages, with their own build mechanisms, and that might want to use 
configure configurations on each machine. What would we have to account 
for to handle this? For instance, I don't know how I would handle a 
``configure''ed package that I wanted to build within the PCI source tree. 
Normally, for stuff like libtiff, and the JPEG library I just stripped out 
much of the provided portability framework, trying to hook things up to 
the PCI way of doing things. 

Convenience Functions
---------------------

Likewise, we have some examples of what we might want in this category
from the PCI libraries as well. String parsing, filename parsing, a
``string list'' type. But how far do we go with this? I worry a bit 
about library size creep on this front. 

In particular, I think I should touch on one of the principles that I 
would like to apply to the low level support library. I think that it 
should be ``light''. I would like to be able to build code like the
shapefile access library that might be used without the rest of GDAL 
on the low level support library. But if someone wants to use one little
sub-library like that in an application, and are force to drag along 
100KLOC of support libraries, I think they will balk. This suggest that
we shouldn't be trying to build a sophisticated object frame work in the
low level library (such as the JOLTObject). 

... man, it's going to be harder to have these discussions once there is
someone on the list who doesn't share our common PCI heritage ... 

Anyways, I would appreciate input on what are important core convenience
capabilities. 

Virtualized APIs
----------------

Some software environments like to ``manage'' everything about the 
system. For instance, at PCI we liked all memory allocation to go through
HMalloc() and co, and all disk file access to go through IMPDIOC 
(DKOpen()...). This allows that software environment to track resource
usage, and do better debugging. It also allows virtualizing things to
look like files for instance. 

I could imagine that if Oracle wanted to include GDAL in their spatial 
data cartridge they might want to be able to operate on BLOBs in the 
database as if they were regular files. 

One of the things that I really liked about libtiff is that it gave a
standard way of hooking the memory management, and file access. 

I would like to use a similar approach within GDAL. To this end I have
defined the VSI (Virtual System Interface) API within GDAL. The set of
stdio, and memory functions now all have VSI equivelents that I would 
like all GDAL code under our direct control to use.

The PCI libraries started with a somewhat VAX/FORTRAN centric approach
to file access (DKGet(), etc.). Unlike this, I intend for the VSI 
functions to all be direct analogs to standard C (or Posix) functions. 
For instance VSIFopen(), and VSIFRead() are direct analogs to fopen(), and
fread(). No extra file type parameter, of whatever. 

Normally these functions just directly call the standard C functions. 


Error Handling
--------------

In the PCI libraries we had IMPError() and company. The old PACE model was
that all real errors were fatal, and resulted in immediate application
termination with a message. 

When I wrote the C libraries, and discovered that it wasn't OK for 
ImageWorks to exit() in response to a user error, I added the use of 
setjmp(), and longjmp() with IMPProtect()/IMPUnprotect() error rings
so that errors could be handled like an exception. However, the mechanism
was at times tricky, and it introduced a few (though suprisingly few)
bugs of it's own. 

I would like to implement an error handling system that:

o Allows applications to hook error handling functions, as long as
the satisfy some standard responsibilities. 

o Allows nicely formatted error messages from low level functions to be
transported up when they are needed. 

o Allows some very exceptional circumstances to be treated as fatal errors
that don't have to be further handled by the calling code. For instance,
the OGDI code tries to recover, and cleanup gracefully if an out of memory
condition occurs an any memory allocation. This greatly complicates their
code, and it seems likely that much of their ``cleanup'' code is likely
broken and might cause a crash of it's own. I would like to ignore the 
possibilty of running out of memory for small allocations calling a safe
malloc() that would just blow an exception if it runs out. 

For the E00 library Daniel wrote for Safe Software he implemented a simple
little error handling API (http://24.65.38.156/gdal/dev/frmts/e00/e00error.c).
This does some of what I want, but doesn't support some error types that are
fatal or trigger a non-local exception. All error reports still return to the
calling code which has to return to it's caller and so forth. 

I partially implemented a GDAL error handling module with a similar approach,
but that takes an error class (GE_None, GE_Warning, GE_Failure, or GE_Fatal). 
Of these GE_Fatal is considered to be fatal, and the caller can be assured
that control will not return to them. 

http://24.65.38.156/gdal/dev/port/gbserror.cpp

In EASI/PACE, this would be hooked to use IMPError(). In a commandline GDAL
program situation, the GE_Fatal's would just result in an exit() after 
reporting the error. 

I recently read an article in Dr. Dobb's Journal which described the GEF
(General Exception Facility) library. This is a C library for providing
C++ like exceptions using setjmp(), and longjmp(). On the downside it
is a bit heavy, but on the good side it is quite powerful. Should we
adopt it? 

See the gef stuff on the Dr. Dobb's web site:

http://www.ddj.com/ftp/1998/1998_11/

Or review a PostScript document on GEF:

http://24.65.38.156/~warmerda/gef.ps


Low Level Geospatial Support
----------------------------

Finally, let's not forget that GDAL is about GeoSpatial data. To that end
I think that a few base feature types (a GDALShape class/structure) should
be part of the low level library. This wouuld allow libraries such as
the shapelib library to be used standalone, but so that they would actually
be marshalling objects as GDALShape's internally so the features wouldn't
have to be re-marshalled into another type of object. 

Anyways, I am interested in peoples feedback on what is appropriate, what
isn't and what is missing. 

Best regards,

-----------------------------------+---------------------------------------
Who can give them back their lives | Frank Warmerdam, Software Developer
and all those wasted years? - Rush | URL http://members.home.com/warmerda
| warmerda at h...
------------------------------------------------------------------------
Free Web-based e-mail groups -- http://www.eGroups.com