[gdal-dev] Re: Low Level Library

[Daniel Morissette]

Frank,

I think the timing is very good to discuss these things, since I have to
make these same decisions myself about a base library for OWT (for those
who don't know about OWT yet, see
http://pages.infinit.net/danmo/owt/owt.html). As you know, I built a
first implementation of OWT very quickly while building a demo, so it's
quite disorganized now and I have to do a big cleanup/reorganisation in
it before I go any further.

I also need a set of portability and convenience functions for OWT, I
have already some stuff implemented and my hope is that we could share
the same low-level libs.

I'll give my first reactions to what you wrote, but I'll probably come
back on the subject later, after putting some more thoughts in it.

Frank Warmerdam wrote:
> 
> Folks,
> 
> I would like to start a discussion of how a low level support library
> should be organized. In the current GDAL tree I have a ``port'' directory
> which I intend to hold:
> 
> o portability functions for system dependent functions.
> 
> o convenience functions for common stuff (Tokenize() type stuff for
> instance).
> 
> o virtualized APIs for things that different application environments
> might want to hook (memory management, file access)
> 
> o an application hookable error handling mechanism.
> 
> o low level data objects for geospatial stuff ... perhaps the GDALShape.
> 

Once again, I will try to promote a clean separation of the files by
category (which do not absolutely mean directories :). For instance, I
would like it to be very easy to use only the portability and error
handling stuff while not bringing in anything else... a bit like the
idea that was behind the CCL, but this time let's try to do it before
the lib is built, instead of trying to separate things after the fact.

It should be more than just #defines in the headers, because I would
like to have a simple way to grab all the C/C++ source files and headers
corresponding to the APIs that I selected when I prepare a new release
of a source package (other than GDAL) that uses the stuff in /port.

The first thing I can think of is to create separate directories, but
since I know you're not a big fan of that Frank ;-) and I realize that
this sometimes leads to complicated makefiles, we could try to think of
something else.

There may be some tools that already handle these problems???? If not,
then a simple way may be to have a directory file for each component:

port.dir
mem.dir
file.dir
strlist.dir
convmisc.dir
error.dir
gdal.dir
...

each or these .dir file contains the list of all files from the /port
directory that belong to a given API. A .dir file could also contain
the name other .dir files if a module depends on other ones.

Then (I hope) we could build a makefile that automatically grabs the
source files we need for our specific package at the time we release a
new source code version of the package or when we ship the deliverable
of a contract.

> Now a rationale for each, and some comments:
> 
> Portability Functions
> ---------------------
> 
> This would be simple stuff like strcasecmp() vs. stricmp() ... which is
> standard? It would also include more sophisticated capabilities like
> getting a function pointer in a DLL/shared library (IMPGetSymbol()), and
> so forth. This aspect would also have an include file (gdal_port.h) for
> providing stuff like NULL, MAX(), ABS() and a byte order macro.
> 


Another point: you seem to be prefixing everything with GDAL (i.e.
gdal_port.h, etc.), but since I would like to include some of that
portability stuff in other packages, I suggest that we come up with a
generic name for everything that's part of the /port library.

The only name I can think of for now is CPL, for Common Portability
Library... but I'm sure we can find something better than that... and
maybe even something with a dual meaning ;-)

> We have a decent example in the PCI libraries for what we would want in
> this.
> 
> One thing I am not too sure about it how to keep track of what system
> we are on. At PCI we defined a short list of well known SYSTEM_xx codes
> for different operating system configurations, with a few broad defines
> (SYSTEM_DF (dos filesystem), SYSTEM_UX (Unix)). However, I want GDAL to
> be easily built on many configurations, including those I didn't plan in
> advance.
> 
> Does anyone know much about how ``configure'' based GNU products keep
> track of stuff internally? I think this approach would be taken for GDAL
> as well.
> 

I don't personally know much about it... but I think we'll have to learn
it if we release source code stuff... it must do a good job if so many
packages use it. I'll probably give it a look when I release a first
version of OWT... but I don't know when this will be. (No release
date... this way I'm sure I won't break the deadline ;)

> Although, I also want to ensure that GDAL is easily built into commercial
> packages, with their own build mechanisms, and that might want to use
> configure configurations on each machine. What would we have to account
> for to handle this? For instance, I don't know how I would handle a
> ``configure''ed package that I wanted to build within the PCI source tree.
> Normally, for stuff like libtiff, and the JPEG library I just stripped out
> much of the provided portability framework, trying to hook things up to
> the PCI way of doing things.
> 
> Convenience Functions
> ---------------------
> 
> Likewise, we have some examples of what we might want in this category
> from the PCI libraries as well. String parsing, filename parsing, a
> ``string list'' type. But how far do we go with this? I worry a bit
> about library size creep on this front.
> 

FYI, I have already implemented the stringlist, tokenize (basic one),
SPrintf(), SLPrintf()... actually SLPrintf() is pretty cool: it's like
fprintf() but it writes at the end of a stringlist! I agree that it may
not be very useful for GDAL, but...

> In particular, I think I should touch on one of the principles that I
> would like to apply to the low level support library. I think that it
> should be ``light''. I would like to be able to build code like the
> shapefile access library that might be used without the rest of GDAL
> on the low level support library. But if someone wants to use one little
> sub-library like that in an application, and are force to drag along
> 100KLOC of support libraries, I think they will balk. This suggest that
> we shouldn't be trying to build a sophisticated object frame work in the
> low level library (such as the JOLTObject).
> 

I agree that it must be light... or that it must be possible to select
only what we need as I described above.

Hopefully if we separate things well, and try to avoid dependencies
between modules as much as possible, we could have a quite powerful
library, and be able to grab a light version of it when needed.

> 
> Error Handling
> --------------
> 
> In the PCI libraries we had IMPError() and company. The old PACE model was
> that all real errors were fatal, and resulted in immediate application
> termination with a message.
> 
> When I wrote the C libraries, and discovered that it wasn't OK for
> ImageWorks to exit() in response to a user error, I added the use of
> setjmp(), and longjmp() with IMPProtect()/IMPUnprotect() error rings
> so that errors could be handled like an exception. However, the mechanism
> was at times tricky, and it introduced a few (though suprisingly few)
> bugs of it's own.
> 

I don't think we should have something that complicated in a library
that you make publicly available. Unless you're ready to write tons of
documentation and do training seminars about it...

> 
> I partially implemented a GDAL error handling module with a similar approach,
> but that takes an error class (GE_None, GE_Warning, GE_Failure, or GE_Fatal).
> Of these GE_Fatal is considered to be fatal, and the caller can be assured
> that control will not return to them.
> 
> http://24.65.38.156/gdal/dev/port/gbserror.cpp
> 

I agree with adding an error class.

It's unfortunate that there does not seem to be a "perfect" way to
handle errors in a program... the problem in our case is that we have to
make sure that what we choose will be compatible with anyone else's
approach...

> 
> I recently read an article in Dr. Dobb's Journal which described the GEF
> (General Exception Facility) library. This is a C library for providing
> C++ like exceptions using setjmp(), and longjmp(). On the downside it
> is a bit heavy, but on the good side it is quite powerful. Should we
> adopt it?
> 

I'll have a look and I'm sure I'll have some comments after that ;-)

Later,
-- 
------------------------------------------------------------
Daniel Morissette danmo at v...
http://pages.infinit.net/danmo/
------------------------------------------------------------
Don't put for tomorrow what you can do today, because if 
you enjoy it today you can do it again tomorrow.
------------------------------------------------------------------------
Free Web-based e-mail groups -- http://www.eGroups.com