[gdal-dev] Re: Low Level Library

[Daniel Morissette]

Frank Warmerdam wrote:
> 
> Daniel Morissette wrote:
> > Once again, I will try to promote a clean separation of the files by
> > category (which do not absolutely mean directories :). For instance, I
> > would like it to be very easy to use only the portability and error
> > handling stuff while not bringing in anything else... a bit like the
> > idea that was behind the CCL, but this time let's try to do it before
> > the lib is built, instead of trying to separate things after the fact.
> 
> Daniel,
> 
> I agree with this in principle, though often one base component (ie
> error handling) may depend on another (ie. memory allocation).
> 

Frank,

My original idea was to go as far as providing a standard way to stub
the memory allocation stuff if somebody does not want to use it. It's
just that I had the E00 compress/uncompress library in mind, and this
lib is so simple that it would not be justified to carry the whole
memory allocation stuff when I only need the error handling part. But I
agree that this may be going a bit too far.

> > There may be some tools that already handle these problems???? If not,
> > then a simple way may be to have a directory file for each component:
> >
> > port.dir
> > mem.dir
> > file.dir
> > strlist.dir
> > convmisc.dir
> > error.dir
> > gdal.dir
> > ...
> >
> > each or these .dir file contains the list of all files from the /port
> > directory that belong to a given API. A .dir file could also contain
> > the name other .dir files if a module depends on other ones.
> >
> > Then (I hope) we could build a makefile that automatically grabs the
> > source files we need for our specific package at the time we release a
> > new source code version of the package or when we ship the deliverable
> > of a contract.
> 
> I agree, though perhaps with less subcomponents. Simple wrappers for
> memory and file access might be together for instance. I am doubtful
> whether splitting strings lists out from other convenience stuff is a good
> idea. I would certainly agree to splitting out GDAL specific convenience
> functions.
> 

The size of the components is a choice we have to make and that will
have a very big impact on how easy it is to derive a light version of
the CPL for specific needs... say we put together the memory and file
wrappers as you suggest, but my E00 lib (from my previous example above)
only needs error handling from the CPL... I end up pulling probably a
dozen of files... only because I wanted to use the error handling (1
single file).

I don't claim that I have a perfect solution, but that's the kind of
problem I would like to prevent. Perhaps a way to isolate components
would be to include stubs inside an #ifdef at the beginning of the
source files that have dependencies on external components.

For instance, say the error handling component has a dependency on the
memory allocation stuff, then the file error.c could stub the memory
allocation functions when they are not available:

/* Error.c ... Error handling functions
*/
#include ...
#include ...

#ifndef MEM_MODULE_AVAILABLE
/* Memory allocation module not available ... load stubs instead */
#include "vsimalloc_stubs.c"
#endif

...
...

And the file vsimalloc_stubs.c would contain:

/* vsimalloc_stubs.c ... stubs for Memory allocation module 
*/
#include <stdlib.h>

void *VSIMalloc(size_t nsize) 
{
return malloc(nSize);
}

/* ... and stub all other mem allocation functions */


When we release a source package, we could replace the #include
"*_stubs.c" with the actual stub code so that it looks cleaner to the
users...

OK, I realize that there is a possibility that all this may bring more
complications than real benefits since it probably won't be that often
that we release very small source code packages... what do you think?

> > Another point: you seem to be prefixing everything with GDAL (i.e.
> > gdal_port.h, etc.), but since I would like to include some of that
> > portability stuff in other packages, I suggest that we come up with a
> > generic name for everything that's part of the /port library.
> >
> > The only name I can think of for now is CPL, for Common Portability
> > Library... but I'm sure we can find something better than that... and
> > maybe even something with a dual meaning ;-)
> 
> I am agreeable to this. Let us plan on the low level library being called
> `cpl'. Within this I think we might want to break appart subsystems (ie.
> memory, convenience) to have their own prefixes.
> 
> For instance, VSI* for all the standard library cover functions. We could
> also have a standard prefix for some of the other systems (CES (Common Error
> System) for the errors, and so on.
> 

That's fine with me...

> > FYI, I have already implemented the stringlist, tokenize (basic one),
> > SPrintf(), SLPrintf()... actually SLPrintf() is pretty cool: it's like
> > fprintf() but it writes at the end of a stringlist! I agree that it may
> > not be very useful for GDAL, but...
> 
> I don't think it is too soon for sharing code directly. I am happy to
> provide access to my GDAL (and CPL) CVS trees to anyone who wants.
> 

I never used CVS before... does it allow remote checkin/checkout? I
think I already read something that made me believe it did...

> > I agree with adding an error class.
> >
> > It's unfortunate that there does not seem to be a "perfect" way to
> > handle errors in a program... the problem in our case is that we have to
> > make sure that what we choose will be compatible with anyone else's
> > approach...
> 
> One thing I noticed is that your API had an error number. I am wondering
> to what extent error numbers are really useful. I found myself more and
> more often using ERR_APPDEFINED (-1) as the error code with IMP, and just
> providing a meaningful message. Maintaining a meaningful list of error
> numbers central is a big pain in the butt. Used poorly the error number
> based messages often don't quite apply. The main use of error numbers is
> to ensure that a program can distinquish between different error conditions
> but I found that this was rarely used.
> 

I also used ERR_APPDEFINED a lot at PCI, and for the same reason you
mention, I didn't include error numbers in my first version of the
library, except that Safe asked that the E00 library return error
numbers that they can test after each library function call...

> One alternate approach might be to use error names, distinct from the error
> messages formatted for the user.
> 
> For instance, I might throw the "FileNotFound" error, rather that doing this
> with a code number. This makes it very easy to define new groups of errors
> for a particular package just by making your own name space. For instance,
> I could have a large family of GDAL specific errors which I just prefix
> with "gdal", such as "gdalIllegalShapeid".
> 
> In general when error numbers are used, there is a corresponding set of
> macros so that you can use symbolic values anyways. Why not just make the
> symbolic value (a string) be the value?
> 

The main problem I see with that is that testing error conditions on a
string (with strcmp()) can become quite expensive (in CPU cycles)... for
organisations that check the current error status after each function
call.

I see the benefits of the name space, but before I give up, let me do a
last suggestion assuming that we would use error numbers and macros. 
Maybe we could define a Error code space for each module in a global
header:

#define VSI_ERR_BASE	1000
#define CONV_ERR_BASE	1200
#define GDAL_ERR_BASE	2000
...
...

And in gdal.h, you would define the error codes for GDAL as:

#ifndef GDAL_ERR_BASE
#define GDAL_ERR_BASE 2000
#endif

#define GDAL_UNKNOWN_FORMAT GDAL_ERR_BASE + 1
#define GDAL_INVALID_HANDLE GDAL_ERR_BASE + 2
...
...

> We could have some well known error names used by convention (such as
> "FileNotFound", and "OutOfMemory"), while others are adhoc. When writing
> documentation for a function you can just show the error names that may
> be ``thrown''. A call might look like:
> 
> CESError( CE_Fatal, "OutOfMemory",
> "Out of memory allocating shapeid buffer for file\n%s\n",
> pszFilename );
> 

This would be nice for sure... but how widely is this kind of approach
used? We have to keep in mind, mostly for GDAL drivers, that the error
handling mechanism must be compatible with the one of the applications
that will use GDAL.

> Of course, some of our policy would be set for us if we choose to use
> GEF.
> 

GEF sounds interesting, I wanted to look at the author's web site, but
he moved without leaving any address... have you seen GEF used somewhere
in real life? I would like to see what the code looks like before I
make a final opinion about it.

Anyways... that's a couple more things to think about... on my Ski-Doo
tomorrow ;-)

Later,
-- 
------------------------------------------------------------
Daniel Morissette danmo at v...
http://pages.infinit.net/danmo/
------------------------------------------------------------
Don't put for tomorrow what you can do today, because if 
you enjoy it today you can do it again tomorrow.

------------------------------------------------------------------------
Free Web-based e-mail groups -- http://www.eGroups.com