[gdal-dev] Re: [SoC] GDAL WKT Raster driver weekly report #6

[Frank Warmerdam]

Jorge Arévalo wrote:
> Hello,
> 
> My weekly report: 
> http://www.gis4free.org/blog/2009/07/04/gsoc-09-weekly-report-6-2606-0307/
> 
> As usual, the project page: http://trac.osgeo.org/gdal/wiki/WKTRasterDriver

Jorge,

I am reviewing your documents at:

   http://trac.osgeo.org/gdal/wiki/WKTRasterDriver

I will try answer some outstanding questions and provide a few other
comments.

1) -MODE=ONE_FILE_PER_ROW - there is a GDAL concept of subdatasets that
might be applied to this.  To learn more read the "SUBDATASETS Domain"
section of the data model document (http://www.gdal.org/gdal_datamodel.html).
The idea would be for the ONE_FILE_PER_ROW mode to return a set of
subdatasets - one per row - with the row# encoded into the subdatset name.
Perhaps MODE=FIRSTROW would instead become MODE=ROW:# with a number being
the offset in the result set.  The danger to this approach is that if there
are many records in the result set it produces an unmanagable sized
subdatasets list - this is more suitable for dozens or hundreds of
subdatasets than it is for millions of subdatasets.

Will datasets where only a single row in the table is being addressed
also be treated as a single block for GDAL purposes?  That is - will
the whole image be read into RAM and returned to the user?  Or will there
be some attempt to provide a finer grained access?  If fine grained access
is to be supported it will be complicated to keep the image accessor
from one blockread/rasterio request to the next in such a way that
no extra work needs to be done.

2) MODE=WHOLE_TABLE
   - this is really the fully unstructured case.
   - It should be possible to use the extent from RASTER_COLUMNS if it is
     not null instead of scanning the whole table.
   - In combination with the pixelsize_x and pixelsize_y you can compute
     an image size in pixels and lines.  It *may* be desirable to support
     irregular images without even an entry in the RASTER_COLUMNS table
     in which case it might be necessary to scan all the rasters to establish
     a minimum pixel size from the actual images.  Ouch!
   - If/when MODE=WHOLE_TABLE is implemented it is important to understand
     that the whole result image cannot be assembled in memory.  Each time
     a block or RasterIO request is made it will be necessary to issue a
     SELECT statement with a corresponding spatial bounds to pull back
     images in the request region, and then mosaic them into the buffer as
     appropriate.  This will be hard since there isn't much supporting
     machinery in GDAL for this sort of case.

The doc does not seem to be specific on this point, but I assume the proposed
MODE parameter would optionally be part of the datasource name?

3) I agree with Pierre that no effort should be made at this time to
address the multi table tiled image.  I don't really understand that model
yet.

4) "QUESTION: Should the RasterBand? read all the tiles (rows of the raster 
table) and put them consecutives in the same memory area? If I expect this from 
RasterBand? class, the memory allocation is easy."

I'm afraid I do not understand this question.  Is it in regard to
regular_blocking images or one of the other cases?  I definately do not
believe the driver should be holding too much stuff in memory at once
- at most the raster data of one row at a time.  In the regular blocking
case I assume the rasterband blocksize will match the size of each
tile in the table and that access will be all handled via the
IBlockRead() method.

5) "The full set of possible GDALDataType values are declared in gdal.h,
and include GDT_Byte, GDT_UInt16, GDT_Int16, and GDT_Float32, but we,
probably, only are going to consider GDT_Byte to this project. "

I am disappointed with this statement.  I would much rather we leave off
stuff like MODE=WHOLE_TABLE, but do a comprehensive implementation of the
full set of data types.  It really should not add much complexity.  For
the midterm evaluation a byte only prototype would be fine of course.


6) "The data fetched with the Raster Band, is in HEXWKB format? (I think so) I 
should code a method to transform to this format to another ones..."

Yes, I believe you will need to implement a HEXWKB to binary decoding taking
care to convert to local machine byte order if necessary. You might want
to skim the OGRPGLayer::HEXToGeometry() method in ogrpglayer.cpp as a guide.

7) "Do we have to encode the data fetched in PNG/JPEG/TIFF format?"

I do not believe you need to have any awareness of PNG/JPEg/TIFF
or other raster formats unless possibly you do the out-of-db case
in which case GDAL should be used to read the external images
anyways, rather than you needing to know anything about the formats.

8) "May we assume that the tiles stored at the same table are homogeneus? What 
does exactly 'homogeneus' mean here? Only same pixel type?"

In a regular_blocking table you can assume that all types are of the
same pixel type, the same tile size (in pixels and lines), and
the same geographic resolution.

9) "How do we manage the data fetched? Simply copy the raster data
sequentially into the destination? Anf what happens if these raster data
come from tiles with different pixeltype or band configuration?"

When you establish the GDALRasterBand corresponding to a band in
the table you will declare the pixel data type based on the
contents of the RASTER_COLUMNS table.   The IReadBlock() method
will be invoked with a buffer that is the block/tile size and
already of the declared pixel data type.  The tile you get
from the database should also be in this type (assuming
regular_blocking).  There should not be any translation of
pixel type required though you will need to be byte order
aware for non-byte image data.  You will basically decode
the HEXWKB image into the buffer passed into IReadBlock.

With regard to your schedule, you can consider Objective 1 to
be "prototype read only support for regular blocking rasters".
You can cut a few corners such as support of different
pixel data types or worrying about byte swapping as long as
you have an Objective 1.1 following a while later that includes
all this done well.

The schedule does not explicitly address support for the
"MODE=FIRST_ROW" mode.  I would suggest you try and
address that after Objective 3 and before Objectives
4-6.

I can definitely see that your grasp of the issues and
approach is crystalizing!

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent