[postgis-devel] [WKT Raster] Regular blocking in gdal2wktraster.py

[Pierre Racine]

>> This is the traditional GeoRaster way of storing images (e.g. landsat)
>> in a database.
>>
>> Types 1 and 2 correspond better to the notion of a coverage (much like
>> a vector coverage):
>>
>> -in type 1 the global extent is not necessarily rectangular (although
>> we could have an option to "force" it to be rectangular by adding
>> nodatavalue or empty tiles but this is from my point of view an
>> overcomplicated artefact)
>
>IMHO, shape of extent we can handle is important but it doesn't have
>highest score. More important (usable) restrictions seem to be:
>
>1. All tiles are of the same determined by size of "natural" block.
>
>2. Tiles fit common grid grid defined by pixel size and
>bounding box of global extent (not necessarily rectangular).
>
>3. Tiles do not overlap.
>
>Constraint on shape of extent seems to be optional.
>This is how I understand regular blocking should work.

You mean that as far as those three conditions are met, we can set regular_blocking to true, even if there are some missing tiles? This is what I understand from Frank's comment about sparse rasters in the specs.

>> -for both types, the source is composed of many raster files that most
>> be retiled and stored into the same db table
>
>Agreed. At the moment, gdal2wktraster supports single raster file source
>in regular blocking. Although, I can't see any reason to not to support
>multiple input files, tile them all according to a common
>block (tile) size. There is just one factor that makes this operation
>more complex, for regular blocking, all tiles being crunched must fit
>belong to common grid.

This is what I need to easily import most of the datasets I want to import.

>> -another characteristic (which I did not mention) is that tiles might
>> not be all of the same size. I could, for example, have many 6030pixels
>> by 6030pixels images as source and want to store them as as many (3600
>> tiles of 100pixels x 100pixels and the rest as 60 30pixels x 100pixels
>> tiles,  60 100x30 tiles and one 30x30 tile).
>
>OK, but then it's not regular blocking.

Right. In this case, regular blocking is broken. The loader should detect it and set regular_blocking to false.

So if the user if bright enough (and read our advices in the future "best practice" part of the documentation) he will tile all the input rasters in a way that do not brake regular blocking. e.g. importing 10 6000x6000 images as 36000 100x100 tiles. And make sure there is no overlapping (this is the case of most datasets).

If he is not (or if, really, he can't do it for whatever reason) then regular blocking is broken and all the application optimization that comes with it. But WKT Raster work seemlessly since regular blocking is not a requirement. Are we on the same line?

Question: What does your application do when regular_blocking is set to false?

In the same line, if he loads many overlapping rasters to the same table using the same multi-input command, then regular blocking is also set to false. Is this easy to detect in gdal2wktraster?

>> I want to be able to import case 1 and 2 in the
>> database without having to reconstruct a big raster using other tools.
>
>You can do that.
>Now, gdal2wktraster can do it for single file input.
>It shouldn't be a problem to implement support for multiple input
>raster datasets, if you need it.

Great! This is what I need! I'll do it if it's not in your priorities.

>> If an application threats each rows of a raster table as a single
>> raster
>> (as I have always try to sell WKT Raster "a tile is a raster and a
>> raster is a tile, there is no distinction") then the application should
>> not see any difference. As for a vector layer.
>
>A raster might be a little ambiguous term, sometimes.
>A raster is a single pixel (a sample) or it is a complete grid of pixels.
>Let's assume a raster is a grid of pixels.
>Since the beginning of my WKT raster adventure, I preferred to say:
>1 table = 1 raster (raster coverage, raster overlay, just raster).

In my mind:

1 raster = 1 tile = 1 row

and 

1 coverage = 1 table

1 "big raster" = 1 table. It is similar to a small coverage but very (too much) constrained (to regular blocking without missing tiles). Let's put it like this: "big raster" is the quintessence of coverage perfection :-) But we can not add tiles (from a second file for example) to it without braking its perfection. If we can (append tiles from many source images) and that we support missing tiles then type 4 is identical to type 2 and the "big raster" type disappears.

many "big raster"s would have to be stored as separate tables (but this would not be true anymore if we support loading of many images at the same time).

>> I understand however that
>> applications prefers to load big rectangular complete tiled rasters but
>> this is very limitative and not very practical from an analytical point
>> of view.
>
>AFAIU, regular blocking is supposed to help to operate on a part of
>big raster (a coverage), on a window of big raster.

In my ming it is more the tiling that helps to operate on a part of big raster than regular blocking. Tiles can be of any size (regular blocking is optional/optimal). This is problably not true in a file like a TIFF but it is certainly in a database with Gist indexing where tiles are selected through a geometric SQL query e.g. "SELECT rast FROM table WHERE rast && geom" (not through address math). An other way to say it would be that a database index is much more flexible than an address based file index, this is why it allows variable tiles size. This is why regular blocking is not as important for me as tiling. WKT Raster does not care if tiles are not all the same size. Agreed?

>> I want to be able to import it's many files, retiled to
>> smallers tiles in a single table in a single loading command. With this
>> I would be very happy (and I guess many of our users) and I could begin
>> to do nice vector/raster analysis :-)
>
>So, three questions:
>1. Do all these files have the same size of pixel?
>2. Can you tile them according to the same size of block (same size of tile)?
>3. Are the tiles non-overlapping?
>
>AFAIK, SRTM answers yes to all of these questions.
>If this it's true, then we do support this use case!

The only missing feature is to be able to import many files while retiling them. We are not very far.

Question: I did not try the -k option yet. It seems to load tiles as they are already defined in the file, right? I can't specify a new block size (this is what I call retiling)? e.g. -k 100x100

Happy that we seems to finally understand each other.

Pierre