[postgis-devel] [WKT Raster] Regular blocking in gdal2wktraster.py

[Mateusz Loskot]

Pierre Racine wrote:
> >> This is the traditional GeoRaster way of storing images (e.g.
> >> landsat) in a database.
> >>
> >> Types 1 and 2 correspond better to the notion of a coverage (much
> >> like a vector coverage):
> >>
> >> -in type 1 the global extent is not necessarily rectangular
> >> (although
> >> we could have an option to "force" it to be rectangular by adding
> >> nodatavalue or empty tiles but this is from my point of view an
> >> overcomplicated artefact)
> >
> >IMHO, shape of extent we can handle is important but it doesn't have
> >highest score. More important (usable) restrictions seem to be:
> >
> >1. All tiles are of the same determined by size of "natural" block.
> >
> >2. Tiles fit common grid grid defined by pixel size and bounding box
> >of global extent (not necessarily rectangular).
> >
> >3. Tiles do not overlap.
> >
> >Constraint on shape of extent seems to be optional.
> >This is how I understand regular blocking should work.
> 
> You mean that as far as those three conditions are met, we can set
> regular_blocking to true, even if there are some missing tiles?

Yes.
 
> This is what I understand from Frank's comment about sparse
> rasters in the specs.

This is how we understand regular blocking too.


> >> -for both types, the source is composed of many raster files that
> >> most be retiled and stored into the same db table
> >
> >Agreed. At the moment, gdal2wktraster supports single raster file
> >source in regular blocking. Although, I can't see any reason to not to
> >support multiple input files, tile them all according to a common
> >block (tile) size. There is just one factor that makes this operation
> >more complex, for regular blocking, all tiles being crunched must fit
> >belong to common grid.
> 
> This is what I need to easily import most of the datasets I want to
> import.

Understood and agreed.
However, as I don't need it, I ended with single file input for now
which is what I need. So, feel free to add support of multiple
file input to gdal2wktraster.

> >> -another characteristic (which I did not mention) is that tiles
> >> might
> >> not be all of the same size. I could, for example, have many
> >> 6030pixels by 6030pixels images as source and want to store them as
> >> as many (3600 tiles of 100pixels x 100pixels and the rest as 60
> >> 30pixels x 100pixels tiles,  60 100x30 tiles and one 30x30 tile).
> >
> >OK, but then it's not regular blocking.
> 
> Right. In this case, regular blocking is broken. The loader should
> detect it and set regular_blocking to false.

Yes.

> So if the user if bright enough (and read our advices in the future
> "best practice" part of the documentation) he will tile all the input
> rasters in a way that do not brake regular blocking. e.g. importing 10
> 6000x6000 images as 36000 100x100 tiles. And make sure there is no
> overlapping (this is the case of most datasets).

Yes.

And if all raster datasets have common attributes, so they
are OK according to constraints 1, 2 and 3 without any additional
processing, then gdal2wktraster can do the whole job.
But if input files have, for instance, different size of pixel then
gdal2wktraster will likely won't be able to generate proper regular
blocking coverage. So, user will need to use some other tools to 
process and "fix" his data.

> If he is not (or if, really, he can't do it for whatever reason) then
> regular blocking is broken and all the application optimization that
> comes with it. But WKT Raster work seemlessly since regular blocking is
> not a requirement. Are we on the same line?

Yes, we are.

> Question: What does your application do when regular_blocking is set to
> false?

At this very early moment, it does nothing. 

> In the same line, if he loads many overlapping rasters to the same
> table using the same multi-input command, then regular blocking is also
> set to false. Is this easy to detect in gdal2wktraster?

I'm not sure if it's easy. I have not thought about subject
as in early stage of testing, I test using correct regular blocking.

> >> I want to be able to import case 1 and 2 in the database without
> >> having to reconstruct a big raster using other tools.
> >
> >You can do that.
> >Now, gdal2wktraster can do it for single file input.
> >It shouldn't be a problem to implement support for multiple input
> >raster datasets, if you need it.
> 
> Great! This is what I need! I'll do it if it's not in your priorities.

That's what I've mentioned, it's not my priority,
so if you want to do it, great!

> >> If an application threats each rows of a raster table as a single
> >> raster (as I have always try to sell WKT Raster "a tile is a raster
> >> and a raster is a tile, there is no distinction") then the
> >> application should not see any difference. As for a vector layer.
> >
> >A raster might be a little ambiguous term, sometimes.
> >A raster is a single pixel (a sample) or it is a complete grid of
> >pixels.
> >Let's assume a raster is a grid of pixels.
> >Since the beginning of my WKT raster adventure, I preferred to say:
> >1 table = 1 raster (raster coverage, raster overlay, just raster).
> 
> In my mind:
> 
> 1 raster = 1 tile = 1 row
> 
> and
> 
> 1 coverage = 1 table
> 
> 1 "big raster" = 1 table. It is similar to a small coverage but very
> (too much) constrained (to regular blocking without missing tiles).
> Let's put it like this: "big raster" is the quintessence of coverage
> perfection :-) But we can not add tiles (from a second file for
> example) to it without braking its perfection. If we can (append tiles
> from many source images) and that we support missing tiles then type 4
> is identical to type 2 and the "big raster" type disappears.
> 
> many "big raster"s would have to be stored as separate tables (but this
> would not be true anymore if we support loading of many images at the
> same time).

I have to think more about it and the terminology we use.

> >> I understand however that
> >> applications prefers to load big rectangular complete tiled rasters
> >> but this is very limitative and not very practical from an
> >> analytical point of view.
> >
> >AFAIU, regular blocking is supposed to help to operate on a part of
> >big raster (a coverage), on a window of big raster.
> 
> In my ming it is more the tiling that helps to operate on a part of big
> raster than regular blocking. Tiles can be of any size (regular
> blocking is optional/optimal). This is problably not true in a file
> like a TIFF but it is certainly in a database with Gist indexing where
> tiles are selected through a geometric SQL query e.g. "SELECT rast FROM
> table WHERE rast && geom" (not through address math). An other way to
> say it would be that a database index is much more flexible than an
> address based file index, this is why it allows variable tiles size.
> This is why regular blocking is not as important for me as tiling. WKT
> Raster does not care if tiles are not all the same size. Agreed?

One correction, WKT Raster can work in two modes:
1. By default, it does not care if tiles are of the same size or not.
2. If regular blocking enabled, WKT Raster does care.

Agreed?

> >> I want to be able to import it's many files, retiled to smallers
> >> tiles in a single table in a single loading command. With this I
> >> would be very happy (and I guess many of our users) and I could
> >> begin to do nice vector/raster analysis :-)
> >
> >So, three questions:
> >1. Do all these files have the same size of pixel?
> >2. Can you tile them according to the same size of block (same size of
> >tile)?
> >3. Are the tiles non-overlapping?
> >
> >AFAIK, SRTM answers yes to all of these questions.
> >If this it's true, then we do support this use case!
> 
> The only missing feature is to be able to import many files while
> retiling them. We are not very far.

Bingo!

> Question: I did not try the -k option yet. It seems to load tiles as
> they are already defined in the file, right? I can't specify a new
> block size (this is what I call retiling)? e.g. -k 100x100

Right, there is no way to specify size of tile.

Simply, I'm a baby of Unix where I use number of scattered but very
specialized tools, instead of a few monolithic big applications.
Also, I don't like to reinvent the wheel :-)
So, if I need to tile a dataset according custom size, I just
use GDAL utilities. Then I use gdal2wktraster to load these tiles.
That's why I can't see any point to implement custom-size-tiling
in gdal2wktraster.

Nevertheless, size option can be added in 10 minutes :-) - add extra
if-else clause in the wkblify_raster function:
If regular_blocking is requested AND user specified custom tile size,
don't call calculate_block_size() but use the given size of tile.

> Happy that we seems to finally understand each other.

I'm happy too :-)

Best regards,
--
Mateusz Loskot
Senior Programmer, Cadcorp
http://www.cadcorp.com