[gdal-dev] Re: Problems with large raster sizes (WMS/TMS)

[Tamas Szekeres]

2009/11/13 Frank Warmerdam <warmerdam at pobox.com>:
> Tamas Szekeres wrote:
>>
>> Hi Even,
>>
>> Yes, I thought it was a bit complicated issue. I would also support
>> replacing the array of pointers with a hashtable to eliminate the
>> unnecessary memory requirement. I've been looking at the code and
>> found a hashset implementation in CPL which could probably be utilized
>> here, however this one doesn't support to auto-grow and it should also
>> store all the items with the same hash value (along with the key) in a
>> linked list instead replacing each other. I don't think it would
>> significantly slow down the lookup process since only the list of the
>> same hash should be iterated.
>
> Tamas / Even,
>
> I would like any transition in this area to include some performance
> analysis with large and modest sized images.
>

Frank,

I guess mostly the tile-based raster data sources would suffer from
this memory issue. In this regard it would only be reasonable to have
those alternative implementations added to those drivers only. We
should probably consider which functions have to be made 'virtual' to
be able to modify the generic block cache implementation by the
driver.

>> The problem of the nRasterXSize limitation is more significant IMO,
>> however I think it would be enough to change the type from integer to
>> float or double and some type casts should be added in the related
>> expressions. But it would indeed be a significant change in gdal,
>> however I think we should start thinking about it in order to support
>> the (virtually) large raster dimensions.
>
> I would prefer to limit ourselves to 2 gigapixels by 2 gigapixels images
> for the time being.  I just can't yet see that larger images are of
> sufficient interest to justify the complexity involved in properly
> supporting larger images throughout GDAL.
>
> If we were to do so, I'd want it handled via an RFC.
>

This is true probably for most of the existing drivers. However for
the server based raster data sources can have no such limit in the
size when the supported level of the detail is increasing, may be
earlier than we can expect.

Best regards,

Tamas