[gdal-dev] Bigtiff question

[Frank Warmerdam]

Lucena, Ivan wrote:
> Hi there,
> 
> I am running a Python script that goes through a relatively large number of
> single band raster files (320) and aggregates it in a big-geotiff (around
> 7Gb) and I am facing three basic problems *poor performance*, *wrong
> results*" and *lost of metadata*.
> 
> I need that geotiff to be interleaved by pixel since the goal is to query
> for historical values in a given location and return then in the best
> response time possible.
> 
> So I am using this create options:
...
> I also need to add custom metadata entries to the dataset and to the
> individuals bands so that we know their data stamps.
> 
> That is how I loop to all the files and copy then to the output big-geotiff
> file:
>
 > --
 >     gdal.SetCacheMax(500)
 >     count = 1
 >     for i in list_tsf:
 >         input_dst = gdal.Open( f )
 >         input_band  = input_dst.GetRasterBand( 1 )
 >         output_band = output_dst.GetRasterBand( count )
 >         output_band.WriteRaster( 0, 0, x_size, y_size, 
input_band.ReadRaster( 0, 0, x_size, y_size ) )
 >         output_band.FlushCache()
 >         count = count + 1
 > --
> 
> *Performance*: The process takes more than 6 hours. 

Ivan,

I'm not surprised that the performance sucks.  Doing
random reads and writes to a pixel interleaved file is
triggering a great deal of extra data movement.

 > (Wrong Results*: All the  pixels are zero (0)
> 
> If I just change the script on 'INTERLEAVE=PIXEL' to 'INTERLEAVE=BAND' the
> whole process takes around 30 minutes.

OK, it sounds like the pixels all being zero is a bug, and
it would be good to file a ticket demonstrating this problem.
Hopefully a somewhat minimalist example of the problem.

> I can then use gdal_translate to convert from GTIFF to GTIFF with -co
> 'INTERLEAVE=PIXEL'. That takes another 30 minutes.
> 
> *Lost Metadata*: That looks like a solution except that I will then loose
> all my custom metadata entries.

Are your custom metadata values in a non-default
domain?  If so they will not be copied by default,
and you would need to add something in the script
to copy them after gdal_translate (or driver.CreateCopy()).

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent