[gdal-dev] Bigtiff question

[Lucena, Ivan]

Hi there,

I am running a Python script that goes through a relatively large number of single band raster files (320) and aggregates it in a big-geotiff (around 7Gb) and I am facing three basic problems *poor performance*, 
*wrong results*" and *lost of metadata*.

I need that geotiff to be interleaved by pixel since the goal is to query for historical values in a given location and return then in the best response time possible. 

So I am using this create options:

--
    driver_tif = gdal.GetDriverByName("GTIFF")

    output_dst = driver_tif.Create( output_tif, x_size, y_size, serie_count, data_type,
        [ 'TILED=NO', 'INTERLEAVE=PIXEL' ])
--

I also need to add custom metadata entries to the dataset and to the individuals bands so that we know their data stamps.

That is how I loop to all the files and copy then to the output big-geotiff file:

--
    gdal.SetCacheMax(500) 
    count = 1
    for i in list_tsf:
        input_dst = gdal.Open( f )
        input_band  = input_dst.GetRasterBand( 1 )
        output_band = output_dst.GetRasterBand( count )
        output_band.WriteRaster( 0, 0, x_size, y_size, input_band.ReadRaster( 0, 0, x_size, y_size ) )
        output_band.FlushCache()
        count = count + 1
--

*Performance*: The process takes more than 6 hours.
(Wrong Results*: All the pixels are zero (0)

If I just change the script on 'INTERLEAVE=PIXEL' to 'INTERLEAVE=BAND' the whole process takes around 30 minutes.

I can then use gdal_translate to convert from GTIFF to GTIFF with -co 'INTERLEAVE=PIXEL'. That takes another 30 minutes.

*Lost Metadata*: That looks like a solution except that I will then loose all my custom metadata entries.

Does anybody have any idea to improve that process.

Thanks is advance,

Ivan