Re: [gdal-dev] Bigtiff question

[Lucena, Ivan]

Frank,

>  > I am running a Python script that goes through a relatively large number of
>  > single band raster files (320) and aggregates it in a big-geotiff (around
>  > 7Gb) and I am facing three basic problems *poor performance*, *wrong
>  > results*" and *lost of metadata*.
>  >
>  Ivan,
>  
>  I'm not surprised that the performance sucks.  Doing
>  random reads and writes to a pixel interleaved file is
>  triggering a great deal of extra data movement.
>  

Yes, that runs a lot of seek's to writes just few bytes here and there. 

I am wondering what the geotiff driver could do to improve that; keeping tiles in memory until they are filled up for 
writing at once for example (?)

BTW, would make any difference if tile the geotiff? In that case what would be the blockxsize, blockysize recommended for 320 bands interleaved by PIXEL?

>  OK, it sounds like the pixels all being zero is a bug, and
>  it would be good to file a ticket demonstrating this problem.
>  Hopefully a somewhat minimalist example of the problem.

I think it would be very hard so send data samples so I would suggest running a script that creates fake raster bands 
with all pixels as 1 on band 1, 2 on band 2, etc. Something like that perhaps:

--
    driver_tif = gdal.GetDriverByName("GTIFF")
    output_dst = driver_tif.Create( output_tif, x_size, y_size, serie_count, data_type,
        [ 'TILED=NO', 'INTERLEAVE=PIXEL' ])
    for i in range(320):
        output_band = output_dst.GetRasterBand( 1 + 1 )
        output_band.Fill(i + 1)
        output_band.FlushCache()
--

>  > *Lost Metadata*: That looks like a solution except that I will then loose
>  > all my custom metadata entries.
>  
>  Are your custom metadata values in a non-default
>  domain?  If so they will not be copied by default,
>  and you would need to add something in the script
>  to copy them after gdal_translate (or driver.CreateCopy()).

That makes sense too.

Thanks.

Best regards,

Ivan