[gdal-dev] Creating modified copies of a file

[Even Rouault]

Le jeudi 07 juillet 2011 19:52:34, Cole, Derek a écrit :
> Well, I am about to give up on this I think and go a different route for
> creating a NITF from this data.
> 
> I tried a small test file, and it takes about 45s to CopyCrate() an 80mb
> NITF.

Yes, that's very poor performance !

> 
> here is the output I got after turning on CPL_DEBUG
> 
> GDAL: GDALOpen(/home/dcole/eraser/Images/dan_data.ntf, this=0x1ad7ec70)
> succeeds as NITF. GDAL: QuietDelete(test1.ntf) invoking Delete()
> GDAL: GDALOpen(test1.ntf, this=0x1ad81180) succeeds as NITF.
> GDAL: GDALDefaultOverviews::OverviewScan()
> GDAL: GDALClose(test1.ntf, this=0x1ad81180)
> Warning 6: NITF only supports WGS84 geographic and UTM projections.
> 
> 0GDAL: Potential thrashing on band 1 of
> /home/dcole/eraser/Images/dan_data.ntf.
> ...10...20...30...40...50...60...70...80...90...100 - done.
> ERROR 6: Apparently no space reserved for IGEOLO info in NITF file.
> NITFWriteIGEOGLO() fails.
> ERROR 6: NITF only supports WGS84 geographic and UTM projections.
> 
> GDAL: GDALDefaultOverviews::OverviewScan()
> ERROR 6: NITF only supports WGS84 geographic and UTM projections.
> 

This error seems to indicate that your source dataset has an incompatible 
projection with what NITF allows (UTM/WGS84 and LongLat/WGS84). But this is 
weird because your source dataset seems to be a NITF itself... What does 
gdalinfo returns as projection for /home/dcole/eraser/Images/dan_data.ntf ?

But I don't believe it relates with performance. And I suppose you get the 
same warnings when using gdal_translate.

> 
> So to try to remedy this I changed the cachemax (programmatically) anywhere
> from 40MB to 3GB trying to see if any setting had any effect. It was about
> the same no matter what.

For a 80MB file, playing with cachemax won't give any significant speed-up.

> 
> I tried doing a gdal_translate on this file, which only took about a
> second.

That's much more reasonable. Well, then that shows it's not an issue in GDAL 
per se, but how you use it. Now, you "just" have to discover what things you 
do which are different from gdal_translate.

> Looking through the source for that, it seems like I am doing
> approximately the same thing in relation to the CreateCopy(), so I am not
> sure what the problem is. However, my viewer which uses GDAL to read the
> original NITF file I am reading in will NOT read the translated NITF file
> that is generated.

I've just reviewed the code snippet you pasted yesterday and I see that you 
use a custom progress function. Isn't there a sleep() or some slow operation 
in it ?

For debugging performance issues, I start by just running the code under a 
debugger and to break and resume execution regularly. In 90% of the cases, you 
will see easily the bottleneck because the same function comes again and 
again. Otherwise you might need more advanced tools. sysprof or kcachegrind 
under Linux might be usefull to get timing profiles.

> 
> Any idea why the gdal_translate might be running faster than my CreateCopy,
> even though the parameters are just about identical?
> 
> Derek
> ________________________________________
> From: Even Rouault [even.rouault at mines-paris.org]
> Sent: Wednesday, July 06, 2011 5:32 PM
> To: gdal-dev at lists.osgeo.org
> Cc: Cole, Derek
> Subject: Re: [gdal-dev] Creating modified copies of a file
> 
> Le mercredi 06 juillet 2011 23:18:21, Cole, Derek a écrit :
> > I have noticed that when attempting to do this, I am able to do a
> > gdalinfo on the source and destination files (even if the destination
> > file is not complete, it seems the header gets created, and gdalinfo
> > recognizes teh file).
> > 
> > Is it possible that this is taking so long because my destination file
> > has no geocoords or projection information? It seems like createcopy is
> > not replicating all of the variables in the original and I am having to
> > manually set some things, such as Block size, ABPP, etc etc.
> 
> This is expected. When doing CreateCopy() the target driver has no idea
> what the source driver is, so he will not recognize ABPP. As far as block
> sizes are concerned, it is up to the user to decide if he wants to keep or
> change the block dimension of the source dataset.
> 
> I'm a bit surprised that CreateCopy() is *that* much slower than cp, but it
> is hard to know why without a hard work of profiling. It might be related
> to pixel or band interleaving, etc... There might cache trashing. You
> could perhaps run with CPL_DEBUG=ON and see if interesting info comes up.
> 
> > And the
> > coordinates and projection information is blank in the new file. Is there
> > perhaps some calculations being done to correct that issue, and thats why
> > the copy is taking forever?
> 
> No, not related at all.
> 
> > I thought based on the documentation
> > CreateCopy() would clone the file's raster, as well as size, type,
> > projection, geotransform, etc, but all of this is not showing up in the
> > partially created file at least.
> 
> You cannot reliably trust gdalinfo on a file still being written. Depending
> on the driver some information might just be written at the end of the
> process. A quick look at the CreateCopy() implementation of the NITF
> driver suggests that this is indeed the case since the geotransform is set
> after the imagery. _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev