[gdal-dev] Creating modified copies of a file
Cole, Derek
dcole at integrity-apps.com
Thu Jul 7 17:46:11 EDT 2011
So that this may help anyone in the future - it seems like I may have solved the problem for now! After looking through gdal_translate, it appears that they are using the C API instead of C++ for the CreateCopy. I switched my code to the C API instead of C++ just for consistency sake, and sure enough, my CreateCopy was sped up tremendously!
For the smaller test file I was playing with, the output file was not viewable in my viewer until I made some adjustments, primarily, setting the block size of the copy to be the same as the original. if you dont set it explicitely you get row-wise blocks.
One other quirk - I was not able to use the GDALSetCacheMax() function , and instead switched that to the CPLSetConfigOption() function, AND I had to move it to be before I call GDALAllRegister() it appears, or I was still getting cache thrashing warnings.
If I get some time, or if someone else does, it might be nice to narrow down what exactly in the C++ API was causing the problem.
Derek
________________________________________
From: gdal-dev-bounces at lists.osgeo.org [gdal-dev-bounces at lists.osgeo.org] on behalf of Cole, Derek [dcole at integrity-apps.com]
Sent: Thursday, July 07, 2011 3:20 PM
To: gdal-dev at lists.osgeo.org
Subject: RE: [gdal-dev] Creating modified copies of a file
I actually just out of curiosity tried to open the file given by this command:
gdal_translate dan_data.ntf dan_data_translate.ntf
And it turns out that dan_data_translate.ntf was actually a TIFF file that I could view. I had to explicitly add the option -of NITF
in which case it still did the translate very fast, but I did get warnings:
$ gdal_translate dan_data.ntf dan_data_translate.ntf -of NITF
Input file size is 8689, 7679
Warning 6: NITF only supports WGS84 geographic and UTM projections.
0...10...20...30...40...50...60...70...80...90...100 - done.
ERROR 6: Apparently no space reserved for IGEOLO info in NITF file.
NITFWriteIGEOGLO() fails.
ERROR 6: NITF only supports WGS84 geographic and UTM projections.
Which jives with what my code was giving me, and further, dan_data_translate.ntf is not able to be opened using my viewer (which uses GDAL to open the files I have been working with, of course, and does open dan_data.ntf) because of a segmentation fault in the RasterIO method. I am able to open the gdal_translate file, and my CreateCopy() file in another ELT, such as ENVI.
I will also make note that the original file is exactly 72.0 MB, and the output file is 63.6MB, but both files say in gdalinfo they are not using compression. This is the same for my CreateCopy() or gdal_translate. So at least I have narrowed down that I am apparently getting the same end-result as gdal_translate, albeit way slower. Finally, the output file seems to have created row-blocks, instead of 1024x1024 blocks like the original.
I will continue working to replicate gdal_translate exactly, even though now i am not optimistic about getting a good result.
________________________________________
From: gdal-dev-bounces at lists.osgeo.org [gdal-dev-bounces at lists.osgeo.org] on behalf of Cole, Derek [dcole at integrity-apps.com]
Sent: Thursday, July 07, 2011 2:48 PM
To: gdal-dev at lists.osgeo.org
Subject: RE: [gdal-dev] Creating modified copies of a file
Yes, the source data is a NITF. Here is the first few lines of the projection info in that source NITF:
Coordinate System is:
PROJCS["unnamed",
GEOGCS["WGS 84",
DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.257223563,
AUTHORITY["EPSG","7030"]],
TOWGS84[0,0,0,0,0,0,0],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0,
AUTHORITY["EPSG","8901"]],
UNIT["degree",0.0174532925199433,
AUTHORITY["EPSG","9108"]],
AUTHORITY["EPSG","4326"]],
PROJECTION["Transverse_Mercator"],
PARAMETER["latitude_of_origin",0],
PARAMETER["central_meridian",-75],
PARAMETER["scale_factor",0.996],
PARAMETER["false_easting",500000],
PARAMETER["false_northing",0]]
I suppose I can go back through my code and change it to be exactly like gdal_translate, which seems to use the C bindings instead of C++. I did end up changing my progress reporting function to the same one that is in gdal_translate as well, which had no change. I will try the debugging route before too much trouble, to see what I can find out.
Thanks for the help so far. I will report back what I find.
________________________________________
From: Even Rouault [even.rouault at mines-paris.org]
Sent: Thursday, July 07, 2011 2:13 PM
To: gdal-dev at lists.osgeo.org
Cc: Cole, Derek
Subject: Re: [gdal-dev] Creating modified copies of a file
Le jeudi 07 juillet 2011 19:52:34, Cole, Derek a écrit :
> Well, I am about to give up on this I think and go a different route for
> creating a NITF from this data.
>
> I tried a small test file, and it takes about 45s to CopyCrate() an 80mb
> NITF.
Yes, that's very poor performance !
>
> here is the output I got after turning on CPL_DEBUG
>
> GDAL: GDALOpen(/home/dcole/eraser/Images/dan_data.ntf, this=0x1ad7ec70)
> succeeds as NITF. GDAL: QuietDelete(test1.ntf) invoking Delete()
> GDAL: GDALOpen(test1.ntf, this=0x1ad81180) succeeds as NITF.
> GDAL: GDALDefaultOverviews::OverviewScan()
> GDAL: GDALClose(test1.ntf, this=0x1ad81180)
> Warning 6: NITF only supports WGS84 geographic and UTM projections.
>
> 0GDAL: Potential thrashing on band 1 of
> /home/dcole/eraser/Images/dan_data.ntf.
> ...10...20...30...40...50...60...70...80...90...100 - done.
> ERROR 6: Apparently no space reserved for IGEOLO info in NITF file.
> NITFWriteIGEOGLO() fails.
> ERROR 6: NITF only supports WGS84 geographic and UTM projections.
>
> GDAL: GDALDefaultOverviews::OverviewScan()
> ERROR 6: NITF only supports WGS84 geographic and UTM projections.
>
This error seems to indicate that your source dataset has an incompatible
projection with what NITF allows (UTM/WGS84 and LongLat/WGS84). But this is
weird because your source dataset seems to be a NITF itself... What does
gdalinfo returns as projection for /home/dcole/eraser/Images/dan_data.ntf ?
But I don't believe it relates with performance. And I suppose you get the
same warnings when using gdal_translate.
>
> So to try to remedy this I changed the cachemax (programmatically) anywhere
> from 40MB to 3GB trying to see if any setting had any effect. It was about
> the same no matter what.
For a 80MB file, playing with cachemax won't give any significant speed-up.
>
> I tried doing a gdal_translate on this file, which only took about a
> second.
That's much more reasonable. Well, then that shows it's not an issue in GDAL
per se, but how you use it. Now, you "just" have to discover what things you
do which are different from gdal_translate.
> Looking through the source for that, it seems like I am doing
> approximately the same thing in relation to the CreateCopy(), so I am not
> sure what the problem is. However, my viewer which uses GDAL to read the
> original NITF file I am reading in will NOT read the translated NITF file
> that is generated.
I've just reviewed the code snippet you pasted yesterday and I see that you
use a custom progress function. Isn't there a sleep() or some slow operation
in it ?
For debugging performance issues, I start by just running the code under a
debugger and to break and resume execution regularly. In 90% of the cases, you
will see easily the bottleneck because the same function comes again and
again. Otherwise you might need more advanced tools. sysprof or kcachegrind
under Linux might be usefull to get timing profiles.
>
> Any idea why the gdal_translate might be running faster than my CreateCopy,
> even though the parameters are just about identical?
>
> Derek
> ________________________________________
> From: Even Rouault [even.rouault at mines-paris.org]
> Sent: Wednesday, July 06, 2011 5:32 PM
> To: gdal-dev at lists.osgeo.org
> Cc: Cole, Derek
> Subject: Re: [gdal-dev] Creating modified copies of a file
>
> Le mercredi 06 juillet 2011 23:18:21, Cole, Derek a écrit :
> > I have noticed that when attempting to do this, I am able to do a
> > gdalinfo on the source and destination files (even if the destination
> > file is not complete, it seems the header gets created, and gdalinfo
> > recognizes teh file).
> >
> > Is it possible that this is taking so long because my destination file
> > has no geocoords or projection information? It seems like createcopy is
> > not replicating all of the variables in the original and I am having to
> > manually set some things, such as Block size, ABPP, etc etc.
>
> This is expected. When doing CreateCopy() the target driver has no idea
> what the source driver is, so he will not recognize ABPP. As far as block
> sizes are concerned, it is up to the user to decide if he wants to keep or
> change the block dimension of the source dataset.
>
> I'm a bit surprised that CreateCopy() is *that* much slower than cp, but it
> is hard to know why without a hard work of profiling. It might be related
> to pixel or band interleaving, etc... There might cache trashing. You
> could perhaps run with CPL_DEBUG=ON and see if interesting info comes up.
>
> > And the
> > coordinates and projection information is blank in the new file. Is there
> > perhaps some calculations being done to correct that issue, and thats why
> > the copy is taking forever?
>
> No, not related at all.
>
> > I thought based on the documentation
> > CreateCopy() would clone the file's raster, as well as size, type,
> > projection, geotransform, etc, but all of this is not showing up in the
> > partially created file at least.
>
> You cannot reliably trust gdalinfo on a file still being written. Depending
> on the driver some information might just be written at the end of the
> process. A quick look at the CreateCopy() implementation of the NITF
> driver suggests that this is indeed the case since the geotransform is set
> after the imagery. _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
More information about the gdal-dev
mailing list