[gdal-dev] Creating modified copies of a file

Thu Jul 7 18:07:12 EDT 2011

Le jeudi 07 juillet 2011 23:46:11, Cole, Derek a écrit :
> So that this may help anyone in the future - it seems like I may have
> solved the problem for now! After looking through gdal_translate, it
> appears that they are using the C API instead of C++ for the CreateCopy. I
> switched my code to the C API instead of C++ just for consistency sake,
> and sure enough, my CreateCopy was sped up tremendously!

Really ??? That doesn't make any sense. The C function GDALCreateCopy() is 
just a one line wrapper over the C++ API. I'm 200% sure there must be another 
difference in the values of the arguments between your C and C++ call. You 
should recheck...

> 
> For the smaller test file I was playing with, the output file was not
> viewable in my viewer until I made some adjustments, primarily, setting
> the block size of the copy to be the same as the original. if you dont set
> it explicitely you get row-wise blocks.

Yes, row-wise blocks are the default. Tiling must be explicitely set up with 
teh BLOCKXSIZE and BLOCKYSIZE creation options.

> 
> One other quirk - I was not able to use the GDALSetCacheMax() function ,

What do you mean by "not being able to use the GDALSetCacheMax()" ? Are you 
sure you specified the value in bytes as specified in the doc. I suspect you 
must have set a value in megabytes, because when the GDAL_CACHEMAX 
configuration option is read, there's a magic to automatically multiply small 
values (< 100000) by one million. That could explain why you manage to achieve 
the desired effect by using CPLSetConfigOption().

> and instead switched that to the CPLSetConfigOption() function, AND I had
> to move it to be before I call GDALAllRegister() it appears, or I was
> still getting cache thrashing warnings.
> 
> If I get some time, or if someone else does, it might be nice to narrow
> down what exactly in the C++ API was causing the problem.
> 
> Derek
> ________________________________________
> From: gdal-dev-bounces at lists.osgeo.org [gdal-dev-bounces at lists.osgeo.org]
> on behalf of Cole, Derek [dcole at integrity-apps.com] Sent: Thursday, July
> 07, 2011 3:20 PM
> To: gdal-dev at lists.osgeo.org
> Subject: RE: [gdal-dev] Creating modified copies of a file
> 
> I actually just out of curiosity tried to open the file given by this
> command:
> 
> gdal_translate dan_data.ntf dan_data_translate.ntf
> 
> And it turns out that dan_data_translate.ntf was actually a TIFF file that
> I could view. I had to explicitly add the option -of NITF
> 
> in which case it still did the translate very fast, but I did get warnings:
> 
> $ gdal_translate dan_data.ntf dan_data_translate.ntf -of NITF
> Input file size is 8689, 7679
> Warning 6: NITF only supports WGS84 geographic and UTM projections.
> 
> 0...10...20...30...40...50...60...70...80...90...100 - done.
> ERROR 6: Apparently no space reserved for IGEOLO info in NITF file.
> NITFWriteIGEOGLO() fails.
> ERROR 6: NITF only supports WGS84 geographic and UTM projections.
> 
> 
> Which jives with what my code was giving me, and further,
> dan_data_translate.ntf is not able to be opened using my viewer (which
> uses GDAL to open the files I have been working with, of course, and does
> open dan_data.ntf) because of a segmentation fault  in the RasterIO
> method. I am able to open the gdal_translate file, and my CreateCopy()
> file in another ELT, such as ENVI.
> 
> I will also make note that the original file is exactly 72.0 MB, and the
> output file is 63.6MB, but both files say in gdalinfo they are not using
> compression. This is the same for my CreateCopy() or gdal_translate. So at
> least I have narrowed down that I am apparently getting the same
> end-result as gdal_translate, albeit way slower. Finally, the output file
> seems to have created row-blocks, instead of 1024x1024 blocks like the
> original.
> 
> I will continue working to replicate gdal_translate exactly, even though
> now i am not optimistic about getting a good result.
> 
> ________________________________________
> From: gdal-dev-bounces at lists.osgeo.org [gdal-dev-bounces at lists.osgeo.org]
> on behalf of Cole, Derek [dcole at integrity-apps.com] Sent: Thursday, July
> 07, 2011 2:48 PM
> To: gdal-dev at lists.osgeo.org
> Subject: RE: [gdal-dev] Creating modified copies of a file
> 
> Yes, the source data is a NITF. Here is the first few lines of the
> projection info  in that source NITF: Coordinate System is:
> PROJCS["unnamed",
>     GEOGCS["WGS 84",
>         DATUM["WGS_1984",
>             SPHEROID["WGS 84",6378137,298.257223563,
>                 AUTHORITY["EPSG","7030"]],
>             TOWGS84[0,0,0,0,0,0,0],
>             AUTHORITY["EPSG","6326"]],
>         PRIMEM["Greenwich",0,
>             AUTHORITY["EPSG","8901"]],
>         UNIT["degree",0.0174532925199433,
>             AUTHORITY["EPSG","9108"]],
>         AUTHORITY["EPSG","4326"]],
>     PROJECTION["Transverse_Mercator"],
>     PARAMETER["latitude_of_origin",0],
>     PARAMETER["central_meridian",-75],
>     PARAMETER["scale_factor",0.996],
>     PARAMETER["false_easting",500000],
>     PARAMETER["false_northing",0]]
> 
> 
> I suppose I can go back through my code and change it to be exactly like
> gdal_translate, which seems to use the C bindings instead of C++.  I did
> end up changing my progress reporting function to the same one that is in
> gdal_translate as well, which had no change. I will try the debugging
> route before too much trouble, to see what I can find out.
> 
> Thanks for the help so far. I will report back what I find.
> 
> 
> 
> ________________________________________
> From: Even Rouault [even.rouault at mines-paris.org]
> Sent: Thursday, July 07, 2011 2:13 PM
> To: gdal-dev at lists.osgeo.org
> Cc: Cole, Derek
> Subject: Re: [gdal-dev] Creating modified copies of a file
> 
> Le jeudi 07 juillet 2011 19:52:34, Cole, Derek a écrit :
> > Well, I am about to give up on this I think and go a different route for
> > creating a NITF from this data.
> > 
> > I tried a small test file, and it takes about 45s to CopyCrate() an 80mb
> > NITF.
> 
> Yes, that's very poor performance !
> 
> > here is the output I got after turning on CPL_DEBUG
> > 
> > GDAL: GDALOpen(/home/dcole/eraser/Images/dan_data.ntf, this=0x1ad7ec70)
> > succeeds as NITF. GDAL: QuietDelete(test1.ntf) invoking Delete()
> > GDAL: GDALOpen(test1.ntf, this=0x1ad81180) succeeds as NITF.
> > GDAL: GDALDefaultOverviews::OverviewScan()
> > GDAL: GDALClose(test1.ntf, this=0x1ad81180)
> > Warning 6: NITF only supports WGS84 geographic and UTM projections.
> > 
> > 0GDAL: Potential thrashing on band 1 of
> > /home/dcole/eraser/Images/dan_data.ntf.
> > ...10...20...30...40...50...60...70...80...90...100 - done.
> > ERROR 6: Apparently no space reserved for IGEOLO info in NITF file.
> > NITFWriteIGEOGLO() fails.
> > ERROR 6: NITF only supports WGS84 geographic and UTM projections.
> > 
> > GDAL: GDALDefaultOverviews::OverviewScan()
> > ERROR 6: NITF only supports WGS84 geographic and UTM projections.
> 
> This error seems to indicate that your source dataset has an incompatible
> projection with what NITF allows (UTM/WGS84 and LongLat/WGS84). But this is
> weird because your source dataset seems to be a NITF itself... What does
> gdalinfo returns as projection for /home/dcole/eraser/Images/dan_data.ntf ?
> 
> But I don't believe it relates with performance. And I suppose you get the
> same warnings when using gdal_translate.
> 
> > So to try to remedy this I changed the cachemax (programmatically)
> > anywhere from 40MB to 3GB trying to see if any setting had any effect.
> > It was about the same no matter what.
> 
> For a 80MB file, playing with cachemax won't give any significant speed-up.
> 
> > I tried doing a gdal_translate on this file, which only took about a
> > second.
> 
> That's much more reasonable. Well, then that shows it's not an issue in
> GDAL per se, but how you use it. Now, you "just" have to discover what
> things you do which are different from gdal_translate.
> 
> > Looking through the source for that, it seems like I am doing
> > approximately the same thing in relation to the CreateCopy(), so I am not
> > sure what the problem is. However, my viewer which uses GDAL to read the
> > original NITF file I am reading in will NOT read the translated NITF file
> > that is generated.
> 
> I've just reviewed the code snippet you pasted yesterday and I see that you
> use a custom progress function. Isn't there a sleep() or some slow
> operation in it ?
> 
> For debugging performance issues, I start by just running the code under a
> debugger and to break and resume execution regularly. In 90% of the cases,
> you will see easily the bottleneck because the same function comes again
> and again. Otherwise you might need more advanced tools. sysprof or
> kcachegrind under Linux might be usefull to get timing profiles.
> 
> > Any idea why the gdal_translate might be running faster than my
> > CreateCopy, even though the parameters are just about identical?
> > 
> > Derek
> > ________________________________________
> > From: Even Rouault [even.rouault at mines-paris.org]
> > Sent: Wednesday, July 06, 2011 5:32 PM
> > To: gdal-dev at lists.osgeo.org
> > Cc: Cole, Derek
> > Subject: Re: [gdal-dev] Creating modified copies of a file
> > 
> > Le mercredi 06 juillet 2011 23:18:21, Cole, Derek a écrit :
> > > I have noticed that when attempting to do this, I am able to do a
> > > gdalinfo on the source and destination files (even if the destination
> > > file is not complete, it seems the header gets created, and gdalinfo
> > > recognizes teh file).
> > > 
> > > Is it possible that this is taking so long because my destination file
> > > has no geocoords or projection information? It seems like createcopy is
> > > not replicating all of the variables in the original and I am having to
> > > manually set some things, such as Block size, ABPP, etc etc.
> > 
> > This is expected. When doing CreateCopy() the target driver has no idea
> > what the source driver is, so he will not recognize ABPP. As far as block
> > sizes are concerned, it is up to the user to decide if he wants to keep
> > or change the block dimension of the source dataset.
> > 
> > I'm a bit surprised that CreateCopy() is *that* much slower than cp, but
> > it is hard to know why without a hard work of profiling. It might be
> > related to pixel or band interleaving, etc... There might cache
> > trashing. You could perhaps run with CPL_DEBUG=ON and see if interesting
> > info comes up.
> > 
> > > And the
> > > coordinates and projection information is blank in the new file. Is
> > > there perhaps some calculations being done to correct that issue, and
> > > thats why the copy is taking forever?
> > 
> > No, not related at all.
> > 
> > > I thought based on the documentation
> > > CreateCopy() would clone the file's raster, as well as size, type,
> > > projection, geotransform, etc, but all of this is not showing up in the
> > > partially created file at least.
> > 
> > You cannot reliably trust gdalinfo on a file still being written.
> > Depending on the driver some information might just be written at the
> > end of the process. A quick look at the CreateCopy() implementation of
> > the NITF driver suggests that this is indeed the case since the
> > geotransform is set after the imagery.
> > _______________________________________________ gdal-dev mailing list
> > gdal-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> 
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev