[gdal-dev] Creating modified copies of a file

Wed Jul 6 17:18:21 EDT 2011

I have noticed that when attempting to do this, I am able to do a gdalinfo on the source and destination files (even if the destination file is not complete, it seems the header gets created, and gdalinfo recognizes teh file).

Is it possible that this is taking so long because my destination file has no geocoords or projection information? It seems like createcopy is not replicating all of the variables in the original and I am having to manually set some things, such as Block size, ABPP, etc etc. And the coordinates and projection information is blank in the new file. Is there perhaps some calculations being done to correct that issue, and thats why the copy is taking forever? I thought based on the documentation CreateCopy() would clone the file's raster, as well as size, type, projection, geotransform, etc, but all of this is not showing up in the partially created file at least.

Derek
________________________________________
From: gdal-dev-bounces at lists.osgeo.org [gdal-dev-bounces at lists.osgeo.org] on behalf of Cole, Derek [dcole at integrity-apps.com]
Sent: Wednesday, July 06, 2011 3:04 PM
To: gdal-dev at lists.osgeo.org
Subject: RE: [gdal-dev] Creating modified copies of a file

Well, my plan originally was to create a Virtual Data set in memory, so that I could just write to the blocks while in memory before writing to disk. It seems like I may be just as well off to go ahead and create the copy, and then just overwrite the block in the copy.

I have tried to switch this back to a NITF driver instead of VRT.
 Do you see anything wrong with this:

    poDataset = (GDALDataset *) GDALOpenShared( myfile, GA_ReadOnly );

    poNITFDriver = (GDALDriver *)GDALGetDriverByName("NITF");
    GDALSetCacheMax(300);
    papszMetadata = poNITFDriver->GetMetadata();

    if( poNITFDriver == NULL )
        exit( 1 );

    char *message = "Copying Current File";
    char **papszOptions = NULL;

    papszOptions = CSLSetNameValue( papszOptions, "BLOCKXSIZE", "1024" );
    papszOptions = CSLSetNameValue( papszOptions, "BLOCKYSIZE", "1024" );
    poNITFDS = poNITFDriver->CreateCopy(pszDstFilename, poDataset, FALSE,  papszOptions, MyTextProgress, message);

As you can see, this would copy from  NITF to NITF, and use the same block size as my original NITF (1024^2), as well as setting the cache to 300 MB. This was something I read in an old thread you had posted to. I dont see any noticable improvement. Is 300MB enough? How is that calculated exactly?  According to that thread, it was image width * block height * 2*sizeof(unit16)

here is from the gdal info of my source file:

Size is 37504, 109324
Band 1 Block=1024x1024 Type=UInt16, ColorInterp=Gray

so it should be ~290MB right? I think uint16 is 4 bytes.

This is still taking hours to copy.

Thanks

Derek
________________________________________
From: Even Rouault [even.rouault at mines-paris.org]
Sent: Wednesday, July 06, 2011 1:20 PM
To: gdal-dev at lists.osgeo.org
Cc: Cole, Derek
Subject: Re: [gdal-dev] Creating modified copies of a file

Le mercredi 06 juillet 2011 18:50:56, Cole, Derek a écrit :
> Sorry to follow up so soon. I also tried what is  int he API, using the
> following:
>
>     char **papszOptions = NULL;
>
>     papszOptions = CSLSetNameValue( papszOptions, "BLOCKXSIZE", "1024" );
>     papszOptions = CSLSetNameValue( papszOptions, "BLOCKYSIZE", "1024" );
>
>
>        poVRTDS = poVRTDriver->CreateCopy( "", poDataset, FALSE,
> papszOptions, MyTextProgress, message);

Yes that's better. The option array need to be NULL terminated, what
CSLSetNameValue() does (otherwise you could get nasty crashes). However, the
BLOCKXSIZE and BLOCKYSIZE options are options of the NITF driver, not the VRT
driver. Not sure why you are using VRT, I thought you wanted NITF, but I
haven't followed closely the whole thread

>
> It seems to have no affect. Is it being passed properly?
>
> Derek
> ________________________________________
> From: gdal-dev-bounces at lists.osgeo.org [gdal-dev-bounces at lists.osgeo.org]
> on behalf of Cole, Derek [dcole at integrity-apps.com] Sent: Wednesday, July
> 06, 2011 12:39 PM
> To: Even Rouault; gdal-dev at lists.osgeo.org
> Subject: RE: [gdal-dev] Creating modified copies of a file
>
> What is the proper way to set those creation options?
>
> I have tried the following:
>
>        char* papszOptions[] = { "BLOCKXSIZE=1024", "BLOCKYSIZE=1024" };
>        poVRTDS = poVRTDriver->CreateCopy( "", poDataset, FALSE,
> papszOptions, MyTextProgress, message); poBand =
> poVRTDS->GetRasterBand(1);
>        CPLAssert( poBand->GetRasterDataType() == GDT_Byte );
>        poBand->GetBlockSize( &nXBlockSize, &nYBlockSize );
>
> the block size variables are both still set to 128 instead of 1024, which
> is the block size of my input file.
>
>
> Thanks
>
> Derek
>
>
> ________________________________________
> From: Even Rouault [even.rouault at mines-paris.org]
> Sent: Tuesday, July 05, 2011 6:38 PM
> To: gdal-dev at lists.osgeo.org
> Cc: Cole, Derek
> Subject: Re: [gdal-dev] Creating modified copies of a file
>
> Le mercredi 06 juillet 2011 00:26:06, Cole, Derek a écrit :
> > Just as an interesting side-note: I tried to do a CreateCopy on one of
> > these files that was 7GB. Using CreateCopy() it took almost 2 hours. When
> > I simply copied to file contents using a cp command (this is CentOS), the
> > copy takes about a minute. I am not sure why that is so terribly slow in
> > GDAL.
>
> This is very well possible if the block dimensions of the source and target
> are not compatible. You might want to play with the BLOCKXSIZE and
> BLOCKYSIZE creation options.
>
> Anyway, you cannot simply compare cp and CreateCopy(). CreateCopy() can do
> much smarter things than just cp...
>
> > Also, I experimented trying to insert a band into an already existing
> > NITF file, but the software spits back an error message that the format
> > does not support it.
>
> Yes very few formats can support AddBand().
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev