[Gdal-dev] reproject a big ecw

Thu Oct 18 10:39:18 EDT 2007

Hi,

Thanks for you prompt response.

The BIL file produced is about 37 Go.
I'm going to try the same function with HFA format like you suggest.
If this doesn't resolve my problem I will wait the next FWTools version.
Frank, do you think that I should try to process gdalwarp without "--config GDAL_CACHEMAX 400 -wm 400" ?

Thanks for your help

Here is all the messages returned by "--DEBUG ON" :

----------------------------------------------------------------------------------
C:\Program Files\FWTools1.3.9>gdalwarp --DEBUG ON --config GDAL_CACHEMAX 400 -wm
 400 -rn -tr 0.5 0.5 -of EHdr -co "INTERLEAVE=BIL" -s_srs "+proj=lcc +a=6378249.
2 +b=6356515.0 +lat_0=46d48'0.0N +lon_0=2d20'14.025E +lat_1=45d53'56.108N +lat_2
=47d41'45.652N +x_0=600000.0 +y_0=2200000.0 +nadgrids=NTF2RGF93.gsb +units=m +wk
text"  -t_srs EPSG:2154 "G:\34-2005-LA2E-C10.ecw" "I:\34-2005-L93-C10.bil"
ECW: NCScbmOpenFileView(G:\34-2005-LA2E-C10.ecw): eErr = 0
ECW: FileInfo: SizeXY=280000,180000 Bands=3
       OriginXY=615000,1.89e+006  CellIncrementXY=0.5,-0.5

ECW: projection=LM2FRANC, datum=NTF
GDAL: GDALOpen(G:\34-2005-LA2E-C10.ecw) succeeds as ECW.
OGRCT: Source: +proj=lcc +a=6378249.2 +b=6356515.0 +lat_0=46d48'0.0N +lon_0=2d20
'14.025E +lat_1=45d53'56.108N +lat_2=47d41'45.652N +x_0=600000.0 +y_0=2200000.0
+nadgrids=NTF2RGF93.gsb +units=m +wktext
OGRCT: Target: +proj=lcc +lat_1=49 +lat_2=44 +lat_0=46.5 +lon_0=3 +x_0=700000 +y
_0=6600000 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs
OGRCT: Source: +proj=lcc +lat_1=49 +lat_2=44 +lat_0=46.5 +lon_0=3 +x_0=700000 +y
_0=6600000 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs
OGRCT: Target: +proj=lcc +a=6378249.2 +b=6356515.0 +lat_0=46d48'0.0N +lon_0=2d20
'14.025E +lat_1=45d53'56.108N +lat_2=47d41'45.652N +x_0=600000.0 +y_0=2200000.0
+nadgrids=NTF2RGF93.gsb +units=m +wktext
GDAL: GDALClose(G:\34-2005-LA2E-C10.ecw)
Creating output file that is 281206P x 182161L.
GDAL: GDALDriver::Create(EHdr,I:\34-2005-L93-C10.bil,281206,182161,3,Byte,017401
68)
GDALRaw: RawRasterBand(017CF2A8,1,017CD768,
              Off=0,PixOff=1,LineOff=843618,Byte,1)

GDALRaw: RawRasterBand(017CF2A8,2,017CD768,
              Off=281206,PixOff=1,LineOff=843618,Byte,1)

GDALRaw: RawRasterBand(017CF2A8,3,017CD768,
              Off=562412,PixOff=1,LineOff=843618,Byte,1)

GDAL: GDALOpen(I:\34-2005-L93-C10.bil) succeeds as EHdr.
ECW: NCScbmOpenFileView(G:\34-2005-LA2E-C10.ecw): eErr = 0
ECW: FileInfo: SizeXY=280000,180000 Bands=3
       OriginXY=615000,1.89e+006  CellIncrementXY=0.5,-0.5

ECW: projection=LM2FRANC, datum=NTF
GDAL: GDALOpen(G:\34-2005-LA2E-C10.ecw) succeeds as ECW.
Processing input file G:\34-2005-LA2E-C10.ecw.
OGRCT: Source: +proj=lcc +a=6378249.2 +b=6356515.0 +lat_0=46d48'0.0N +lon_0=2d20
'14.025E +lat_1=45d53'56.108N +lat_2=47d41'45.652N +x_0=600000.0 +y_0=2200000.0
+nadgrids=NTF2RGF93.gsb +units=m +wktext
OGRCT: Target: +proj=lcc +lat_1=49 +lat_2=44 +lat_0=46.5 +lon_0=3 +x_0=700000 +y
_0=6600000 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs
OGRCT: Source: +proj=lcc +lat_1=49 +lat_2=44 +lat_0=46.5 +lon_0=3 +x_0=700000 +y
_0=6600000 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs
OGRCT: Target: +proj=lcc +a=6378249.2 +b=6356515.0 +lat_0=46d48'0.0N +lon_0=2d20
'14.025E +lat_1=45d53'56.108N +lat_2=47d41'45.652N +x_0=600000.0 +y_0=2200000.0
+nadgrids=NTF2RGF93.gsb +units=m +wktext
ECWDataset: RasterIO(0,0,7332,5711 -> 7332x5711) - doing interleaved read.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=0,0,7332x5711 Dst=0,0,8787x5692
:0ECWDataset: RasterIO(0,5636,7380,5774 -> 7380x5774) - doing interleaved read.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=0,5636,7380x5774 Dst=0,5692,8787x5693
ECWDataset: RasterIO(7283,0,8846,5637 -> 8846x5637) - doing interleaved read.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=7283,0,8846x5637 Dst=8787,0,8788x5692
ECWDataset: RasterIO(7331,5562,8846,5774 -> 8846x5774) - doing interleaved read.

GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=7331,5562,8846x5774 Dst=8787,5692,8788x5693
ECWDataset: RasterIO(0,11335,7428,5773 -> 7428x5773) - doing interleaved read.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=0,11335,7428x5773 Dst=0,11385,8787x5692
ECWDataset: RasterIO(0,17033,7476,5773 -> 7476x5773) - doing interleaved read.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=0,17033,7476x5773 Dst=0,17077,8787x5693
ECWDataset: RasterIO(7379,11261,8846,5773 -> 8846x5773) - doing interleaved read
.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=7379,11261,8846x5773 Dst=8787,11385,8788x5692
ECWDataset: RasterIO(7427,16959,8846,5774 -> 8846x5774) - doing interleaved read
.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=7427,16959,8846x5774 Dst=8787,17077,8788x5693
ECWDataset: RasterIO(16080,0,8845,5563 -> 8845x5563) - doing interleaved read.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=16080,0,8845x5563 Dst=17575,0,8787x5692
ECWDataset: RasterIO(16128,5488,8845,5774 -> 8845x5774) - doing interleaved read
.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=16128,5488,8845x5774 Dst=17575,5692,8787x5693
ECWDataset: RasterIO(24876,0,8846,5489 -> 8846x5489) - doing interleaved read.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=24876,0,8846x5489 Dst=26362,0,8788x5692
ECWDataset: RasterIO(24924,5414,8846,5774 -> 8846x5774) - doing interleaved read
.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=24924,5414,8846x5774 Dst=26362,5692,8788x5693
ECWDataset: RasterIO(16176,11187,8845,5773 -> 8845x5773) - doing interleaved rea
d.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=16176,11187,8845x5773 Dst=17575,11385,8787x5692
ECWDataset: RasterIO(16224,16885,8845,5774 -> 8845x5774) - doing interleaved rea
d.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=16224,16885,8845x5774 Dst=17575,17077,8787x5693
ECWDataset: RasterIO(24972,11113,8846,5773 -> 8846x5773) - doing interleaved rea
d.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=24972,11113,8846x5773 Dst=26362,11385,8788x5692
ECWDataset: RasterIO(25020,16811,8846,5774 -> 8846x5774) - doing interleaved rea
d.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=25020,16811,8846x5774 Dst=26362,17077,8788x5693
ECWDataset: RasterIO(0,22732,7524,5772 -> 7524x5772) - doing interleaved read.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=0,22732,7524x5772 Dst=0,22770,8787x5692
ECWDataset: RasterIO(0,28429,7572,5774 -> 7572x5774) - doing interleaved read.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=0,28429,7572x5774 Dst=0,28462,8787x5693
ECWDataset: RasterIO(7475,22658,8846,5772 -> 8846x5772) - doing interleaved read
.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=7475,22658,8846x5772 Dst=8787,22770,8788x5692
ECWDataset: RasterIO(7523,28355,8846,5774 -> 8846x5774) - doing interleaved read
.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=7523,28355,8846x5774 Dst=8787,28462,8788x5693
ECWDataset: RasterIO(0,34128,7619,5773 -> 7619x5773) - doing interleaved read.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=0,34128,7619x5773 Dst=0,34155,8787x5692
ECWDataset: RasterIO(0,39826,7667,5774 -> 7667x5774) - doing interleaved read.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=0,39826,7667x5774 Dst=0,39847,8787x5693
ECWDataset: RasterIO(7571,34054,8846,5773 -> 8846x5773) - doing interleaved read
.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=7571,34054,8846x5773 Dst=8787,34155,8788x5692
ECWDataset: RasterIO(7618,39752,8846,5774 -> 8846x5774) - doing interleaved read
.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=7618,39752,8846x5774 Dst=8787,39847,8788x5693
ECWDataset: RasterIO(16272,22584,8845,5772 -> 8845x5772) - doing interleaved rea
d.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=16272,22584,8845x5772 Dst=17575,22770,8787x5692
ECWDataset: RasterIO(16320,28282,8845,5773 -> 8845x5773) - doing interleaved rea
d.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=16320,28282,8845x5773 Dst=17575,28462,8787x5693
.ECWDataset: RasterIO(25068,22510,8846,5773 -> 8846x5773) - doing interleaved re
ad.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=25068,22510,8846x5773 Dst=26362,22770,8788x5692
ECWDataset: RasterIO(25116,28208,8846,5773 -> 8846x5773) - doing interleaved rea
d.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=25116,28208,8846x5773 Dst=26362,28462,8788x5693
ECWDataset: RasterIO(16368,33980,8845,5773 -> 8845x5773) - doing interleaved rea
d.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=16368,33980,8845x5773 Dst=17575,34155,8787x5692
ECWDataset: RasterIO(16416,39678,8844,5774 -> 8844x5774) - doing interleaved rea
d.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=16416,39678,8844x5774 Dst=17575,39847,8787x5693
ECWDataset: RasterIO(25164,33906,8846,5773 -> 8846x5773) - doing interleaved rea
d.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=25164,33906,8846x5773 Dst=26362,34155,8788x5692
ECWDataset: RasterIO(25212,39604,8846,5774 -> 8846x5774) - doing interleaved rea
d.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=25212,39604,8846x5774 Dst=26362,39847,8788x5693
ECWDataset: RasterIO(33673,0,8845,5415 -> 8845x5415) - doing interleaved read.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=33673,0,8845x5415 Dst=35150,0,8787x5692
ECWDataset: RasterIO(33721,5340,8845,5774 -> 8845x5774) - doing interleaved read
.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=33721,5340,8845x5774 Dst=35150,5692,8787x5693
ECWDataset: RasterIO(42469,0,8846,5341 -> 8846x5341) - doing interleaved read.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=42469,0,8846x5341 Dst=43937,0,8788x5692
ECWDataset: RasterIO(42517,5266,8846,5774 -> 8846x5774) - doing interleaved read
.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=42517,5266,8846x5774 Dst=43937,5692,8788x5693
ECWDataset: RasterIO(33769,11039,8845,5773 -> 8845x5773) - doing interleaved rea
d.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=33769,11039,8845x5773 Dst=35150,11385,8787x5692
ECWDataset: RasterIO(33817,16737,8845,5774 -> 8845x5774) - doing interleaved rea
d.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=33817,16737,8845x5774 Dst=35150,17077,8787x5693
ECWDataset: RasterIO(42565,10965,8846,5773 -> 8846x5773) - doing interleaved rea
d.
GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
Src=42565,10965,8846x5773 Dst=43937,11385,8788x5692
ERROR 2: Failed to allocate 153230412 byte source buffer.
GDAL: GDALClose(G:\34-2005-LA2E-C10.ecw)
GDAL: GDALClose(I:\34-2005-L93-C10.bil)
GDAL: GDALDeregister_GTiff() called.
---------------------------------------------------------------------------------------

-----Message d'origine-----
De : Frank Warmerdam [mailto:warmerdam at pobox.com]
Envoyé : jeudi 18 octobre 2007 15:51
À : "SAEZ Laurent - CETE Méditerr./DI/ETER"
Cc : gdal-dev at lists.maptools.org
Objet : Re: [Gdal-dev] reproject a big ecw

SAEZ Laurent - CETE Méditerr./DI/ETER wrote:
> Hi,
> 
> I've a big problem with gdalwarp.
> I've to reproject a big ecw from the NTF - Lambert II étendu to the RGF93 - Lambert 93.
> The ecw source file is about 7 Go. The final file will be also in ecw format.
> I use FWTools 1.3.9 on Windows XP SP2 with 2 Go of RAM.
> I have to make an intermediate file in ENVI hdr format because it seems that gdalwarp can't creat ecw files.
> I use the syntax below to do the job :
> 
> gdalwarp --DEBUG ON --config GDAL_CACHEMAX 400 -wm 400 -rn -tr 0.5 0.5 -of EHdr -co "INTERLEAVE=BIL" -s_srs "+proj=lcc +a=6378249.2 +b=6356515.0 +lat_0=46d48'0.0N +lon_0=2d20'14.025E +lat_1=45d53'56.108N +lat_2=47d41'45.652N +x_0=600000.0 +y_0=2200000.0 +nadgrids=NTF2RGF93.gsb +units=m +wktext"  -t_srs EPSG:2154 "G:\34-2005-LA2E-C10.ecw" "I:\34-2005-L93-C10.bil"
> 
> The job is stopped with the following message :
> ECWDataset: RasterIO(42565,10965,8846,5773 -> 8846x5773) - doing interleaved rea
> d.
> GDAL: GDALWarpKernel()::GWKNearestNoMasksByte()
> Src=42565,10965,8846x5773 Dst=43937,11385,8788x5692
> ERROR 2: Failed to allocate 153230412 byte source buffer.
> GDAL: GDALClose(G:\34-2005-LA2E-C10.ecw)
> GDAL: GDALClose(I:\34-2005-L93-C10.bil)
> GDAL: GDALDeregister_GTiff() called.

Larent,

The error is an "out of memory" error.  One possibility is that you
configured things (via GDAL_CACHEMAX and -wm) to use more
memory than you have practically available.  The other is that there
is a memory leak somewhere that is causing problems.

Did you get this error immediately or was there a bunch of progress (as
reported by the percent complete progress monitor) first?

The recent fix to a memory leak described in:

   http://trac.osgeo.org/gdal/ticket/1922

might be related, though I'm a bit doubtful.

PS. using a tiled format such as HFA (Erdas Imagine) may be better than
a scanline oriented format like EHdr for your working file.  It maps better
to the "chunks" that the warper operates on.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org