[gdal-dev] RE: Memory Error in _gdal.Band_ReadRaster(*args,
**kwargs)
Paul Meems
bontepaarden at gmail.com
Fri Aug 14 16:08:56 EDT 2009
I've been in contact with Christian the author of gdal_retile and he told me
this problem is in the main GDAL library.
He also suggested using some of the other GDAL tools to see if the problem
persists.
So I started using gdal_translate to copy my ecw to a new ecw using this
statement:
gdal_translate -a_srs EPSG:28992 -of ECW -co "LARGE_OK=YES" large.ecw
Large_translate.ecw
After 6 hours (!) I got a new ecw file, this time 4.2GB with this info:
Driver: ECW/ERMapper Compressed Wavelets
Files: Large_translate.ecw
Size is 173334, 160000
Coordinate System is:
PROJCS["NUTM31",
GEOGCS["RIJKDRIEHOEKSMETING",
DATUM["RD",
SPHEROID["BESS1841",6377397.155,299.1528128]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433]],
PROJECTION["Transverse_Mercator"],
PARAMETER["latitude_of_origin",0],
PARAMETER["central_meridian",3],
PARAMETER["scale_factor",0.9996],
PARAMETER["false_easting",500000],
PARAMETER["false_northing",0],
UNIT["Meter",1]]
Origin = (33000.000000000000000,381000.000000000000000)
Pixel Size = (0.150000000001455,-0.150000000023283)
Corner Coordinates:
Upper Left ( 33000.000, 381000.000) ( 1d12'3.77"W, 3d26'16.85"N)
Lower Left ( 33000.000, 357000.000) ( 1d12'0.46"W, 3d13'17.23"N)
Upper Right ( 59000.100, 381000.000) ( 0d58'3.16"W, 3d26'20.47"N)
Lower Right ( 59000.100, 357000.000) ( 0d58'0.03"W, 3d13'20.61"N)
Center ( 46000.050, 369000.000) ( 1d 5'1.86"W, 3d19'48.81"N)
Band 1 Block=173334x1 Type=Byte, ColorInterp=Red
Overviews: arbitrary
Band 2 Block=173334x1 Type=Byte, ColorInterp=Green
Overviews: arbitrary
Band 3 Block=173334x1 Type=Byte, ColorInterp=Blue
Overviews: arbitrary
I tried to retile this ecw file using this command:
gdal_retile -v -s_srs EPSG:28992 -of ECW -ps 17335 16000 -targetDir tiles
Large_translate.ecw >retile.txt
But got the same memory error.
Next I tried converting to GeoTiff using this command:
gdal_translate -a_srs EPSG:28992 -of GTiff Large.ecw Large.tiff
After 2.5 hours I got a 77.5GB Tiff file with this info:
Driver: GTiff/GeoTIFF
Files: Large.tiff
Size is 173334, 160000
Coordinate System is:
PROJCS["Amersfoort / RD New",
GEOGCS["Amersfoort",
DATUM["Amersfoort",
SPHEROID["Bessel 1841",6377397.155,299.1528128000009,
AUTHORITY["EPSG","7004"]],
AUTHORITY["EPSG","6289"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4289"]],
PROJECTION["Oblique_Stereographic"],
PARAMETER["latitude_of_origin",52.15616055555555],
PARAMETER["central_meridian",5.387638888888889],
PARAMETER["scale_factor",0.9999079],
PARAMETER["false_easting",155000],
PARAMETER["false_northing",463000],
UNIT["metre",1,
AUTHORITY["EPSG","9001"]],
AUTHORITY["EPSG","28992"]]
Origin = (33000.000000000000000,381000.000000000000000)
Pixel Size = (0.150000000001455,-0.150000000023283)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 33000.000, 381000.000) ( 3d38'2.16"E, 51d24'21.02"N)
Lower Left ( 33000.000, 357000.000) ( 3d38'31.86"E, 51d11'24.60"N)
Upper Right ( 59000.100, 381000.000) ( 4d 0'27.21"E, 51d24'39.10"N)
Lower Right ( 59000.100, 357000.000) ( 4d 0'50.58"E, 51d11'42.56"N)
Center ( 46000.050, 369000.000) ( 3d49'27.95"E, 51d18'2.36"N)
Band 1 Block=173334x1 Type=Byte, ColorInterp=Red
Band 2 Block=173334x1 Type=Byte, ColorInterp=Green
Band 3 Block=173334x1 Type=Byte, ColorInterp=Blue
When I try to retile this tiff using:
gdal_retile -v -s_srs EPSG:28992 -of GTiff -ps 17335 16000 -targetDir tiles
Large.tiff >retile.txt
I got again the memory error.
I don't know anymore how to to this. How can I split my large ecw file into
smaller tiles?
I also noticed that even if I use the same EPSG code I get different
projection info, even differences when using only gdal_translate.
The projection is the same, they only have different naming. Why is that?
I also noticed both gdal_translate and gdal_retile not use the full
potential of my hardware. I'm running Vista 64Bit on a AMD3 2.6Ghz quadcore
with 8GB RAM and only 2-2.5GB RAM was used by gdal_translate, leaving more
than 5 GB free and only 35-55% of my CPU was used. Do I need to set some
settings to speed things up?
Thanks,
Paul
From: Paul Meems <bontepaarden <at> gmail.com>
Subject: Memory Error in _gdal.Band_ReadRaster(*args,
**kwargs)<http://news.gmane.org/find-root.php?message_id=%3c870e056a0908101422x5f38bb6fh32c2bd848ff6c868%40mail.gmail.com%3e>
Newsgroups: gmane.comp.gis.gdal.devel<http://news.gmane.org/gmane.comp.gis.gdal.devel>
Date: 2009-08-10 21:22:06 GMT
I'm trying to split a large ECW file into smaller files.
I've downloaded and installed OSGeo4W with gdal16
I managed to split a small ecw file.
That ecw file had this info:
Driver: ECW/ERMapper Compressed Wavelets
Files: tilertest.ecw
Size is 6800, 7500
Coordinate System is `'
Origin = (13000.000000000000000,381250.000000000000000)
Pixel Size = (2.500000000000000,-2.500000000000000)
Corner Coordinates:
Upper Left ( 13000.000, 381250.000)
Lower Left ( 13000.000, 362500.000)
Upper Right ( 30000.000, 381250.000)
Lower Right ( 30000.000, 362500.000)
Center ( 21500.000, 371875.000)
Band 1 Block=6800x1 Type=Byte, ColorInterp=Red
Overviews: arbitrary
Band 2 Block=6800x1 Type=Byte, ColorInterp=Green
Overviews: arbitrary
Band 3 Block=6800x1 Type=Byte, ColorInterp=Blue
Overviews: arbitrary
I used this statement:
gdal_retile -v -s_srs EPSG:28992 -of ECW -ps 3700 3750 -targetDir tiles
TilerTest.ecw
But when I try to split the large ecw file (3.4GB) I get an error after a
few minutes.
The info is:
Driver: ECW/ERMapper Compressed Wavelets
Files: Large.ecw
Size is 173334, 160000
Coordinate System is:
LOCAL_CS["LOCAL - (unsupported)",
UNIT["Meter",1]]
Origin = (33000.000000000000000,381000.000000000000000)
Pixel Size = (0.150000000001455,-0.150000000023283)
Corner Coordinates:
Upper Left ( 33000.000, 381000.000)
Lower Left ( 33000.000, 357000.000)
Upper Right ( 59000.100, 381000.000)
Lower Right ( 59000.100, 357000.000)
Center ( 46000.050, 369000.000)
Band 1 Block=173334x1 Type=Byte, ColorInterp=Red
Overviews: arbitrary
Band 2 Block=173334x1 Type=Byte, ColorInterp=Green
Overviews: arbitrary
Band 3 Block=173334x1 Type=Byte, ColorInterp=Blue
Overviews: arbitrary
The statement is:
gdal_retile -v -s_srs EPSG:28992 -of ECW -ps 86667 80000 -targetDir tiles
large.ecw >retile.txt
The logfile (retile.txt) shows:
Building internal Index for 1 tile(s) ... finished
Filename: Terneuzen_15cm.ecw
File Size: 173334x160000x3
Pixel Size: 0.150000 x -0.150000
UL:(33000.000000,381000.000000) LR:(59000.100000,356999.999996)
tileWidth 86667
tileHeight 80000
countTilesX: 2
countTilesY: 2
lastTileWidth: 86667
lastTileHeight: 80000
And the error is:
Traceback (most recent call last):
File "C:\OSGeo4W\apps\gdal-16\bin\gdal_retile.py", line 938, in <module>
sys.exit(main(sys.argv))
File "C:\OSGeo4W\apps\gdal-16\bin\gdal_retile.py", line 855, in main
dsCreatedTileIndex = tileImage(minfo,ti)
File "C:\OSGeo4W\apps\gdal-16\bin\gdal_retile.py", line 335, in tileImage
createTile(minfo, offsetX, offsetY, width, height,tilename,OGRDS)
File "C:\OSGeo4W\apps\gdal-16\bin\gdal_retile.py", line 467, in createTile
dec.uly+offsetY*dec.scaleY)
File "C:\OSGeo4W\apps\gdal-16\bin\gdal_retile.py", line 250, in getDataSet
data = s_band.ReadRaster( readOffsetX,readOffsetY,readX,readY,
readX,readY,self.band_type )
File "C:\OSGeo4W\apps\gdal-16\pymod\osgeo\gdal.py", line 760, in
ReadRaster return _gdal.Band_ReadRaster(*args, **kwargs) MemoryError
I've searched using Google and found a similar error:
http://article.gmane.org/gmane.comp.gis.gdal.devel/18936 and contacted
Sarah, but she's still having the problem.
Does anybody know how to solve this?
I run it on a Vista 64-Bit 4-core, with 8GB RAM. I'm getting the same error
on a WinXP sp3 2-core, with 2 GB RAM
Thanks,
Paul Meems
The Netherlands
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/gdal-dev/attachments/20090814/5d4c12f7/attachment.html
More information about the gdal-dev
mailing list