[gdal-dev] ECW reading is taking too much time

[Even Rouault]

Le vendredi 14 juin 2013 19:43:22, Jorge Arévalo a écrit :
> Hello,
> 
> I'm having problems reading a ECW file with GDAL. It's taking several hours
> to get an equivalent GeoTIFF file, executing gdal_translate. Besides, the
> ECW file is 1.9GB and the resulting GeoTIFF is 21GB. 

Not choking at all.

> The CPU is at almost
> 80% during the read operation.

Not choking at all. ECW decompression is expected to burn CPU

> 
> I don't know why there isn't srid information, but it should be epsg:2158

The mapping from ECW way of encoding SRS to WKT is far from being perfect. Try 
enabling CPL_DEBUG=ECW, and if there's some SRS information in the ECW file, 
you should see a line like :

ECW: projection=GEODETIC, datum=WGS84, units=DEGREES

> 
> I guess the speed depends on the reading strategy (looking at debug
> messages, looks like the driver is reading line by line), and you may get
> better performance combining ECW specific options with GDAL general
> options.
> 
> I'm using GDAL 1.10 and ECW 3.3. I've read about a known bug with ECW 3.3
> and machines with more than 8GB RAM
> (http://osgeo-org.1560.x6.nabble.com/Re-Qgis-developer-Large-raster-ecw-id
> entify-very-long-td4996966.html), but it's not my case.
> 
> Is there any know strategy to make the ECW driver reading faster? I'm just
> executing these operations:
> 
> - For the GeoTiff transformation:
> 
> gdal_translate -a_srs epsg:2158 PNOA_MR_OF_ETRS89_HU29_h50_0223.ecw
> PNOA_MR_OF_ETRS89_HU29_h50_0223.tif

Yes tweaking ECW reading performance can be tricky. I've done a few 
experiments (on a 3 band 141970x141970 Landsat ECW) , and couldn't really 
found something that would improve things dramatically.
You can try defining the GDAL_SWATH_SIZE configuration option to a bigger value 
than its default (the default is 10 MB). Beware you must specify the value in 
bytes (lot of zeros). I've tried with 100 MB or 500 MB but couldn't really 
observe a real improvement.

Note: to benefit from the option, I'd recommand that you just do 
"gdal_translate src.ecw out.tif" and remove the -a_srs option, that would use 
intermediary VRT. You can alter the SRS of out.tif with the gdal_edit.py 
script.

I've also tried the following patch that indicates to the ECW driver that we 
will read the whole image at full resolution, but it seemed to decrease 
performance a bit...

Index: gcore/rasterio.cpp
===================================================================
--- gcore/rasterio.cpp	(revision 26076)
+++ gcore/rasterio.cpp	(working copy)
@@ -2927,6 +2927,16 @@
             "GDALDatasetCopyWholeRaster(): %d*%d swaths, bInterleave=%d", 
             nSwathCols, nSwathLines, bInterleave );
 
+    int* panBandMap = (int*) CPLMalloc(sizeof(int) * nBandCount);
+    int i;
+    for(i=0;i<nBandCount;i++)
+        panBandMap[i] = i + 1;
+    GDALDatasetAdviseRead( (GDALDatasetH)poSrcDS, 
+                           0, 0, nXSize, nYSize,
+                           nXSize, nYSize, eDT,
+                           nBandCount, panBandMap, NULL );
+    CPLFree(panBandMap);
+
 /* ==================================================================== */
 /*      Band oriented (uninterleaved) case.                             */
 /* ==================================================================== */

> 
> - For the PostGIS Raster loading:
> 
> raster2pgsql -s epsg:2158 -t 15x15 -I -M -C -Y
> PNOA_MR_OF_ETRS89_HU29_h50_0223.ecw > output.sql (I've tried with
> different tile sizes, with the same result)

raster2pgsql uses its own reading logic, so the above tricks might not apply. 
-t 15x15 looks like a really tiny value. That will cause a lot of overhead in 
the database I think. 256x256 would probably be better.

> 
> Any clues?
> 
> Many thanks in advance,

-- 
Geospatial professional services
http://even.rouault.free.fr/services.html