[gdal-dev] ECW reading is taking too much time

[Jorge Arévalo]

Hello,  


El viernes 14 de junio de 2013 a las 20:50, Even Rouault escribió:

> Le vendredi 14 juin 2013 19:43:22, Jorge Arévalo a écrit :
> > Hello,
> >  
> > I'm having problems reading a ECW file with GDAL. It's taking several hours
> > to get an equivalent GeoTIFF file, executing gdal_translate. Besides, the
> > ECW file is 1.9GB and the resulting GeoTIFF is 21GB.  
>  
>  
>  
> Not choking at all.
>  
> > The CPU is at almost
> > 80% during the read operation.
>  
>  
>  
> Not choking at all. ECW decompression is expected to burn CPU
>  
> >  
> > I don't know why there isn't srid information, but it should be epsg:2158
>  
> The mapping from ECW way of encoding SRS to WKT is far from being perfect. Try  
> enabling CPL_DEBUG=ECW, and if there's some SRS information in the ECW file,  
> you should see a line like :
>  
> ECW: projection=GEODETIC, datum=WGS84, units=DEGREES
>  
Yes. This is what I see

ECW: projection=NUTM29, datum=ETRS89, units=METERS

That is http://spatialreference.org/ref/epsg/25829/, not epsg:2158. I was wrong.

Many thanks!
  
>  
> >  
> > I guess the speed depends on the reading strategy (looking at debug
> > messages, looks like the driver is reading line by line), and you may get
> > better performance combining ECW specific options with GDAL general
> > options.
> >  
> > I'm using GDAL 1.10 and ECW 3.3. I've read about a known bug with ECW 3.3
> > and machines with more than 8GB RAM
> > (http://osgeo-org.1560.x6.nabble.com/Re-Qgis-developer-Large-raster-ecw-id
> > entify-very-long-td4996966.html), but it's not my case.
> >  
> > Is there any know strategy to make the ECW driver reading faster? I'm just
> > executing these operations:
> >  
> > - For the GeoTiff transformation:
> >  
> > gdal_translate -a_srs epsg:2158 PNOA_MR_OF_ETRS89_HU29_h50_0223.ecw
> > PNOA_MR_OF_ETRS89_HU29_h50_0223.tif
>  
>  
>  
> Yes tweaking ECW reading performance can be tricky. I've done a few  
> experiments (on a 3 band 141970x141970 Landsat ECW) , and couldn't really  
> found something that would improve things dramatically.
> You can try defining the GDAL_SWATH_SIZE configuration option to a bigger value  
> than its default (the default is 10 MB). Beware you must specify the value in  
> bytes (lot of zeros). I've tried with 100 MB or 500 MB but couldn't really  
> observe a real improvement.
>  
> Note: to benefit from the option, I'd recommand that you just do  
> "gdal_translate src.ecw out.tif" and remove the -a_srs option, that would use  
> intermediary VRT. You can alter the SRS of out.tif with the gdal_edit.py  
> script.
>  
That actually worked fine. I got a 40% GeoTIFF version. I built some overviews and added srs with gdal_edit.py, as you suggested. The quality is enough for me, and the size of the file, overviews included, is about 8GB. Not so big. Many thanks again!

> I've also tried the following patch that indicates to the ECW driver that we  
> will read the whole image at full resolution, but it seemed to decrease  
> performance a bit...
>  
> Index: gcore/rasterio.cpp
> ===================================================================
> --- gcore/rasterio.cpp (revision 26076)
> +++ gcore/rasterio.cpp (working copy)
> @@ -2927,6 +2927,16 @@
> "GDALDatasetCopyWholeRaster(): %d*%d swaths, bInterleave=%d",  
> nSwathCols, nSwathLines, bInterleave );
>  
> + int* panBandMap = (int*) CPLMalloc(sizeof(int) * nBandCount);
> + int i;
> + for(i=0;i<nBandCount;i++)
> + panBandMap[i] = i + 1;
> + GDALDatasetAdviseRead( (GDALDatasetH)poSrcDS,  
> + 0, 0, nXSize, nYSize,
> + nXSize, nYSize, eDT,
> + nBandCount, panBandMap, NULL );
> + CPLFree(panBandMap);
> +
> /* ==================================================================== */
> /* Band oriented (uninterleaved) case. */
> /* ==================================================================== */
>  
My CPU is burning again, but doesn't matter. With the 40% outsize is enough.
  
>  
> >  
> > - For the PostGIS Raster loading:
> >  
> > raster2pgsql -s epsg:2158 -t 15x15 -I -M -C -Y
> > PNOA_MR_OF_ETRS89_HU29_h50_0223.ecw > output.sql (I've tried with
> > different tile sizes, with the same result)
>  
>  
>  
> raster2pgsql uses its own reading logic, so the above tricks might not apply.  
> -t 15x15 looks like a really tiny value. That will cause a lot of overhead in  
> the database I think. 256x256 would probably be better.
>  
Yes. I was trying with several scenarios (small tiles than fit in postgres page size, and some other tests). But this tiny tile size causes an overhead that burns the cpu again. So, I'll look for a bigger one.

Great help, thanks again!
  
>  
> >  
> > Any clues?
> >  
> > Many thanks in advance,
>  
> --  
> Geospatial professional services
> http://even.rouault.free.fr/services.html