[gdal-dev] Converting ecw to tif for Geoserver

Sun Feb 10 11:30:09 PST 2013

After almost 27 hours of processing I finally have a result which looks
good in MapWindow.
I did however noticed that the 'normal' translate-addOverviews process
produces a tif-file with a size of 1.1GB. The original ecw is 2.6GB.
But when I convert the ecw using these steps to make it transparent:

echo # Make almost white pixel truly white:
nearblack -of GTiff -white -o white.tif my.ecw

echo Warp and mark white pixels as transparent
gdalwarp --config GDAL_CACHEMAX 500 -wm 500 -multi -of GTiff -dstalpha
-srcnodata "255 255 255" -co tiled=yes -co "BLOCKXSIZE=512" -co
"BLOCKYSIZE=512" -s_srs EPSG:28992 -r lanczos -overwrite white.tif
warped.tif

echo Compress
gdal_translate -of GTiff -a_srs EPSG:28992 -b 1 -b 2 -b 3 -co tiled=yes -co
compress=jpeg -co photometric=YCBCR -co BLOCKXSIZE=512 -co BLOCKYSIZE=512
--config GDAL_TIFF_INTERNAL_MASK YES warped.tif compressed.tif

echo nodata is black now. Make al blackish pixels black:
nearblack -of GTiff -o compressed-black.tif compressed.tif

echo set nodata value to black:
gdal_translate -a_nodata 0 compressed-black.tif compressed-nodata.tif

echo Add overviews
gdaladdo -r average --config COMPRESS_OVERVIEW JPEG --config
PHOTOMETRIC_OVERVIEW YCBCR compressed-nodata.tif 2 4 8 16 32 64 128

I get a tif file with a size of 31GB!

When I do a gdalinfo I get this:

Driver: GTiff/GeoTIFF
Files: compressed-nodata.tif
Size is 100000, 80000
Coordinate System is:
PROJCS["Amersfoort / RD New",
[cut]
Origin = (149000.000000000000000,384000.000000000000000)
Pixel Size = (0.100000000000000,-0.100000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (  149000.000,  384000.000) (  5d18' 4.75"E, 51d26'45.47"N)
Lower Left  (  149000.000,  376000.000) (  5d18' 5.24"E, 51d22'26.57"N)
Upper Right (  159000.000,  384000.000) (  5d26'42.66"E, 51d26'45.54"N)
Lower Right (  159000.000,  376000.000) (  5d26'42.34"E, 51d22'26.63"N)
Center      (  154000.000,  380000.000) (  5d22'23.75"E, 51d24'36.13"N)
Band 1 Block=100000x1 Type=Byte, ColorInterp=Red
  NoData Value=0
  Overviews: 50000x40000, 25000x20000, 12500x10000, 6250x5000, 3125x2500,
1563x1
250, 782x625
Band 2 Block=100000x1 Type=Byte, ColorInterp=Green
  NoData Value=0
  Overviews: 50000x40000, 25000x20000, 12500x10000, 6250x5000, 3125x2500,
1563x1
250, 782x625
Band 3 Block=100000x1 Type=Byte, ColorInterp=Blue
  NoData Value=0
  Overviews: 50000x40000, 25000x20000, 12500x10000, 6250x5000, 3125x2500,
1563x1
250, 782x625

I don't see anything about compression.
Did I loose it along the way? Or is it expected to have such a large sized
end result.
Because it is now so large I think I'll get into trouble when I add it to
Geoserver. I hoped to use the tiff directly and not use Geoserver's pyramid
plug-in.

Thanks,

Paul

2013/2/9 Paul Meems <bontepaarden at gmail.com>

> Thanks Even,
>
> The low-res option really helped. Now I can test the commands in seconds
> instead of hours.
> I did these commands which seem to produce a tif file with overviews and
> correct transparency.
> I've opened it in MapWindow for now to check. MapWindow is using GDAL to
> read raster files.
>
> I will now convert the hi-res version and try that version in Geoserver.
> Here are the commands I used:
>
> # Create a low-res version:
> gdal_translate my.ecw test\lowres.tif -outsize 1% 1%
>
> # Fix the almost white pixels:
> nearblack -of GTiff -white -o lowres-white.tif lowres.tif
>
>
> # Warp and mark white pixels as transparent
> gdalwarp -multi -of GTiff -dstalpha -srcnodata "255 255 255" -co tiled=yes
> -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -s_srs EPSG:28992 -r lanczos
> -overwrite lowres-white.tif lowres-transparent.tif
>
> # Create internal tiles, don't use mask band because Geoserver doesn't
> like that:
> gdal_translate -of GTiff -a_srs EPSG:28992 -b 1 -b 2 -b 3 -co tiled=yes
> -co compress=jpeg -co photometric=YCBCR -co BLOCKXSIZE=512 -co
> BLOCKYSIZE=512 --config GDAL_TIFF_INTERNAL_MASK YES lowres-transparent.tif
> lowres-compressed.tif
>
> # nodata is now black, but not all pixels are truly black:
> nearblack -of GTiff -o lowres-compressed-black.tif lowres-compressed.tif
>
> # Set the nodata value to black:
> gdal_translate -a_nodata 0 lowres-compressed-black.tif
> lowres-compressed-transparent.tif
>
> # Add overviews
> gdaladdo -r average --config COMPRESS_OVERVIEW JPEG --config
> PHOTOMETRIC_OVERVIEW YCBCR lowres-compressed-transparent.tif 2 4 8 16 32 64
> 128
>
>
> Thanks all.
>
>
> Paul
>
> 2013/2/9 Even Rouault <even.rouault at mines-paris.org>
>
>> Le samedi 09 février 2013 13:48:13, Paul Meems a écrit :
>> > The last days I've been trying to convert my ecw file to a GeoTiff so I
>> can
>> > use it in Geoserver.
>> > The ecw is a mozaik of aerial photos.
>> > When I do these two commands it is working in Geoserver:
>> >
>> >    - gdal_translate -of GTiff -co tiled=yes -co compress=jpeg -co
>> >    photometric=YCBCR -a_srs EPSG:28992 -co "BLOCKXSIZE=512" -co
>> >    "BLOCKYSIZE=512" my.ecw my.tif
>> >    - gdaladdo -r average --config COMPRESS_OVERVIEW JPEG --config
>> >    PHOTOMETRIC_OVERVIEW YCBCR  my.tif 2 4 8 16 32 64 128
>> >
>> > But I want to add a second reference layer beneath this one which is
>> > larger. The translated ecw has white edges, so that is not looking nice.
>> > I've asked about this before and suggested was to use a cutline. So I
>> > create a polygon that is inside the ecw file and ran this command:
>> >
>> >    - gdalwarp -multi -cutline "clip.shp" -of GTiff -co tiled=yes -co
>> >    compress=jpeg -co photometric=YCBCR -co "BLOCKXSIZE=512" -co
>> >    "BLOCKYSIZE=512" -s_srs EPSG:28992 -r lanczos -overwrite my.ecw
>> > clipped.tif
>> >
>> > When I open clipped.tif in MapWindow I can see the white edges are gone.
>> > But when I call *gdaladdo* white artifacts are back again.
>>
>> I'm not clear why gdaladdo would add while artifacts if clipped.tif has no
>> more white borders (i.e. if your clipping shape is a rectangle completely
>> inside the validity area of the ECW). Unless clipped.tif has an alpha
>> channel,
>> but the gdalwarp command line you showed has no -dstalpha. But that
>> wouldn't
>> work actually since you use JPEG YCbCr compression, which doesn't support
>> alpha channels.
>>
>> A solution - provided that MapWindows supports GDAL mask bands as
>> transparency
>> channels - would be :
>>
>> # Warp and mark white pixels as transparent
>> gdalwarp 1.tif 2.tif -dstalpha -srcnodata "255 255 255" [other_options]
>>
>> # Transform 2.tif as a JPEG-YCBCR-in-TIFF with an internal mask band
>> gdal_translate 2.tif 3.tif -b 1 -b 2 -b 3 -mask 4 -co compress=jpeg -co
>> photometric=YCBCR -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 --config
>> GDAL_TIFF_INTERNAL_MASK YES
>>
>> # Add overviews
>> gdaladdo 3.tif  -r average --config COMPRESS_OVERVIEW JPEG --config
>>    PHOTOMETRIC_OVERVIEW YCBCR  2 4
>>
>> > I'm now in the process of calling *nearblack* on both my.tiff and
>> > clipped.tif but I'm not sure I'm doing it right. I think I'm using the
>> > correct commands but perhaps in the wrong order.
>> > The ecw file isn't very large (2.6GB) but all steps take a long time,
>> > especially *nearblack *which is running for 4 hours now and is just at
>> 10%.
>> >
>>
>> For the experiments, I'd encourage working with a low res extract of the
>> ECW :
>> gdal_translate src.ecw lowres.tif -outsize 1% 1%
>>
>> > Does anybody have some suggestions how to process the steps to convert
>> an
>> > ecw file which has white outside areas to a transparent tif file with
>> > overviews?
>> >
>> > I'm running the latest GDAL v1.10 version from Tamas on Windows Vista
>> on a
>> > 8GB 4core machine. Speed is not a big issue when I know the result will
>> be
>> > OK.
>>
>> Another point: It is well known that warping with compression doesn't
>> produce
>> files with optimal size. See
>> http://trac.osgeo.org/gdal/wiki/UserDocs/GdalWarp#GeoTIFFoutput-
>> coCOMPRESSisbroken
>>
>> >
>> > Thanks,
>> >
>> > Paul
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20130210/ad566718/attachment-0001.html>