<div dir="ltr"><div><div><div><div><div><div><div><div><div><div>Hi All,<br><br></div>I'm still working on my process to make my ecw ready for GeoServer and enable transparency.<br></div>I've written about it before and got some valuable feedback that allowed me to convert my first ecw.<br>
</div>I now have a second ecw that doesn't work properly with my first steps. For this ecw the resulting artifacts are too big.<br><br></div>I have an aerial photo in ECW format which is composed from several smaller photos (mosaic) giving me large white areas in the corners. These white areas need to be one color so Geoserver can make it transparent and I can put a second layer beneath it. I've tried this with my first ecw file and it is working great in Geoserver.<br>
<br></div>These are the steps that I take to end up with a tiff file with correct nodata values and compression:<br></div><b>Step 1</b>: Convert the ecw to tiff and make all white pixels true white:<br>nearblack -of GTiff -white -o white.tif my.ecw<br>
<b>Step 2</b>: Mark white pixels as transparent<br>gdalwarp --config GDAL_CACHEMAX 500 -wm 500 -multi -of GTiff -srcnodata "255 255 255" -dstnodata "255 255 255" -co tiled=yes -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -s_srs EPSG:28992 -overwrite white.tif nodata.tif<br>
</div></div><b>Step 3</b>: Compress<br>gdal_translate -of GTiff -a_srs EPSG:28992 -co "TILED=YES" -co "BIGTIFF=YES" -co "PROFILE=GEOTIFF" -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -co "COMPRESS=JPEG" -co "JPEG_QUALITY=80" -co "PHOTOMETRIC=YCBCR" --config GDAL_TIFF_INTERNAL_MASK YES nodata.tif compressed.tif<br>
</div><div><b>Step 4</b>: Add overviews<br>gdaladdo -r average --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW YCBCR --config INTERLEAVE_OVERVIEW BAND compressed.tif 2 4 8 16 32 64 128 256<br></div><div><br>
</div><div>I only do step 2 because nearblack doesn't set the nodata value. Why is that? Doesn't it make sense to do so or is it technically not possible?<br><br></div><div>My input ECW file is 3.5GB, white.tif (created by nearblack) is 75GB. After compression and adding overviews I have a 3.8GB tiff file. This tiff file has white artifacts at the edge of the photo data and nodata.<br>
</div><div>I don't understand why. My input file (nodata.tif) for the compression has proper nodata values set, why doesn't the compression step retain them? I can imagine due to the compression algorithm some nodata values are changed but isn't it possible to reset them again before saving?<br>
<br></div><div>Because of the artifacts I created a masked tif after step 2 from nodata.tif. In this masked tiff all data that is not nodata is set to black. Next I run gdal_polygonize.py on this masked data and after 7 days! I end up with a 110mb shapefile. I have such a large shapefile because step 2 also set some white pixels in the photo area to nodata and those pixels are also converted to shapes.<br>
</div><div>After some vector processing I finally have my outline of my photo area as a shapefile.<br></div><div>Now I clip the result of step 4 with this shapefile:<br><b>Step 5</b>: Clip raster with shapefile:<br>gdalwarp -cutline clip.shp -cl clip -crop_to_cutline --config GDAL_CACHEMAX 500 -wm 500 -multi -of GTiff -srcnodata "255 255 255" -dstnodata "255 255 255" -co tiled=yes -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -s_srs EPSG:28992 compressed.tif crop.tif<br>
<br></div><div>This cropped file looks great, the artifacts are gone but the overviews are lost.<br></div><div>So I call gdal_addo again:<br>gdaladdo -r average -ro --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW YCBCR --config INTERLEAVE_OVERVIEW PIXEL crop.tif 2 4 8 16 32 64 128 256<br>
</div><div>Now the artifacts are back again ;(<br></div><div>Now I tried again without compression:<br>gdaladdo -r average -ro crop.tif 2 4 8 16 32 64 128 256<br></div><div>This results in a huge .ovr file, it is still processing. But at 25% the size is already 7.2GB<br>
<br></div><div>Meanwhile I tried something else:</div><div>Because I already have my cutline I've tried combining all steps in to one command and use the output of step 2 as my input:<br>gdalwarp -cutline clip.shp -cl clip -crop_to_cutline --config GDAL_CACHEMAX 1500 -wm 500 -multi -wo "USE_OPENCL=TRUE" -of GTiff -srcnodata "255 255 255" -dstnodata "255 255 255" -co tiled=yes -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -co "BIGTIFF=YES" -co "PROFILE=GEOTIFF" -co "COMPRESS=JPEG" -co "JPEG_QUALITY=80" -co "PHOTOMETRIC=YCBCR" --config GDAL_TIFF_INTERNAL_MASK YES -s_srs EPSG:28992 nodata.tif cropped.tif<br>
</div><div>This time I've used USE_OPENCL=TRUE but it doesn't seems to work. Most likely my version of gdal doesn't have openCL enabled.<br></div><div>I also increased the GDAL_CACHEMAX. I'm not sure what a good value would be. I'm running gdal v1.10-64Bit on WinVista64 with 8GB RAM.<br>
</div><div>While running above command I still have 3GB to spare.<br></div><div>The above step also results in a good tiff file of just 3.1GB, but without overviews so it loads very slowly.<br><br></div><div></div><div>I see a few enhancements. The first one is to use faster hardware. I've ordered a fast SSD, it will arrive in a few days.<br>
</div><div>Hopefully this will speed up my steps.<br></div><div>But the best optimization would be if step 1 and 2 could be combined and if I don't need to create a masked file to get a cutline.<br></div><div>The combined step only takes 6 hours so that is acceptable, but all goes wrong when I add overviews which introduces the artifacts again.<br>
</div><div>All steps mentioned here took me almost 4 weeks of computer processing time. And still the result is not what I need ;(<br><br></div><div>So if anybody can help me with adding overviews to my cropped.tif without loosing compression but also without introducing artifacts I would be very happy.<br>
</div><div><br></div><div>Sorry for the verbose mail. I really would like to read your thoughts about this.<br><br></div><div>Thanks,<br><br>Paul<br></div></div><br></div>