[Gdal-dev] warp and translate problems

[Chris Hodgson]

> You can actually use gdal_translate to subset to something like jpeg2000,
> but what happens inside gdal_translate is that it makes up a "virtual 
> dataset"
> with the desired characteristics that fetches image data from the real
> source file as needed.

So I should be able to use the "-a_srs" option on gdal_translate to do a 
"one-pass" conversion from TIF/UTM to JP2/BCALB - even though I can't 
use the "-of" option in  gdalwarp to do the same thing - just due to the 
different architectures of the programs? I'll have to try it. Though, it 
may not help considering the answer to question #3 :-(

I don't really mind the intermediate step, I just thought that if I can 
read the data from disk fewer times, it should speed things up.

>> 2) I'm starting with .tif and .tfw files, then gdalwarp gives me 
>> single geotiff file. then when I use gdal_translate to convert to jp2 
>> format, I don't have any georeferencing (no wld file)... do I need to 
>> compile with jasper as well so that I can output geojp2 files, or is 
>> there some way to tell gdal_translate to output a .wld file for the 
>> output file?
>
> The JP2KAK driver writes out GeoJP2 (tm) files as long as the input is
> georeferenced.  If you really need .wld files for some reason you could
> write a little python script to do it.  I don't believe the JP2KAK driver
> supports generating .wld files directly.

I think the real problem was that gdaltindex wasn't recognizing the 
GeoJP2 georeferencing - it complained that my jp2 files weren't 
georeferenced. However, they loaded just fine in ArcMap - wow, GeoJP2 
must be a real standard if even ESRI recognizes it :-)

>> 3) My intent is to build a tileindex of the output files and serve 
>> them up using mapserver. I'm worried about the black "collar" that 
>> I'm left with around the images after I've reprojected them - will 
>> mapserver be able to deal with the fact that these all overlap with 
>> data from the neighboring tile? Can mapserver deal with tileindexes 
>> that have overlaps? How about tiles that aren't square in the given 
>> coordinate system - if I use the grid from the original orthophoto 
>> files, will it use those boundaries to select which tile to get the 
>> data from? That would be the optimal solution, if it works that way. 
>> Tell me if this is a question better posted to the MapServer list but 
>> I know it is gdal doing most of this work anyway.
>
> Hmm, this will indeed be a problem.  MapServer doesn't mind overlapping
> images, but it is just going to draw them in some arbitrary order and the
> black stuff will get written over valid imagery in some cases.  Normally
> I would suggest using the MapServer OFFSITE keyword to indicate that the
> black should be ignored (treated as transparent), but with a lossy
> compression format like JPEG2000 you will find the black pixels vary a
> bit from 0/0/0 and thus you won't be able to use OFFSET effectively.

Yah, this is what I figured.

> My suggestion would be to generate output tiles that are not related 1:1
> to your input files.  Instead, pick a tile size and then fill it 
> completely
> by gdalwarping multiple images into it.

I was contemplating exactly this over lunch...

> My understanding is you have 2TB of imagery.  If you broke your region
> down into 100000 x 100000 chunks (roughly 30GB each assuming RGB 
> uncompressed)
> then you could just compress each of those chunks.  To warp the input
> images into the you would need to use gdalwarp to exactly create the
> correct output tiles with the -tr and -ts switches to gdalwarp.  Then 
> just
> warp each file that intersects the region into the working file (or all
> the files if it is to much trouble to work out which ones intersect).
> This should give you (big) resulting files with no black areas with the
> possible exception of areas that have no data.

An annoyance is that some of our orthos are at different resolutions, 
some are in colour (most B&W), and some are already reprojected to 
albers... this can all be overcome, but causes additional trouble... 
this was supposed to be a simple task, it has become a nightmare (and 
that never happens...)

> Another possibility would be to work with me to add alpha band support
> to gdalwarp.  Then you could warp each file to an RGBA file with a=0
> for pixels that are not set, and a=255 for pixels that are set.
> MapServer knows how to utilize per-pixel alpha information when 
> available.

This actually seems like the best solution, since we want to be able to 
add new orthos easily (without having to rebuild the tiles for that 
area...), and we're not really worried about edge artifacts, or which 
ortho gets used on slightly-overlapping edge cases...

Let me know if I'm missing something, but this doesn't seem too 
difficult... Let me know off-list how long this would take you to do, 
and how soon you could do it...

Thanks,
Chris