[mapserver-users] Processing JP2000 files

Mon Mar 31 01:05:47 PDT 2014

Hi,

I can also have a look if you can put original, gray scale creater by Kakadu and the one created with GM somewhere for downloading. Private message is OK, I can mail my findings back to the list.

-Jukka Rahkonen-

EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE wrote:
> 
> Hi Jukka,
> This has certainly been entertaining.  I grabbed kakadu, and it could convert the
> images from jp2000 to jp2000 in about 5 seconds.  The speed was impressive.
> Unfortunately, probably due to my limited jp2000 experience, kakadu sees the
> image as gray scale only.  When I export the image to another jp2000, my
> mapserver can process it just fine, but it's only in gray scale.  I then tried Global
> Mapper scripting.  It exports the jp2 image to jp2 in about 30 seconds, but
> retains the color.  Plus, the file size is reduce about a third.  Fortunately, I can
> run about 3 of these scripts at once on a machine, and still write them out ever
> 30 seconds or so, so I'm averaging about 1 file ever 10 seconds.  I've got 3
> machines working on this so it should take about a week or two to convert the
> whole set.  If I try to open one of the new jp2 files produced by GM in kakadu, it
> complains that there is something illegal in the header, but then does display the
> image and it looks just fine.  I'm not sure what encoder the USDA is using to
> produce these files, but hopefully they get an update before we have to get
> more files from them.
> Thanks,
> James
> 
> 
> 
> ________________________________________
> From: Rahkonen Jukka  (Tike) [jukka.rahkonen at mmmtike.fi]
> Sent: Friday, March 21, 2014 7:53 AM
> To: EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE; 'mapserver-
> users at lists.osgeo.org'
> Subject: Re: [mapserver-users] Processing JP2000 files
> 
> Hi James,
> 
> It has been a while since I have done any experiments with JPEG2000 and
> stopwatch but I happened to have a rather recent versions of Kakadu demo
> executables on my machine (version 7.3.2, the very latest is 7.3.3).  We do not
> have a license for the Kakadu speed pack, but even without it the conversion
> from lossless JPEG2000 into lossless GeoTIFF feels quite speedy. Here comes the
> facts:
> 
> Hardware is a Windows 7 laptop with Intel i7, 8 cores and 8 GB RAM.
> Test image is 12000 x 12000 pixel 3-band aerial image, 443 MB as
> uncompressed, 170 MB as lossless JPEG2000 Here is the kdu_expand command
> and conversion log:
> 
> Kakadu_732\bin_x64\kdu_expand -i test_in.JP2  -o test_out.tif -cpu 0 -mem
> Copying Geo  box info, size =     389
> End-to-end CPU time = 6.317000 seconds (0.014623 us/sample)
> 
> Consumed 8 tile-part(s) from a total of 1 tile(s).
> Consumed 174,371,002 codestream bytes (excluding any file format) = 9.687278
> bits/pel. Processed using the multi-threaded environment, with 8 parallel
> threads of execution
> 
> Native Kakadu application used only 6.3 seconds CPU time for this task.   I also
> made a comparison with GDAL 64-bit with driver
> JP2ECW (rov): ERDAS JPEG2000 (SDK 5.0)
> End-to-end time including disk write was 21 second. Kadudu timing was 6.3
> seconds but I took timing also to include disk write and it was still under 7
> seconds. It may be that file system was still doing some delayed writing but the
> scale should be about right.
> My test image was compressed as one tile and JP2OpenJPEG driver could not
> convert it.
> 
> Final comparison: gdal_translate from uncompressed tif to uncompressed tif: 3
> seconds.
> 
> TIFF to TIFF with GDAL: 3 seconds
> JP2 to TIFF with native Kakadu: 7 seconds
> JP2 to TIFF with GDAL ECWJP2 (SDK 5.0): 21 seconds
> 
> Your Global Mapper timing with much smaller image was 3 minutes. You have
> 400000 files...
> 
> -Jukka Rahkonen-
> 
> 
> 
> 
> > -----Alkuperäinen viesti-----
> > Lähettäjä: EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE
> > [mailto:james.evans.4 at us.af.mil]
> > Lähetetty: 21. maaliskuuta 2014 14:22
> > Vastaanottaja: Rahkonen Jukka (Tike); 'mapserver-users at lists.osgeo.org'
> > Aihe: RE: [mapserver-users] Processing JP2000 files
> >
> > Hi Jukka,
> > I have Global Mapper, and used it to do a batch convert to geotiff on
> > a test set of images.  The converted files work fine with the OpenJPEG
> > drive, but the size almost doubles and on my 10 year old pc at home it
> > takes about 3 minutes per file to convert.  I will look at all the options you
> mentioned below.
> > Thanks!
> > James
> >
> >
> >
> > ________________________________________
> > From: Rahkonen Jukka  (Tike) [jukka.rahkonen at mmmtike.fi]
> > Sent: Friday, March 21, 2014 3:33 AM
> > To: EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE; 'mapserver-
> > users at lists.osgeo.org'
> > Subject: Re: [mapserver-users] Processing JP2000 files
> >
> > Hi,
> >
> > From your other mail I concluded that your images are converted into
> > JPEG2000 with some Lizardtech product so it is not a surprise that
> > MrSID/MrSIG JPEG2000 driver is happy with the SOT markers. I do not
> > know if it is a real error in file structure or only a different
> > JPEG2000 dialect. JPEG2000 is very feature rich and complicated.
> >
> > It seems that OpenJPEG driver is not most happy with the JPEG2000
> > structure in your images. It could be happier if the data streams
> > inside JP2 file were organized in some other way. There are tools (for
> > example Kakadu
> > kdu_transcode) for re-organizing JPEG2000 files without recompression
> > and data loss but that would still mean a rewrite for all your images.
> > OpenJPEG is as its best is rather fast and almost usable for
> > production but it depends on how much load you wait for your server.
> > However, myself I would not follow that route. I would consider two
> alternatives:
> >
> > - Acquire Mapserver + GDAL with some fast JPEG2000 driver:  JPKAK,
> > JP2ECW, or JP2MrSID. Test with your images and take care of a license.
> > Those SDKs are not free and you can't use them on server without paying.
> > - Convert your images into GeoTIFF format (tiled, jpeg compressed,
> > with compressed overviews). You need a fast JPEG200O driver also for
> > that but for example JP2ECW license allows desktop use without payed license
> (check it!).
> > Conversion will take some time because there are 1440 minutes in a day
> > and you said you have 400000 images. With one computer it would take
> > several months so I suggest to gather lots of computers and lots of
> > USB drives so that you can read from one drive and write converted
> > images into another. There is lots of computing power around nowadays
> > and from any medium scaled office you can find 100 laptops which are
> > idle for all nights and weekends. They could handle your data within a
> > week or two. GeoTIFFs will take a bit more disk space mainly because
> > of overviews (JPEG2000 streams are progressive and overviews are
> > unnecessary) but they will be faster with your Mapserver and other
> > GDAL based software like QGIS and with Geoserver too.
> >
> > -Jukka Rahkonen-
> >
> > EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE wrote:
> >
> > > Hi Jukka,
> > >
> > > The MS4W version says it is using Driver: JP2MrSID/MrSID JPEG2000,
> > > and the GISInternals version is using Driver: JP2OpenJPEG/JPEG-2000
> > > driver based on OpenJPEG library.  I tried to do a gdal_translate on
> > > the same file, using the each of the drivers.  They both translated
> > > the file into a 512x512 jpg, but the JP2OpenJPEG driver listed over
> > > a 1000 of these warnings before it stopped listing
> > > warnings:
> > > Warning 1: Empty SOT marker detected: Psot=12.
> > >
> > > So, I have 400,000 of these JP2000 images, equally 4.3TBs of data.
> > > Should I consider a massive batch conversion to JPG or Geotiff, or
> > > is there some way to get the JP2OpenJPEG driver to play nice with these
> files?
> > >
> > > Thanks,
> > > James
> > > ________________________________________
> > > From: Rahkonen Jukka  (Tike) [jukka.rahkonen at mmmtike.fi]
> > > Sent: Thursday, March 20, 2014 7:44 AM
> > > To: EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE; mapserver-
> > > users at lists.osgeo.org
> > > Subject: VS: [mapserver-users] Processing JP2000 files
> > >
> > > Hi,
> > >
> > > Are your JPEG2000 images the same?  SOT marker is JPEG2000 stuff and
> > > it means "start of tilepart".  If images are the same then the
> > > message means that something has changed in GDAL part. GDAL has
> > > several
> > > JPEG200 drivers and they all behave in a bit different way. Check
> > > which driver you have in your old system and which one you have now
> > > with
> > "gdalinfo --formats".
> > > I warn that from the free alternatives only OpenJPEG driver is
> > > something that can be considered for real use.
> > >
> > > -Jukka Rahkonen-
> > >
> > >
> > >
> > > James_in_Utah wrote:
> > > >
> > > > There must have been something wrong with my shapefile, because I
> > > > tried this at home and I was able to get the same data to work.
> > > > Now the problem is that the image I see on my client is all washed
> > > > out, not a variant RGB image but mostly brownish with weak greens.
> > > > None of the reds or blues are making it through.  Plus, where the
> > > > tiles overlap I can see the edges are blended from the overlapping tiles.
> > > > This wasn't the case with the older GDAL from MS4W.  Has something
> > > changed?  Is there some directive in the layer that I'm missing?
> > > > Thanks,
> > > > James
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > View this message in context: http://osgeo-
> > > > org.1560.x6.nabble.com/Processing-JP2000-files-tp5129997p5130136.h
> > > > tm l Sent from the Mapserver - User mailing list archive at
> > > > Nabble.com.
> > > > _______________________________________________
> > > > mapserver-users mailing list
> > > > mapserver-users at lists.osgeo.org
> > > > http://lists.osgeo.org/mailman/listinfo/mapserver-users