[mapserver-users] Processing JP2000 files

Sun Mar 30 07:13:16 PDT 2014

Hi Jukka,
This has certainly been entertaining.  I grabbed kakadu, and it could convert the images from jp2000 to jp2000 in about 5 seconds.  The speed was impressive.  Unfortunately, probably due to my limited jp2000 experience, kakadu sees the image as gray scale only.  When I export the image to another jp2000, my mapserver can process it just fine, but it's only in gray scale.  I then tried Global Mapper scripting.  It exports the jp2 image to jp2 in about 30 seconds, but retains the color.  Plus, the file size is reduce about a third.  Fortunately, I can run about 3 of these scripts at once on a machine, and still write them out ever 30 seconds or so, so I'm averaging about 1 file ever 10 seconds.  I've got 3 machines working on this so it should take about a week or two to convert the whole set.  If I try to open one of the new jp2 files produced by GM in kakadu, it complains that there is something illegal in the header, but then does display the image and it looks just fine.  I'm not sure what encoder the USDA is using to produce these files, but hopefully they get an update before we have to get more files from them.
Thanks,
James

________________________________________
From: Rahkonen Jukka  (Tike) [jukka.rahkonen at mmmtike.fi]
Sent: Friday, March 21, 2014 7:53 AM
To: EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE; 'mapserver-users at lists.osgeo.org'
Subject: Re: [mapserver-users] Processing JP2000 files

Hi James,

It has been a while since I have done any experiments with JPEG2000 and stopwatch but I happened to have a rather recent versions of Kakadu demo executables on my machine (version 7.3.2, the very latest is 7.3.3).  We do not have a license for the Kakadu speed pack, but even without it the conversion from lossless JPEG2000 into lossless GeoTIFF feels quite speedy. Here comes the facts:

Hardware is a Windows 7 laptop with Intel i7, 8 cores and 8 GB RAM.
Test image is 12000 x 12000 pixel 3-band aerial image, 443 MB as uncompressed, 170 MB as lossless JPEG2000
Here is the kdu_expand command and conversion log:

Kakadu_732\bin_x64\kdu_expand -i test_in.JP2  -o test_out.tif -cpu 0 -mem
Copying Geo  box info, size =     389
End-to-end CPU time = 6.317000 seconds (0.014623 us/sample)

Consumed 8 tile-part(s) from a total of 1 tile(s).
Consumed 174,371,002 codestream bytes (excluding any file format) = 9.687278
bits/pel. Processed using the multi-threaded environment, with 8 parallel threads of execution

Native Kakadu application used only 6.3 seconds CPU time for this task.   I also made a comparison with GDAL 64-bit with driver
JP2ECW (rov): ERDAS JPEG2000 (SDK 5.0)
End-to-end time including disk write was 21 second. Kadudu timing was 6.3 seconds but I took timing also to include disk write and it was still under 7 seconds. It may be that file system was still doing some delayed writing but the scale should be about right.
My test image was compressed as one tile and JP2OpenJPEG driver could not convert it.

Final comparison: gdal_translate from uncompressed tif to uncompressed tif: 3 seconds.

TIFF to TIFF with GDAL: 3 seconds
JP2 to TIFF with native Kakadu: 7 seconds
JP2 to TIFF with GDAL ECWJP2 (SDK 5.0): 21 seconds

Your Global Mapper timing with much smaller image was 3 minutes. You have 400000 files...

-Jukka Rahkonen-

> -----Alkuperäinen viesti-----
> Lähettäjä: EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE
> [mailto:james.evans.4 at us.af.mil]
> Lähetetty: 21. maaliskuuta 2014 14:22
> Vastaanottaja: Rahkonen Jukka (Tike); 'mapserver-users at lists.osgeo.org'
> Aihe: RE: [mapserver-users] Processing JP2000 files
>
> Hi Jukka,
> I have Global Mapper, and used it to do a batch convert to geotiff on a test set
> of images.  The converted files work fine with the OpenJPEG drive, but the size
> almost doubles and on my 10 year old pc at home it takes about 3 minutes per
> file to convert.  I will look at all the options you mentioned below.
> Thanks!
> James
>
>
>
> ________________________________________
> From: Rahkonen Jukka  (Tike) [jukka.rahkonen at mmmtike.fi]
> Sent: Friday, March 21, 2014 3:33 AM
> To: EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE; 'mapserver-
> users at lists.osgeo.org'
> Subject: Re: [mapserver-users] Processing JP2000 files
>
> Hi,
>
> From your other mail I concluded that your images are converted into JPEG2000
> with some Lizardtech product so it is not a surprise that MrSID/MrSIG JPEG2000
> driver is happy with the SOT markers. I do not know if it is a real error in file
> structure or only a different JPEG2000 dialect. JPEG2000 is very feature rich and
> complicated.
>
> It seems that OpenJPEG driver is not most happy with the JPEG2000 structure in
> your images. It could be happier if the data streams inside JP2 file were
> organized in some other way. There are tools (for example Kakadu
> kdu_transcode) for re-organizing JPEG2000 files without recompression and
> data loss but that would still mean a rewrite for all your images. OpenJPEG is as
> its best is rather fast and almost usable for production but it depends on how
> much load you wait for your server. However, myself I would not follow that
> route. I would consider two alternatives:
>
> - Acquire Mapserver + GDAL with some fast JPEG2000 driver:  JPKAK, JP2ECW, or
> JP2MrSID. Test with your images and take care of a license. Those SDKs are not
> free and you can't use them on server without paying.
> - Convert your images into GeoTIFF format (tiled, jpeg compressed, with
> compressed overviews). You need a fast JPEG200O driver also for that but for
> example JP2ECW license allows desktop use without payed license (check it!).
> Conversion will take some time because there are 1440 minutes in a day and you
> said you have 400000 images. With one computer it would take several months
> so I suggest to gather lots of computers and lots of USB drives so that you can
> read from one drive and write converted images into another. There is lots of
> computing power around nowadays and from any medium scaled office you can
> find 100 laptops which are idle for all nights and weekends. They could handle
> your data within a week or two. GeoTIFFs will take a bit more disk space mainly
> because of overviews (JPEG2000 streams are progressive and overviews are
> unnecessary) but they will be faster with your Mapserver and other GDAL based
> software like QGIS and with Geoserver too.
>
> -Jukka Rahkonen-
>
> EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE wrote:
>
> > Hi Jukka,
> >
> > The MS4W version says it is using Driver: JP2MrSID/MrSID JPEG2000, and
> > the GISInternals version is using Driver: JP2OpenJPEG/JPEG-2000 driver
> > based on OpenJPEG library.  I tried to do a gdal_translate on the same
> > file, using the each of the drivers.  They both translated the file
> > into a 512x512 jpg, but the JP2OpenJPEG driver listed over a 1000 of
> > these warnings before it stopped listing
> > warnings:
> > Warning 1: Empty SOT marker detected: Psot=12.
> >
> > So, I have 400,000 of these JP2000 images, equally 4.3TBs of data.
> > Should I consider a massive batch conversion to JPG or Geotiff, or is
> > there some way to get the JP2OpenJPEG driver to play nice with these files?
> >
> > Thanks,
> > James
> > ________________________________________
> > From: Rahkonen Jukka  (Tike) [jukka.rahkonen at mmmtike.fi]
> > Sent: Thursday, March 20, 2014 7:44 AM
> > To: EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE; mapserver-
> > users at lists.osgeo.org
> > Subject: VS: [mapserver-users] Processing JP2000 files
> >
> > Hi,
> >
> > Are your JPEG2000 images the same?  SOT marker is JPEG2000 stuff and
> > it means "start of tilepart".  If images are the same then the message
> > means that something has changed in GDAL part. GDAL has several
> > JPEG200 drivers and they all behave in a bit different way. Check
> > which driver you have in your old system and which one you have now with
> "gdalinfo --formats".
> > I warn that from the free alternatives only OpenJPEG driver is
> > something that can be considered for real use.
> >
> > -Jukka Rahkonen-
> >
> >
> >
> > James_in_Utah wrote:
> > >
> > > There must have been something wrong with my shapefile, because I
> > > tried this at home and I was able to get the same data to work.  Now
> > > the problem is that the image I see on my client is all washed out,
> > > not a variant RGB image but mostly brownish with weak greens.  None
> > > of the reds or blues are making it through.  Plus, where the tiles
> > > overlap I can see the edges are blended from the overlapping tiles.
> > > This wasn't the case with the older GDAL from MS4W.  Has something
> > changed?  Is there some directive in the layer that I'm missing?
> > > Thanks,
> > > James
> > >
> > >
> > >
> > >
> > > --
> > > View this message in context: http://osgeo-
> > > org.1560.x6.nabble.com/Processing-JP2000-files-tp5129997p5130136.htm
> > > l Sent from the Mapserver - User mailing list archive at Nabble.com.
> > > _______________________________________________
> > > mapserver-users mailing list
> > > mapserver-users at lists.osgeo.org
> > > http://lists.osgeo.org/mailman/listinfo/mapserver-users