[mapserver-users] Processing JP2000 files

Fri Mar 21 05:21:30 PDT 2014

Hi Jukka,
I have Global Mapper, and used it to do a batch convert to geotiff on a test set of images.  The converted files work fine with the OpenJPEG drive, but the size almost doubles and on my 10 year old pc at home it takes about 3 minutes per file to convert.  I will look at all the options you mentioned below.
Thanks!
James

________________________________________
From: Rahkonen Jukka  (Tike) [jukka.rahkonen at mmmtike.fi]
Sent: Friday, March 21, 2014 3:33 AM
To: EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE; 'mapserver-users at lists.osgeo.org'
Subject: Re: [mapserver-users] Processing JP2000 files

Hi,

>From your other mail I concluded that your images are converted into JPEG2000 with some Lizardtech product so it is not a surprise that MrSID/MrSIG JPEG2000 driver is happy with the SOT markers. I do not know if it is a real error in file structure or only a different JPEG2000 dialect. JPEG2000 is very feature rich and complicated.

It seems that OpenJPEG driver is not most happy with the JPEG2000 structure in your images. It could be happier if the data streams inside JP2 file were organized in some other way. There are tools (for example Kakadu kdu_transcode) for re-organizing JPEG2000 files without recompression and data loss but that would still mean a rewrite for all your images. OpenJPEG is as its best is rather fast and almost usable for production but it depends on how much load you wait for your server. However, myself I would not follow that route. I would consider two alternatives:

- Acquire Mapserver + GDAL with some fast JPEG2000 driver:  JPKAK, JP2ECW, or JP2MrSID. Test with your images and take care of a license. Those SDKs are not free and you can't use them on server without paying.
- Convert your images into GeoTIFF format (tiled, jpeg compressed, with compressed overviews). You need a fast JPEG200O driver also for that but for example JP2ECW license allows desktop use without payed license (check it!).
Conversion will take some time because there are 1440 minutes in a day and you said you have 400000 images. With one computer it would take several months so I suggest to gather lots of computers and lots of USB drives so that you can read from one drive and write converted images into another. There is lots of computing power around nowadays and from any medium scaled office you can find 100 laptops which are idle for all nights and weekends. They could handle your data within a week or two. GeoTIFFs will take a bit more disk space mainly because of overviews (JPEG2000 streams are progressive and overviews are unnecessary) but they will be faster with your Mapserver and other GDAL based software like QGIS and with Geoserver too.

-Jukka Rahkonen-

EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE wrote:

> Hi Jukka,
>
> The MS4W version says it is using Driver: JP2MrSID/MrSID JPEG2000, and the
> GISInternals version is using Driver: JP2OpenJPEG/JPEG-2000 driver based on
> OpenJPEG library.  I tried to do a gdal_translate on the same file, using the each
> of the drivers.  They both translated the file into a 512x512 jpg, but the
> JP2OpenJPEG driver listed over a 1000 of these warnings before it stopped listing
> warnings:
> Warning 1: Empty SOT marker detected: Psot=12.
>
> So, I have 400,000 of these JP2000 images, equally 4.3TBs of data.  Should I
> consider a massive batch conversion to JPG or Geotiff, or is there some way to
> get the JP2OpenJPEG driver to play nice with these files?
>
> Thanks,
> James
> ________________________________________
> From: Rahkonen Jukka  (Tike) [jukka.rahkonen at mmmtike.fi]
> Sent: Thursday, March 20, 2014 7:44 AM
> To: EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE; mapserver-
> users at lists.osgeo.org
> Subject: VS: [mapserver-users] Processing JP2000 files
>
> Hi,
>
> Are your JPEG2000 images the same?  SOT marker is JPEG2000 stuff and it
> means "start of tilepart".  If images are the same then the message means that
> something has changed in GDAL part. GDAL has several JPEG200 drivers and they
> all behave in a bit different way. Check which driver you have in your old system
> and which one you have now with "gdalinfo --formats".
> I warn that from the free alternatives only OpenJPEG driver is something that
> can be considered for real use.
>
> -Jukka Rahkonen-
>
>
>
> James_in_Utah wrote:
> >
> > There must have been something wrong with my shapefile, because I
> > tried this at home and I was able to get the same data to work.  Now
> > the problem is that the image I see on my client is all washed out,
> > not a variant RGB image but mostly brownish with weak greens.  None of
> > the reds or blues are making it through.  Plus, where the tiles
> > overlap I can see the edges are blended from the overlapping tiles.
> > This wasn't the case with the older GDAL from MS4W.  Has something
> changed?  Is there some directive in the layer that I'm missing?
> > Thanks,
> > James
> >
> >
> >
> >
> > --
> > View this message in context: http://osgeo-
> > org.1560.x6.nabble.com/Processing-JP2000-files-tp5129997p5130136.html
> > Sent from the Mapserver - User mailing list archive at Nabble.com.
> > _______________________________________________
> > mapserver-users mailing list
> > mapserver-users at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/mapserver-users