[OSGeo-Discuss] Open File FormatsandProprietaryAlgorithms[SEC=UNCLASSIFIED]
Landon Blake
lblake at ksninc.com
Fri Aug 21 09:57:19 PDT 2009
Wow. Whoddu thunk we'd spend so much time and energy trying to make big
pictures smaller.
Landon
Office Phone Number: (209) 946-0268
Cell Phone Number: (209) 992-0658
-----Original Message-----
From: discuss-bounces at lists.osgeo.org
[mailto:discuss-bounces at lists.osgeo.org] On Behalf Of Michael P. Gerlek
Sent: Friday, August 21, 2009 9:55 AM
To: OSGeo Discussions
Subject: RE: [OSGeo-Discuss] Open File
FormatsandProprietaryAlgorithms[SEC=UNCLASSIFIED]
Not a stupid question -- but it doesn't work that way. The artifacts
are due to the wavelet processing of the pixels near the tile
boundaries, and the boundaries have to be treated reflectively within
their individual tiles (in order to keep each tile separate), which
means you can sometimes "see" where those boundaries are. "Overlapping"
doesn't help because the wavelet footprint spans a large width, in order
to handle the lower-resolution scales. Which in turn means you need to
be able "reach" far away parts of the image at various (some might say
"arbitrary") stages in the wavelet pipeline.
Just trust me, it is a nontrivial problem to solve. Brighter minds than
ours have spent a lot of energy on this problem -- a literature search
would reveal a number of PhD theses and patents.
-mpg
-----Original Message-----
From: discuss-bounces at lists.osgeo.org
[mailto:discuss-bounces at lists.osgeo.org] On Behalf Of Landon Blake
Sent: Friday, August 21, 2009 10:45 AM
To: OSGeo Discussions
Subject: RE: [OSGeo-Discuss] Open File Formats
andProprietaryAlgorithms[SEC=UNCLASSIFIED]
MPG wrote: "Tiling" essentially means you can take a large file and
compress pieces of it independently. This avoids having to deal with
the large memory footprint issues, but it can also lead to seam-line
artifacts under certain conditions. Ideally, one would prefer to have
the option of compressing large images without resorting to using
tiles."
This is probably a stupid question, since I know absolutely nothing
about image compression, but couldn't you overlap the tiles slightly to
avoid the seam lines?
This would obviously result in a slightly larger file size because some
pixels would be compressed twice. But that might be OK if you were
trying to compress a huge image.
What about reading chunks of the image off disk, instead of trying to
put the whole image in memory? This would be slower, but might make an
impossible task possible.
We run into this problem with vector datasets to. Some datasets are just
to stinking BIG. One of my tasks for OpenJUMP is to write a core module
that displays vector data accessed directly from disk, instead of from
memory. This will be slower, but it is better than crashing the program
because there isn't enough RAM.
Things must be more complicated than can be described in an e-mail,
because we've got people a lot smarter than me working on these
problems. I am just curious. (I tried reading about wavelet compression
on Wikipedia yesterday and quickly got a headache.) :]
Landon
Office Phone Number: (209) 946-0268
Cell Phone Number: (209) 992-0658
-----Original Message-----
From: discuss-bounces at lists.osgeo.org
[mailto:discuss-bounces at lists.osgeo.org] On Behalf Of Michael P. Gerlek
Sent: Friday, August 21, 2009 9:36 AM
To: OSGeo Discussions
Subject: RE: [OSGeo-Discuss] Open File Formats
andProprietaryAlgorithms[SEC=UNCLASSIFIED]
"Tiling" essentially means you can take a large file and compress pieces
of it independently. This avoids having to deal with the large memory
footprint issues, but it can also lead to seam-line artifacts under
certain conditions. Ideally, one would prefer to have the option of
compressing large images without resorting to using tiles.
Note too that, in addition to the large image issue, many of the JP2
implementations out there are either not fully compliant or are not
tuned for performance. A viable solution would need both of these.
-mpg
-----Original Message-----
From: discuss-bounces at lists.osgeo.org
[mailto:discuss-bounces at lists.osgeo.org] On Behalf Of Chris Puttick
Sent: Friday, August 21, 2009 9:37 AM
To: OSGeo Discussions
Subject: Re: [OSGeo-Discuss] Open File Formats and
ProprietaryAlgorithms[SEC=UNCLASSIFIED]
Well, according to this page: http://jpeg2000.epfl.ch/ v.5.1, courtesy
in part Eastman Kodak, provides "complete JP2 support at the decoding
side" - not sure whether that covers the tiling or other geo needs, but
doesn't it sound worth investigating?
Chris
----- "Christopher Schmidt" <crschmidt at crschmidt.net> wrote:
> On Fri, Aug 21, 2009 at 08:27:13AM -0700, Landon Blake wrote:
> > Thanks for the information Michael. I am downloading Opticks right
> now.
> > :]
> >
> > I also found this Java library for JP2, thought I'm not sure how
> > complete/up-to-date it is:
> >
> > http://jj2000.epfl.ch/
> >
> > Maybe we need a JPEG 2000 page on the OSGeo wiki.
>
> Note that "JPEG 2000 support" is different from "JPEG 2000 support
> which
> works on geo-sized images." The tiling (or 'paging'? as Michael calls
> it) support that's supposed to be provided by OpenJPEG2000 has been
> coming 'real soon now' for about 18 months now from my uneducated
> observations, and until it's there, most tools using OpenJPEG for JP2s
> are
> going to suffering under much the same limitations.
>
> -- Chris
>
> > Landon
> > Office Phone Number: (209) 946-0268
> > Cell Phone Number: (209) 992-0658
> >
> >
> >
> > -----Original Message-----
> > From: discuss-bounces at lists.osgeo.org
> > [mailto:discuss-bounces at lists.osgeo.org] On Behalf Of Considine,
> Michael
> > Sent: Friday, August 21, 2009 8:09 AM
> > To: OSGeo Discussions
> > Subject: RE: [OSGeo-Discuss] Open File Formats and
> > ProprietaryAlgorithms[SEC=UNCLASSIFIED]
> >
> > All,
> >
> > Opticks is an open source remote sensing application and
> development
> > framework. We recently started the process to add JPEG 2000 support
> to
> > our framework. We picked OpenJpeg to add JPEG 2000 support to our
> > application. They are also open source. We currently support
> importing
> > JPEG 2000 files but we are currently limited to the 4GB memory size
> > after decoding.
> >
> > Our plan is to continue development and to upgrade to OpenJpeg 2.0
> once
> > they have a stable release. That will allow Opticks to use a pager
> to
> > display and support much larger files.
> >
> > Michael Considine
> >
> > -----Original Message-----
> > From: discuss-bounces at lists.osgeo.org
> > [mailto:discuss-bounces at lists.osgeo.org] On Behalf Of Bruce
> Bannerman
> > Sent: Thursday, August 20, 2009 8:15 PM
> > To: 'OSGeo Discussions'
> > Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary
> > Algorithms[SEC=UNCLASSIFIED]
> >
> >
> > IMO:
> >
> >
> > Just another thought on this issue (though we do seem to be
> recycling
> > arguments over the years...):
> >
> >
> > Assuming that I have a very large archive of spatial data, be it
> imagery
> > or any other spatial format and that I store my data in a variety
> of
> > proprietary formats:
> >
> >
> > In ten years from now, can I be sure that:
> >
> > - the company that created, understands, and holds the IP in the
> > data format will still be around?
> >
> > - there will still be software that runs on the then current
> > operating environment, that can read and 'fully exploit' the data
> > in the proprietary standard?
> >
> > - that this future software will work seamlessly with my then
> current
> > spatial environment?
> >
> > - if all of the above risks prove to eventuate, can I be sure that
> I'll
> > be able to salvage my data into another format, retaining its
> complete
> >
> > semantic context?
> >
> >
> > IMO, it is a high risk proposition to lock public (or private)
> archives
> > away in proprietary data formats. It makes more sense to use open
> > standards and formats that are publically available.
> >
> >
> >
> > Bruce Bannerman
> >
> >
> >
> >
> >
> > > -----Original Message-----
> > > From: discuss-bounces at lists.osgeo.org
> > > [mailto:discuss-bounces at lists.osgeo.org] On Behalf Of Michael
> > > P. Gerlek
> > > Sent: Friday, 21 August 2009 6:55 AM
> > > To: OSGeo Discussions
> > > Subject: RE: [OSGeo-Discuss] Open File Formats and
> > > Proprietary Algorithms
> > >
> > > Some clarifications:
> > >
> > >
> > >
> > > - MrSID has both lossy and lossless modes
> > >
> > > - MrSID is not fractal based; it uses wavelets (and
> > > arithmetic encoding)
> > >
> > > - you can't copyright algorithms; the MrSID source code
> > > certainly is, however
> > >
> > > - MrSID relies on a number of patents, not all of which are
> > > owned by LizardTech
> > >
> > > - reading MrSID does not require any fees; we have libraries
> > > you can download, although they are not open source
> > >
> > >
> > >
> > > That said, some editorial comments (although I'm now wishing
> > > I hadn't been so quick to rise to Landon's bait :-)
> > >
> > >
> > >
> > > - Some of you know the history of trying to open source
> > > MrSID; I won't go into that here, except to say that
> > > LizardTech doesn't own all of the required IP needed to make
> > > that happen.
> > >
> > > - If we are speaking of the NAIP data, then no, it is not
> > > exclusively available in MrSID format; it is also shipped as
> GeoTIFFs.
> > >
> > > - JPEG 2000 is a very robust open standard alternative to
> > > MrSID, and a number of players already support it (including
> > > LizardTech), but not enough to make it viable for certain
> > > domains like NAIP.
> > >
> > > - some of you also know the history on open JP2 support:
> > > there is today no open source implementation of JP2 that is
> > > suitable for geo work. Alas.
> > >
> > >
> > >
> > > -mpg
> > >
> > >
> > >
> > >
> > >
> > > From: discuss-bounces at lists.osgeo.org
> > > [mailto:discuss-bounces at lists.osgeo.org] On Behalf Of Eric Wolf
> > > Sent: Thursday, August 20, 2009 2:15 PM
> > > To: OSGeo Discussions
> > > Subject: Re: [OSGeo-Discuss] Open File Formats and
> > > Proprietary Algorithms
> > >
> > >
> > >
> > > The MRSID format is a very special case - and perhaps an
> > > opportunity for a new FOSS file format. MRSID is a lossless,
> > > fractal-based, multi-scale raster compression format.
> > > LizardTech has the algorithms to encode and decode MRSID
> > > locked up in copyrights, and I believe, patents. Even
> > > companies like ESRI shell out big bucks to LizardTech to be
> > > able to read and write the MRSID format.
> > >
> > >
> > >
> > > I guess I missed the context of the discussion. Is the
> > > government releasing certain data exclusively in this format?
> > > If so, I think the argument can be made against this
> > > practice. The different in compression between MRSID and
> > > gziped TIFFs isn't really that great in this day of cheap
> > > disks and fat pipes.
> > >
> > >
> > >
> > > -Eric
> > >
> > >
> > > -=--=---=----=----=---=--=-=--=---=----=---=--=-=-
> > > Eric B. Wolf New! 720-334-7734
> > > USGS Geographer
> > > Center of Excellence in GIScience
> > > PhD Student
> > > CU-Boulder - Geography
> > >
> > >
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > Discuss mailing list
> > Discuss at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/discuss
> >
> >
> >
> > This message and any enclosures are intended only for the
> addressee.
> > Please
> > notify the sender by email if you are not the intended recipient.
> If
> > you are
> > not the intended recipient, you may not use, copy, disclose, or
> > distribute this
> > message or its contents or enclosures to any other person and any
> such
> > actions
> > may be unlawful. Ball reserves the right to monitor and review all
> > messages
> > and enclosures sent to or from this email address.
> > _______________________________________________
> > Discuss mailing list
> > Discuss at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/discuss
> >
> >
> > Warning:
> > Information provided via electronic media is not guaranteed against
> defects including translation and transmission errors. If the reader
> is not the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication is
> strictly prohibited. If you have received this information in error,
> please notify the sender immediately.
> > _______________________________________________
> > Discuss mailing list
> > Discuss at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/discuss
>
> --
> Christopher Schmidt
> Web Developer
> _______________________________________________
> Discuss mailing list
> Discuss at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/discuss
------
Files attached to this email may be in ISO 26300 format (OASIS Open
Document Format). If you have difficulty opening them, please visit
http://iso26300.info for more information.
_______________________________________________
Discuss mailing list
Discuss at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss
Warning:
Information provided via electronic media is not guaranteed against
defects including translation and transmission errors. If the reader is
not the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication is strictly
prohibited. If you have received this information in error, please
notify the sender immediately.
_______________________________________________
Discuss mailing list
Discuss at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss
_______________________________________________
Discuss mailing list
Discuss at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/discuss
More information about the Discuss
mailing list