[gdal-dev] Misc. subjects : OSGeo Vienna code sprint, release plans, GDAL 2.0

Thu Apr 3 07:22:58 PDT 2014

On Tue, Apr 1, 2014 at 5:57 PM, Kyle Shannon <kyle at pobox.com> wrote:

> On Tue, Apr 1, 2014 at 12:36 PM, Etienne Tourigny
> <etourigny.dev at gmail.com> wrote:
> > The 2 objections I have with json are :
> >
> > - it is so verbose that editing by hand is not as easy as .csv
> > - the xml tags make file size much larger than .csv files, unless they
> would
> > be stored in a compressed file (gzip)
> >
> > On the other hand, who messes with theses files on a regular basis
> anyway?
> >
> > It seems like a nice idea. In what ways would it be better than Even's
> > suggestion to use sqlite?
> >
> >
> > On Mon, Mar 31, 2014 at 4:37 PM, Dmitriy Baryshnikov <
> bishop.dev at gmail.com>
> > wrote:
> >>
> >> Hi,
> >>
> >> I think the JSON format is good for metadata storing and representation.
> >> JSON support to store string, digits, bool, dates, etc. Write such data
> and
> >> read such data from files.
> >> But need good library or set of classes to work with it in memory
> >> representation.
> >> For example I like  wxJSON - it support to get value by name or iterate
> >> values by index.
> >>
> >> int majorVer = root["wxWidgets"]["Version"]["Major"].AsInt();
> >>
> >>
> >> root["wxWidgets"]["Authors"][0] = "J. Smart";
> >> root["wxWidgets"]["Authors"][1] = "V. Zeitling";
> >> root["wxWidgets"]["Authors"][2] = "R. Roebling";
> >>
> >>
> >> It consist only 3 files (json reader, json writer and json value).
> >>
> >> It seems to me that json-c library is more complicated. Unfortunately
> >> wxJSON cannot be used in GDAL as it have bindings to wxWidgets library,
> but
> >> the approach is interesting. By the way wxWidgets have more democratic
> >> license instead of qt.
> >>
> >> Best regards,
> >>     Dmitry
> >>
> >> 31.03.2014 23:03, Even Rouault пишет:
> >>
> >> Hi Etienne,
> >>
> >> Thanks for your ideas.
> >>
> >> Hi all,
> >>
> >> I have a few suggestions for gdal 2.0, based on my personal experience
> in
> >> learning to use, enhance and maintain gdal/ogr code.
> >>
> >> - replace cpl/csl/string/xml code with a mainstream, modern
> cross-platform
> >> toolkit such as QT, boost, etc.
> >>
> >> QT is certainly a dependency we wouldn't want to draw. Too big for some
> >> embededded usage, and it would make GDAL to be practially bound by the
> >> LGPL.
> >> I guess standard C++ libraries classes, or perhaps boost, should do the
> >> job
> >> for what you mention below.
> >>
> >> While cpl/csl classes are robust and "do the job", they are not well
> >> documented and not very intuitive for a new gdal coder. This is from my
> >> personal experience, some may not agree.
> >> They are also not used outside gdal, as such do not benefit from
> >> enhancements as other toolkits.
> >>
> >> Well, at least, MapServer uses a few CPL functions : CPL minixml,
> >> CPLMalloc/CPLFree, CPLReadDir, CPLFormFilename, CPLGetFilename,
> >> CSLInsertString, etc..
> >>
> >> The drawback is that some data/methods would not be easily available in
> c
> >> and other bindings. Also it might not be available for all platforms.
> >> Existing code would still be able to use cpl/csl code but be deprecated
> >> until a given release.
> >>
> >>
> >> - upgrade/migrate gdal support file (files in .csv format such as
> gcs.csv)
> >> reading to use containers (e.g. hash maps) instead of reading the
> relevant
> >> .csv files every time
> >>
> >> Current reading of gdal support files is sub-optimal as any query for
> >> support data requires reading the relevant support file(s). It would be
> >> more efficient to read these once and cache into a container (such as
> hash
> >> map). This could be done using a cross-platform toolkit mentioned
> earlier.
> >>
> >> Another option would be to use a .sqlite database with proper indices.
> >> SQLite
> >> would become a runtime requirement, but I don't think that would be a
> big
> >> problem.
> >>
> >> - modify metadata treatment to be able to store/retrieve data in formats
> >> other than strings (e.g. floats, doubles, boolean), and query the actual
> >> type of a given metadata item.
> >>
> >> Currently there is no way to know if a given metadata item represents an
> >> integer, float, double (or string).
> >>
> >> In the netcdf driver, I overcame this limitation in 2 ways:
> >> 1) parsing the value to test if it is an int, float or double
> (potentially
> >> error-prone)
> >> 2) add an extra metadata item to specify the type of the data
> represented
> >> (cumbersome and non-standard)
> >>
> >> My suggestion would be to be able to store metadata value along with its
> >> data type (default string as previously). This would require some kind
> of
> >> container for internal representation (e.g. in qt: QMap<QString,
> >> QVariant>)
> >> and a way to represent this on file (as .aux.xml or otherwise in files
> >> that
> >> support metadata) such as gtiff. Values could be stored as strings to
> >> avoid
> >> big/little endian conversion problems. Data type could be stored in a
> >> (hidden) metadata domain, one item for each "real" metadata key.
> >>
> >> Interesting idea.
> >>
> >> Well, we need coders now ;-)
> >>
> >> Even
> >>
> >>
> >>
> >> _______________________________________________
> >> gdal-dev mailing list
> >> gdal-dev at lists.osgeo.org
> >> http://lists.osgeo.org/mailman/listinfo/gdal-dev
> >
> >
> >
> > _______________________________________________
> > gdal-dev mailing list
> > gdal-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> What problem are we trying to solve exactly?  Is CPL* thought to be
> buggy because it isn't exposed to the wild?

I thought that gdal 2.0 could be a good occasion to upgrade/modernize some
of the legacy code in gdal, which is a bit hard to learn for newcomers and
not that well documented.

> As far as the csv and
> support files, is it a performance issue we are trying to solve?.

Although I don't have any benchmarks to support this, I perceive that the
way the .csv reading is handled is not very efficient. I recall reading
some comments in the code that this should be fixed, but that never
happened.

> Is
> the metadata system insufficient for some purposes?  It may be in
> Etienne's case, due to typing, but are there other examples?  I'm just
> curious.  Thanks.
>

I find that having metadata values as strings can be limiting and can
induce errors an inaccuracies due to rounding.

But again - these are merely suggestions - I think things work well as they
are but could need some improvements and modernizing/standardization.

cheers
Etienne

>
> --
> Kyle
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20140403/f7629c1d/attachment-0001.html>