[gdal-dev] Store also native geometry into Elastic Search?

[Even Rouault]

Le jeudi 03 septembre 2015 14:47:16, Adam Estrada a écrit :
> Even,
> 
> I think the reason to store everything by default was just to be on
> the safe side and it satisfied some of the project requirements at the
> time. That was back in 2011 so ElasticSearch and this driver are
> probably not in sync any longer.

OK, do you remember which ES version was targetted at that time ? So I can 
potentially be able to find in the doc the behaviour and see if it has changed.

> There is a lot of work that can be
> done to the driver to get up to date with all the cool stuff that is
> now available in ES. I'd be interested in starting a roadmap
> specifically for the ES driver starting with a read capability and the
> ability to store other geometry types.

Actually, you should have a look at the recent additions at
http://gdal.org/drv_elasticsearch.html ;-)

> 
> Adam
> 
> On Thu, Sep 3, 2015 at 7:42 AM, Even Rouault <even.rouault at spatialys.com> 
wrote:
> > Le jeudi 03 septembre 2015 12:49:30, Jukka Rahkonen a écrit :
> >> Hi,
> >> 
> >> What if somebody would like to put and deliver some official spatial
> >> data (cadastral data) which are not in EPSG:4326 with Elastic Search?
> >> Could it be possible to translate the geometries into EPSG:4326 and
> >> store and index them as geo_shape, but also keep the native geometries
> >> and save them into another field? Perhaps json does not like to have
> >> many geometry fields but how about saving the native geometry as WKT?
> >> Indexing WKT field feels stupid but maybe ES could be mapped to skip
> >> it. Reasoning is that conversion from native to EPSG:4326 and back to
> >> native is not necessary accurate and coordinates may change a bit.
> > 
> > Jukka (and little question for you Adam if you read this),
> > 
> > Well, you can do that with a Spatialite SQL query like :
> > 
> > -sql "select *, ST_AsText(geometry) as WKT_32631 from simple_EPSG_32631"
> > -dialect sqlite
> > 
> > As far as not indexing the WKT_32631 field, you would need to:
> > 1) export the default mapping with -lco WRITE_MAPPING=mapping.json
> > 2) add "index":"no" to the WKT_32631 definition
> > (https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping
> > -core-types.html) 3) run again with -lco MAPPING=mapping.json
> > 
> > Doing all the above steps in a fully automated way, perhaps with a
> > dedicated PRESERVE_SRC_GEOMETRY layer creation option, would certainly
> > be possible, but personnaly, it's more in the
> > something-to-perhaps-consider-someday queue ;-)
> > 
> > Perhaps we would need a few creation options to alter more easily the
> > mapping of the created fields (rather than the above
> > export/modify/import workflow). Not sure at this point which ones would
> > make sense, and if we aren't careful, there's a risk we end up with
> > something complicated that would be redundant with the JSon API. I also
> > see that the existing write-only driver creates a mapping that
> > explicitely stores fields, and as we don't disable storing the _source
> > field, so apparently this wouldn't be necessary. Hum, might be wise to
> > remove the explicit "store": "yes"
> > 
> > @adam : do you remind the reason for adding "store": "yes" in the
> > generated ES mapping ?
> > 
> >> Am I even right that the conversion into EPSG:4326 is necessary for
> >> using the geo_shape query?
> > 
> > That's my understanding of
> > https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-
> > geo-shape-type.html "In GeoJSON, and therefore Elasticsearch, the correct
> > coordinate order is longitude, latitude (X, Y) within coordinate arrays"
> > 
> > Actually I've just tried to manually insert with invalidate lon,lat and
> > the behaviour is hard to decypher:
> > 
> > e.g
> > 
> > curl -d '{  "geometry": { "type": "linestring", "coordinates": [ [ 800.0,
> > 0.0 ], [ 3.0, 50.0 ] ] } }'
> > "http://localhost:9200/test/FeatureCollection?pretty"
> > 
> > works,
> > 
> > but
> > 
> > curl -d '{  "geometry": { "type": "linestring", "coordinates": [ [
> > 1000.0, 0.0 ], [ 3.0, 50.0 ] ] } }'
> > "http://localhost:9200/test/FeatureCollection?pretty"
> > 
> > throws an error:
> > "MapperParsingException[failed to parse [geometry]]; nested:
> > IllegalArgumentException[This method does not support GeometryCollection
> > arguments]; "
> > 
> > And I assume that any spherical geospatial search will behave in
> > undefined ways if you insert coordinates that are not (lon,lat)
> > 
> > Even
> > 
> > --
> > Spatialys - Geospatial professional services
> > http://www.spatialys.com

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com