[gdal-dev] Layer operations, a proposal

[Jason Roberts]

I agree that there are many scenarios small numbers of features, and for
these it is "fast enough" to do something like make a call to GEOS for all
pair-wise combinations of features. A layer or collection of scripts on top
of OGR would work fine here.

For scenarios involving large numbers of features, I suspect it is much
harder to do it fast and within available memory. It is probably necessary
to do a multi-pass approach, where the first step operates only on the
spatial indexes of the layers involved. It is probably too slow--or at least
very suboptimal--to even read all of the features into memory, much less
call GEOS on them; all of that would happen in a later pass, when a subset
of features was identified via the spatial index pass. I suspect all of this
must be implemented in the core OGR code, where it is possible to quickly
and selectively probe spatial indexes, unless OGR wants to expose an
abstraction allowing higher level code to do that. I could definitely
appreciate the view that either of these approaches is significant scope
creep.

In any case, I appreciate that OGR developers are considering this
functionality, even if it is ultimately only for the small case. I hope you
are able to develop something.

Jason

-----Original Message-----
From: gdal-dev-bounces at lists.osgeo.org
[mailto:gdal-dev-bounces at lists.osgeo.org] On Behalf Of Ari Jolma
Sent: Wednesday, April 18, 2012 12:45 PM
To: gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] Layer operations, a proposal

On 04/18/2012 03:07 AM, Jason Roberts wrote:
> As I have mentioned previously
> (http://lists.osgeo.org/pipermail/gdal-dev/2010-January/023089.html)

Thanks for pointing this out.

>   I think
> something like this would be a great addition to OGR. No doubt it 
> would be a huge job--basically turning OGR into a GIS--but if it could 
> be done well, with sufficient performance and in the core API, then 
> many systems could potentially leverage it, instead of everybody 
> implementing their own versions of Ari's pseudocode.

I don't think it would be a huge job. Maybe I'm missing something but I see
the basic implementation very simple and generic. Perhaps there is a
performance problem somewhere (looping through 1 million features within a
loop of 1 million features and using shapefiles) but for common cases it
should be ok, especially if the layers fit into memory.

Once I get something done (currently testing the 2nd algorithm), it would be
interesting to do some benchmarking with available datasets.

Best regards,

Ari

>
> Best,
> Jason
>
> -----Original Message-----
> From: gdal-dev-bounces at lists.osgeo.org 
> [mailto:gdal-dev-bounces at lists.osgeo.org] On Behalf Of Frank Warmerdam
> Sent: Tuesday, April 17, 2012 7:32 PM
> To: Ari Jolma
> Cc: gdal-dev at lists.osgeo.org
> Subject: Re: [gdal-dev] Layer operations, a proposal
>
> Ari,
>
> I think this would be an interesting addition.   Would you be willing
> to write up an RFC?
>
> I think the layer creation step should be a different call than the 
> feature processing step to maximize the chance that folks who need to 
> do something very format specific at layer creation can do so and 
> still use the generic algorithm for processing the features.
>
> Your operations appear to be focused on spatial relationships, but 
> I've also heard a lot of interest in feature joins on attribute 
> fields.  One case is just merging features from a layer with a common 
> attribute value.  Another case would be some sort of attribute based 
> join between layers.  Hopefully we can use a reasonably similar approach
for a variety of operations.
>
> I'm curious how you plan to handle the spatial relationships efficiently?
> An in memory spatial index?  I'd like have the API overly constrain 
> how we implement this as I can imagine the internal operation 
> implementation evolving over time, and perhaps taking different 
> strategies for different styles of input data.  For instance an 
> in-memory spatial index might not make much sense for large numbers of 
> very simple point features while it would be good for heavy polygon
features.
>
>
> Best regards,
> Frank
>
> On Tue, Apr 17, 2012 at 2:42 PM, Ari Jolma<ari.jolma at gmail.com>  wrote:
>> Folks,
>>
>> I propose a set of new methods for OGR layers (see the PDF). I took 
>> the basic ideas from page 
>> http://courses.washington.edu/gis250/lessons/Model_Builder/ which 
>> seems rather comprehensive. I wrote the pseudo code myself quickly 
>> (thus it may not be optimal and may contain bugs - improvement 
>> suggestions are of course welcome).
>>
>> I was considering to write these in Perl (on top of the Perl 
>> bindings) but probably they could also be in OGR core. Code-wise they 
>> would probably go into ogrlayer.cpp.
>>
>> If there is support/interest, I assume I should submit this as a RFC 
>> (with a patch)?
>>
>> Best regards,
>>
>> Ari
>>
>>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>

_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev