[gdal-dev] Open source vector geoprocessing libraries?

Thu Jan 14 09:40:10 EST 2010

Ragi,

Nicely done. That is pretty funny. I'm sure there's a few others lurking on
the list who have thought of such a thing as well. Could you please send me
the code? Thanks!

Do you know if it would work if invoked from Python-based geoprocessing
tools? We expose a lot of our work to ArcGIS as those. An interesting
feature of these is that they can access layers created at run time by the
user in the ArcGIS applications. They can also run both in and out of the
ArcGIS process. The arcgisscripting Python API (and
esriGeoprocessing.GPDispatch COM object) use named pipes to communicate with
the application, or something like that. I am wondering if your driver would
have to initialize ArcObjects in a special way to somehow integrate with
that. I have never seen documentation from ESRI on how to use ArcObjects
from a "script-based" geoprocessing tool (only from a "function-based"
geoprocessing tool, which always runs in process and always uses
ArcObjects).

What is the startup cost like? I have written a lot of standalone Python
geoprocessing scripts that you run from the command line. These always take
a long time to initialize the geoprocessor (several 5 to 20 seconds,
depending on ArcGIS version) before I can call any APIs. I don't know if it
is because the geoprocessor is internally allocating an ArcObjects
"application" object, or if it is due to the license system (doesn't matter
if you use a local or remote license server). In any case, it would be great
if your driver did not suffer from this. I have not written any ArcObjects
programs that run outside of ArcGIS applications, so I don't know if it
would suffer this. (Sadly, I am betting yes.)

I do not know the process for submitting an OGR driver but the team seems
very receptive to user contributions. I would be happy to review the code
and offer comments (I'm also busy this month, so this might take some time).
Based on some thinking I've done in the past 24 hours, I'm not sure that I
will end up using it in my final application, but it would certainly be
valuable. There are periodic inquiries to gdal-dev about reading file
geodatabases, and your driver would solve that one.

There are precedents for other drivers that depend on other software (e.g.
Oracle, I think), so I don't think anyone will strenuously object to your
driver being dependent on COM and ArcObjects. Obviously it would have to be
an optional driver. It would be good to write some tests for it; that may be
a requirement from the GDAL team, although I doubt they are in a position to
run them as part of a normal GDAL release, as I don't think any of them have
ArcGIS.

A possible issue is that you'd probably need to recompile it every time a
different version of ArcGIS came out. Ideally, you would maintain versions
of it for each release of ArcGIS. The driver documentation should point out
to the user that they must obtain the right version. Alternatively you might
be able to craft one driver that dynamically determines the ArcGIS version
and does the right thing.

Best,

Jason

From: gdal-dev-bounces at lists.osgeo.org
[mailto:gdal-dev-bounces at lists.osgeo.org] On Behalf Of Ragi Y. Burhum
Sent: Wednesday, January 13, 2010 12:53 PM
To: gdal-dev at lists.osgeo.org
Subject: RE: [gdal-dev] Open source vector geoprocessing libraries?

Date: Wed, 13 Jan 2010 10:27:43 -0500
From: "Jason Roberts" <jason.roberts at duke.edu>
Subject: RE: [gdal-dev] Open source vector geoprocessing libraries?
To: "'Peter J Halls'" <P.Halls at york.ac.uk>
Cc: 'gdal-dev' <gdal-dev at lists.osgeo.org>
Message-ID: <008001ca9464$f4059f10$dc10dd30$@roberts at duke.edu>
Content-Type: text/plain;       charset="US-ASCII"

Peter,

> are you constrained to retaining your data in an ArcGIS compatible format?

We are attempting to build tools that can work with data stored in a variety
of formats. Our current user community uses mostly shapefiles, ArcGIS
personal geodatabases, and ArcGIS file geodatabases. Many of them are
ecologists who do not have the interest or skills to deploy a real DBMS
system. Thus we are hoping to provide tools that can work without one. This
is one reason I was exploring how embeddable PostGIS and SpatiaLite might be
in the other fork of this thread.

> Until the File
> Geodatabase format is published (later this year?) and someone has the
effort to
> build an OGR interface, the DBMS route is probably the best route to
> compatibility.

It would be really great for that to happen, but I'm not holding my breath.
If it does get published, I would seriously contemplate building an OGR
driver.

I have contemplated building an ArcObjects- or arcgisscripting-based driver.
This would at least allow people who have ArcGIS to use OGR to access any
ArcGIS layer, including those created by ArcGIS's tools for joining
arbitrary layers, etc. That would handle file geodatabases, as well as ALL
formats accessible from ArcGIS. If such a driver existed, then we could use
OGR as the base interface inside our application. But creating such a driver
would be a lot of work and have funky dependencies because it either needs
to use Windows COM (for ArcObjects) or Python (for arcgisscripting) to call
the ArcGIS APIs. I am certainly capable of implementing it but because most
of our code is in Python, it is probably easier for me to wrap OGR and
arcgisscripting behind a common abstraction, and then have our tools work
against that abstraction rather than OGR directly.

I find it very amusing you mention this right now. 

Why?

I asked Frank if there was an ArcObjects based OGR driver this very past
Thursday and he said "not that I know of". What I wanted was, among other
things, to get data out of FileGDB to PostGIS with one shot and add some
custom behavior for a client of mine. So I spent the past three days looking
at OGR drivers and wrote an ArcObjects based one. I got it working
yesterday.

- Right now I only instantiate 3 factories (Enterprise GDB aka ArcSDE,
AccessDB and FileGDB). This means it reads FileGDB just fine. If you want
more factories, the driver only has to be modified with one line to add any
other factories and everything else would just work.

- I only implemented the parts that I needed, so it is readonly (should be
straight forward to expand if need be). 

- Although, it can read other GeoDatabase abstractions (Topology, Geometric
Networks, Annotations, Cadastral Fabrics, etc), currently I am explicitly
filtering for FeatureClasses and FeatureDatasets.

- It is a ATL / COM / C++ based one, so it will only compile on Windows. It
can be modified to use the cross platform ArcEngine SDK since all the COM
Objects that I use are called the same and behave the same way... I just did
not have an ArcEngine SDK installer, so I could not test this.

Anyway, if you are interested in the source code, let me know. Perhaps we
can add it as an ogr driver contribution (what is the process for that
anyway?). I may not respond fast enough to e-mail, since the next 4 weeks
are pretty crazy for me.

- Ragi Burhum

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/gdal-dev/attachments/20100114/174cb91f/attachment-0001.html