[postgis-users] ESRI GDB format into PostGIS

[Simon Greener]

Ragi,

Lots of good, sensible comment. I particularly agree about SDE vs GeoDatabase. I believe Manifold GIS understands the small handful of tables in SDE that go way back to 1.0 but does not understand the GDB_* tables added at 8.x or the versioning tables. But I will check.

> Now I and (hopefully) you understand why open formats are a
> "good thing", but the philosophy behind it has not struck ESRI management
> strong enough. 

I just am not one who ascribes to this viewpoint any more for spatial data management within
enterprise, though I do agree we need specialised interchange formats where nothing exists
in the IT world for data interchagne. 

Is anyone seriously arguing that we need to force Oracle, Microsoft or IBM to make their storage 
formats public so programmers can create APIs and hack their way into the data that is stored within? 
How many GIS programmers here have gone to PostgreSQL's documentation to work out how a table
is stored? 

ESRI's problem is that they believe spatial data has an inherent logic to it that forces them to create new 
and wonderful (effectively) database formats to manage spatial (and attribue) data. They continue to force 
a divide into data management within organisations into two camps "spatial" (ie "mine") and "attribute" (ie "yours").

As Chris Date and Hugh Darwen wrote in their book "Foundation for Future Database Systems: The Third Manifesto".

“What we are saying is that, in the relational world, a domain is a data type, system- or user-defined, whose values are manipulable soley by means of the operators defined for the type in question (AND WHOSE INTERNAL REPRESENTATION CAN BE ARBITRARILY COMPLEX BUT IS HIDDEN FROM THE USER).”  [My emphasis]

What Date and Darwen say for the "relational world" applies to all database technologies: what we need for access is logical interfaces not a dependence on accessing data by knowing the physical storage formats.

ESRI's efforts deliver neither open logical access technologies (eg spatial SQL via open JDBC implementations) or open physical formats.

Just my 4c worth.

S.

On Tue, 02 Jun 2009 06:13:32 +1000, Ragi Y. Burhum <rburhum at gmail.com> wrote:

> On Sat, May 30, 2009 at 12:00 PM, <
> postgis-users-request at postgis.refractions.net> wrote:
>
>> Date: Sat, 30 May 2009 09:46:46 +0700
>> From: Bruce Foster <gis.foster at gmail.com>
>> Subject: Re: [postgis-users] ESRI GDB format into PostGIS
>> To: PostGIS Users Discussion <postgis-users at postgis.refractions.net>
>> Message-ID:
>>        <c97454c50905291946m76c5e023n74842b3fabcc5dec at mail.gmail.com>
>> Content-Type: text/plain; charset=ISO-8859-1
>>
>> HI
>>
>> this could help;
>
>
> I think it could, except that it is not entirely accurate :)
>
>
>>
>>
>> ESRI Geodatabases
>
>
> ...
>
>
>>
>> Nomenclature
>>
>> Many users are baffled by ESRI nomenclature when it comes to parsing
>> the bewildering variety of marketing phrases ESRI has used to describe
>> ESRI "geodatabase" formats. If you feel baffled, you are not alone.
>
>
> Amen to that!
>
>
>
>> In
>> a nutshell, ESRI at one point introduced the idea of storing geometry
>> in DBMS using a format that more or less boiled down to storing
>> shapefiles within blobs. This was done in a complex way using
>> middleware called ArcSDE that worked with serious databases like
>> Oracle, and it was also done in a somewhat simplified way in Personal
>> Geodatabase products that used Access .MDB files and were later
>> apparently updated to work with MSDE (a free version of Microsoft SQL
>> Server) or with SQL Server Express. In recent years, the ArcSDE
>> product name seems to have been dropped by ESRI: more recent versions
>> of this technology have been packaged as part of the ArcGIS product
>> family and have been referred to as geodatabases.
>>
>
> Yep
>
>
>>
>> All such storage methods are technically similar and are generally
>> referred to as SDE geodatabase formats or as Personal geodatabase
>> formats when in the somewhat simplified form that uses Access .MDB for
>> file-based storage. Since all such formats are similar or derived from
>> ArcSDE, they are referred to by Manifold documentation as ESRI SDE or
>> as ESRI Geodatabase or as Personal Geodatabase data sources, the
>> various terms being used interchangeably, regardless of which file
>> format or DBMS system is used to store the data.
>
>
> Here, there is no mention of FileGDB, which is what the discussion was
> about.
>
>
>>
>>
>> The terms are used interchangeably because some ESRI users come from a
>> long ArcSDE tradition and don't realize that "geodatabase" is the new
>> term for the same old thing, while some newer ESRI users might not
>> realize that their "geodatabase" is really SDE technology with a new
>> name. Because of the confusion caused by ESRI names for their SDE and
>> their Personal technology being so similar, Manifold documentation
>> will often refer to SDE and Personal geodatabases to underline that a
>> particular capability is available whether one is working with either
>> SDE geodatabases or the somewhat simpler Personal geodatabases.
>
>
> This is incorrect. ArcSDE is, in a nutshell, the middleware that acts as a
> data access layer that is used by the GeoDatabase. The GeoDatabase is the
> term used to encapsulate the collections of technologies that include
> Topology, Geometric Networks, Network Datasets, Annotation, Representation
> Layers, Versioning, editing behavior, Disconnected Editing, Replication,
> Raster Catalogs, etc etc. So Personal GeoDatabase (mdb files), FileGDB (.gdb
> files), an Enterprise GeoDatabase (databases accessed through ArcSDE) are
> just implementations of the ESRI idea of what "GeoDatabase" entails.
>
>
>>
>> Manifold can also connect to ESRI SDE and Personal geodatabase data
>> sources for full read / write / edit capability.
>
>
> This is the scary part as well as a bold statement. If you don't use any of
> the complex dataset types (i.e. you just use Simple Features within the ESRI
> environment), then "read/write/edit" is very simple to implement. Just write
> SQL directly (insome cases), stored procedures, use the ArcSDE client
> libraries, whatever.
>
> However, "Full read / write / edit capability" would mean that you
> understand 100% all the different components inside the GeoDatabase and have
> implemented routines that cause edits to dirty, percolate, update, etc the
> different tables/rows involved in all GeoDatabase abstractions. In fact,
> within ESRI, nobody is able to claim that they understand the entire
> GeoDatabase (OK well, maybe only one person - and no, it is not Scott
> Morehouse). The reason is not technical difficulty, it is just a moving
> target: the behavior changes and since the format is not "published" it is
> easy to tweak and change within ArcGIS versions as long as the ArcObjects
> API is tweaked accordingly. No open spec, easy to change within versions, no
> need to explain it to anybody.
>
> Hence, why there is no public FileGDB specification.
>
> To do this, it would mean to open the box and let everyone know all the
> "precious ESRI secrets about the other complex types" and there hasn't been
> any "strong business argument" to cause enough presure inside ESRI to make
> that happen. Now I and (hopefully) you understand why open formats are a
> "good thing", but the philosophy behind it has not struck ESRI management
> strong enough. **My*** interpretation of the idea of "open architecture"
> inside ESRI (and I may be wrong since it has been a few years since I left
> my Redlands job) is that open interoperability is reached from the
> webservice tier and that there is no need for openess at any other level of
> the architecture stack.
>
> ...
>
>
>>
>> http://www.manifold.net/doc/manifold.htm
>>
>> Bruce
>>
>>
>> On Fri, May 29, 2009 at 6:38 AM, Simon Greener
>> <simon at spatialdbadvisor.com> wrote:
>> > Frank and Piotr,
>> >
>> > Yes, Manifold has reverse-engineered Personal and Enterprise
>> GeoDatabases. DOn't know how well.
>> >
>> > Type "GeoDatabase" into Search in the Manifold Help.
>> >
>> > But, the way it is done is that one uses Tools>Database Console tto go to
>> the database (eg Access .mdb) file that holds the
>> > geodata. When Manifold opens it it will recognise the ESRI crap and allow
>> you to import/link to it.
>> >
>> > regards
>> > Simon
>> >> Frank Warmerdam wrote:
>> >>>
>> >>> Are you suggesting that Manifold GIS has reverse engineered the file
>> >>> geodatabase format?  Can you provide any pointers supporting that?
>> >>> If those guys can reverse engineer it, then so could we given enough
>> >>> desire.
>
>
> Frank W.:  years ago when the first talks about the Open FileGDB API
> started, I personally suggested a GDAL/OGR driver as the primary venue for
> this open API. I know that the primary engineer dedicated to FileGDB
> contacted you directly with an inquiry about this topic. I know this,
> because he stopped by my office and we talked about it. If there is some
> momentum/need to reverse-engineer FileGDB (for Simple Features), I would
> recommend to start by contacting him directly. He is an extremely friendly
> fellow and will no doubt get you pointers that would help creating the open
> source driver a much easier task.
>
>
> My two cents,
>
> - Ragi Burhum
> 



-- 
SpatialDB Advice and Design, Solutions Architecture and Programming,
Oracle Database 10g Administrator Certified Associate; Oracle Database 10g SQL Certified Professional
Oracle Spatial, SQL Server, PostGIS, MySQL, ArcSDE, Manifold GIS, FME, Radius Topology and Studio Specialist.
39 Cliff View Drive, Allens Rivulet, 7150, Tasmania, Australia.
Website: www.spatialdbadvisor.com
  Email: simon at spatialdbadvisor.com
  Voice: +61 362 396397
Mobile: +61 418 396391
Skype: sggreener
Longitude: 147.20515 (147° 12' 18" E)
Latitude: -43.01530 (43° 00' 55" S)
NAC:W80CK 7SWP3