[gdal-dev] Fastest vector format for combining shapefiles

[Matt Wilkie]

Simon, it's clear you have a great deal of experience in this area. If 
it were up to you to chart out a course for an open source spatial 
vector format that trancends the current limitations of shape, gml, etc. 
what would you advise? Perhaps all that is missing is an architectural 
plan.
> The only reason I ever hear for physical file formats is "we need to do this for performance reasons

Personally I like physical files because they accessible and portable. I 
don't have to install, configure and run an application just to get to 
the point of reading it. Neither do I have to export to an intermediate 
state to transfer to another machine or medium and then import on the 
other end. Maybe this is more properly a limitation of the readily 
available tools than the storage format though.

best regards,

matt wilkie
--------------------------------------------
Geomatics Analyst
Information Management and Technology
Yukon Department of Environment
10 Burns Road * Whitehorse, Yukon * Y1A 4Y9
867-667-8133 Tel * 867-393-7003 Fax
http://environmentyukon.gov.yk.ca/geomatics/
-------------------------------------------- 



Simon Greener wrote:
> The need for a new vector file format has been discussed many times with no action initiated by the open source community on what to do.
>
>   
>> ESRI has said they will do so, but it's been several
>> years since they first announced it and when it is finally is released
>> there is no guarantee it will be under license terms open source
>> projects can use. This isn't to say it will be unusable either, we
>> just don't know.
>>     
>
> Ahhh, isn't it wonderful waiting for the "crumbs that will fall from the master's table"!
>
> And, "to wait" is part and parcel of being an ESRI-centric customer or user: strange that open source people are willing to do the same.
> (Well, at least you aren't paying for the privilege of using the stuff.)
>
> Why won't ESRI release an FDO (an Open Source open access API) provider for FGDBs rather than their own API? (I can find no reference to ESRI offering to do so for any of its formats.) Sounds like API lock-in is a design goal for the FGDB API!
>
> Don't forget that a FGDB is full of ESRI concepts (not OGC or SQL/MM or those promoted by any other standards body) - more lock-in if it becomes the much hoped for replacement for shapefiles. And, what's more, we know nothing about what will be in the API. Where is the community engagement? Will we end up with an API via which we cannot (four examples will suffice):
>
> 1. Properly design (cf CASE tool) an FGDB (cf ESRI $$ extensions to Visio);
> 2. Create an FGDB from scratch;
> 3. Write data or create important objects (ie versions);
> 4. Create FGDB spatial and attribute indexes or even use them via the API (cf shapefile indexing);
>
> These are points which have grounding in past ESRI practices. All done deliberately so you have to have a copy of ArcGIS to construct, design and get the best out of a fully specified FGDB?
>
> And then, when there are serious bugs, you have to wait for 18 months for a fix while in the Open Source community you could get one in a matter of days or weeks?
>
> Seriously, though, isn't open source about taking control of one's destiny and being a part of a truly open, collaborative, process and not waiting for the bully in the playground to tell you what you can and can't do, or who really isn't interested in your deadlines and real needs? Many times in my long GIS career I've had conversations with the 'true believers' over in Redlands. One was like this: "When will you support an Oracle Sdo_Geometry circle object in ArcSDE?". Reply: "Circles in GIS? We don't think you need them....".
>
> The fixated concentration of the GIS community on physical file formats feels very much like a 1960s form of data management and computing. Logical separation from physical implementation, normalise for edit/denormalise for performance, logical separation from physical implementation, normalise for edit/denormalise for performance, logical..... oops the record is broken ....
>
> The only reason I ever hear for physical file formats is "we need to do this for performance reasons"..... but this usually masks a lot of other reasons and assumptions (like it is "quicker and easier" that soon morphs into a management nightmare).....
>
> cynically
> Simon
>