"best" data format for attribute searches

[Bart van den Eijnden]

Although I am not Frank I'll take the opportunity to answer :-)

OGR can use Mapinfo attribute index files on shapefiles, it can't  use dbf indexes.

See here:

http://ogr.maptools.org/drv_shapefile.html

Currently the OGR Shapefile driver only supports attribute indexes for looking up specific values in a unique key column. To create a spatial index for a column issue an SQL command of the form "CREATE INDEX ON tablename USING fieldname". To drop the attribute indexes issue a command of the form "DROP INDEX ON tablename". The attribute index will accelerate WHERE clause searches of the form "fieldname = value". The attribute index is actually stored as a mapinfo format index and is not compatible with any other shapefile applications.

Best regards,
Bart

Bart van den Eijnden
Syncera IT Solutions
Postbus 270
2600 AG  DELFT

tel.nr.: 015-7512436
email: BEN at Syncera-ITSolutions.nl
>>> James Perrins <JamesP at ESDM.CO.UK> 08/24/05 12:34 PM >>>
Frank,

Would you mind elaborating on shape files supporting indexed attribute
queries in OGR ?

I was under the impression from all that I had read that Mapserver
didn't use dbf indexes for shape files - (I know it can use spatial
indexes) - or is the difference here that it does if you go via OGR - or
am I missing the point !

I would be very grateful if you could point me at an example /
documentation of how to use indexed attribute queries in mapserver
without an RDBMS.

What should you use to create the dbf indexes ?
Will indexes still work if accessed via an indexed layer created using
tile4ms.exe ?

It would be great if I can use indexed attribute queries as I'm starting
to hit performance issues with large data volumes and am trying for a
variety of reasons to delay the move to something like PostGIS

Many Thanks
James

-----Original Message-----
From: Frank Warmerdam [mailto:fwarmerdam at GMAIL.COM] 
Sent: 18 July 2005 15:15
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] "best" data format for attribute
searches

On 7/18/05, Bart van den Eijnden <BEN at syncera-itsolutions.nl> wrote:
> Hi list,
> 
> AFAIK Mapserver does not use the Mapinfo attribute indexes.
> 
> I am trying to perform attribute searches on a 25 Mb Mapinfo file and
am worried about performance, since the indexes won't be used which will
mean scanning the whole data file for every attribute search.
> 
> What would be a wise format to shift to for this type of operation? Or
doesn't it make any sense to shift to eg PostGIS because of the way
Mapserver does queries to RDBMS's?
> 
> Thanks in advance for any insight on this.

Bart, 

The only non-RDBMS driver in OGR that supports indexed attribute
queries is Shapefiles, and there only exact lookups are indexable.
If you don't want the hassle of setting up a complex RDBMS like
PostGIS you might consider a lightweight equivelent like SQLite but
I do think Postgresql/PostGIS would be the ideal option.

PS. I do have a vague intention of implementing indexed attribute
query support for mapinfo tab files at some point, and the underlying
MITAB does support the index files so it shouldn't be too hard.  But
no timeline on such work. 

Best regards,

-- 
---------------------------------------+--------------------------------
------
I set the clouds in motion - turn up   | Frank Warmerdam,
warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent