"best" data format for attribute searches

[James Perrins]

Thanks Bart,

But I think I must be being dim - I still haven't quite grasped it...

"OGR can use Mapinfo attribute index files on shapefiles"
Do you really mean this ? or should shapefiles really be read as MapInfo tab
files ?

I think I'm confused by the interchangeable way you are referring to
shapefile drivers reading MapInfo files etc ?

A) If I have a mapinfo tab file layer with mapinfo indexes (.ind) on the
attribute file (.DAT / .DBF) are you saying that mapserver can make use of
these ?

B) When you say create an index by: "CREATE INDEX ON tablename USING
fieldname" - is this a MapInfo command ?

c) My issue is similar to yours I think - I don't really care if I use
MapInfo tab files or shape files - but I have some large data files and am
concerned about the speed of attribute queries and would rather avoid going
to an RDBMS if I can.

I can live with attribute indexes only supporting exact matches - and under
some circumstances I could live with just unique indexes working, but in
others it would be good if exact matches on non unique indexes (i.e.
returning multiple records) could also be supported.  Am I best off using
MapInfo indexed tab files ?

Sorry for all the questions - but I've read various posts about attribute
indexes not being supported for dbf files in Mapserver - but in none of them
did anyone suggest using OGR and Tab files as an alternative ?

Many Thanks
James

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
Behalf Of Bart van den Eijnden
Sent: 24 August 2005 11:39
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] "best" data format for attribute searches

Although I am not Frank I'll take the opportunity to answer :-)

OGR can use Mapinfo attribute index files on shapefiles, it can't  use dbf
indexes.

See here:

http://ogr.maptools.org/drv_shapefile.html

Currently the OGR Shapefile driver only supports attribute indexes for
looking up specific values in a unique key column. To create a spatial index
for a column issue an SQL command of the form "CREATE INDEX ON tablename
USING fieldname". To drop the attribute indexes issue a command of the form
"DROP INDEX ON tablename". The attribute index will accelerate WHERE clause
searches of the form "fieldname = value". The attribute index is actually
stored as a mapinfo format index and is not compatible with any other
shapefile applications.

Best regards,
Bart

Bart van den Eijnden
Syncera IT Solutions
Postbus 270
2600 AG  DELFT

tel.nr.: 015-7512436
email: BEN at Syncera-ITSolutions.nl
>>> James Perrins <JamesP at ESDM.CO.UK> 08/24/05 12:34 PM >>>
Frank,

Would you mind elaborating on shape files supporting indexed attribute
queries in OGR ?

I was under the impression from all that I had read that Mapserver
didn't use dbf indexes for shape files - (I know it can use spatial
indexes) - or is the difference here that it does if you go via OGR - or
am I missing the point !

I would be very grateful if you could point me at an example /
documentation of how to use indexed attribute queries in mapserver
without an RDBMS.

What should you use to create the dbf indexes ?
Will indexes still work if accessed via an indexed layer created using
tile4ms.exe ?

It would be great if I can use indexed attribute queries as I'm starting
to hit performance issues with large data volumes and am trying for a
variety of reasons to delay the move to something like PostGIS

Many Thanks
James

-----Original Message-----
From: Frank Warmerdam [mailto:fwarmerdam at GMAIL.COM] 
Sent: 18 July 2005 15:15
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] "best" data format for attribute
searches

On 7/18/05, Bart van den Eijnden <BEN at syncera-itsolutions.nl> wrote:
> Hi list,
> 
> AFAIK Mapserver does not use the Mapinfo attribute indexes.
> 
> I am trying to perform attribute searches on a 25 Mb Mapinfo file and
am worried about performance, since the indexes won't be used which will
mean scanning the whole data file for every attribute search.
> 
> What would be a wise format to shift to for this type of operation? Or
doesn't it make any sense to shift to eg PostGIS because of the way
Mapserver does queries to RDBMS's?
> 
> Thanks in advance for any insight on this.

Bart, 

The only non-RDBMS driver in OGR that supports indexed attribute
queries is Shapefiles, and there only exact lookups are indexable.
If you don't want the hassle of setting up a complex RDBMS like
PostGIS you might consider a lightweight equivelent like SQLite but
I do think Postgresql/PostGIS would be the ideal option.

PS. I do have a vague intention of implementing indexed attribute
query support for mapinfo tab files at some point, and the underlying
MITAB does support the index files so it shouldn't be too hard.  But
no timeline on such work. 

Best regards,

-- 
---------------------------------------+--------------------------------
------
I set the clouds in motion - turn up   | Frank Warmerdam,
warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent