shptree not faster than no index

[Stephen Woodbridge]

John,

I have tiger data running under mapserver
   http://imaptools.com/tiger/
I have about 18-19 GB of data for these maps.

I just leave the data the way it is translated to shapefiles and build
tileindexes and shptree indexes on the everything and it works great! I
wouldn't spend extra time trying to merge data into a single layer
shapefiles.

In your test you found the it was fast and it didn't make a lot of
difference. Unless you are trying to squeeze every ounce of performance
you can out of the setup then you would have to do a bunch of
experimentation to find out what makes it fast or not.

-Steve W.

John Bolster wrote:

> Thanks for the advice. I'll try it this way on the counties. However, when I
> start working with the other files, like roads, water and places, I'm
> dealing with files that are currently broken down by county and each one may
> have anywhere from 3,000 - 30,000 items per file. So that multiplied by
> 3,000 or so counties comes to 9 - 90 million items if I combined each record
> type into one big file each. Would you still suggest combining them in these
> cases, or do you think that gets too large?
>
> John
>
>
>>-----Original Message-----
>>From: ANDY CANFIELD [mailto:andy_canfield at hotmail.com]
>>Sent: Thursday, September 23, 2004 6:51 PM
>>To: jb at NUCOMP.COM; MAPSERVER-USERS at LISTS.UMN.EDU
>>Subject: Re: [UMN_MAPSERVER-USERS] shptree not faster than no index
>>
>>
>>I'd go with the one file of all counties. It's just easier to
>>maintain one
>>file than 3,000. I have at least 50 layers in one of my maps that
>>each layer
>>can have 300,000 to 1,000,000 plus objects in it. Using the shptree to
>>create a .qix file makes these layers incredibly fast. I mean fast as in,
>>you click on the map to zoom or pan, and as fast as your browser
>>can render
>>the map has been updated to reflect that pan or zoom. I think my smallest
>>layer has 1,400 hundred polygons in it. Using the shptree and having one
>>file is way easier than maintaining a single file for each boundary. I
>>honestly don't think the map I'm using could be any faster, it is the
>>browser/desktop rendering speed for me that limits how fast the image
>>changes, so that's really, really fast. Plus if you have 3,000
>>files and are
>>zoomed to an extent where you can see half of them or more that's
>>1,500 or
>>more open and read operations that Mapserver has to execute vice
>>one if you
>>put them all in the same file. My two cents anyway.
>>
>>
>>
>>>From: John Bolster <jb at NUCOMP.COM>
>>>Reply-To: John Bolster <jb at NUCOMP.COM>
>>>To: MAPSERVER-USERS at LISTS.UMN.EDU
>>>Subject: Re: [UMN_MAPSERVER-USERS] shptree not faster than no index
>>>Date: Thu, 23 Sep 2004 18:18:46 -0400
>>>
>>>Thank you for your quick response.
>>>
>>>In terms of the county outlines, do you think that one county
>>
>>per file, and
>>
>>>selecting which files to use with only a tileindex would be
>>
>>faster than all
>>
>>>counties being in one huge file of over 3000 counties and selecting which
>>>counties to draw using a qix index?
>>>
>>>Also, does the qix index on the actual tileindex do anything to speed up
>>>the
>>>tileindex?
>>>
>>>Thanks,
>>>John
>>>
>>>
>>>>-----Original Message-----
>>>>From: Frank Warmerdam [mailto:warmerdam at pobox.com]
>>>>Sent: Thursday, September 23, 2004 6:13 PM
>>>>To: John Bolster
>>>>Cc: MAPSERVER-USERS at LISTS.UMN.EDU
>>>>Subject: Re: [UMN_MAPSERVER-USERS] shptree not faster than no index
>>>>
>>>>
>>>>John Bolster wrote:
>>>>
>>>>>Hi all,
>>>>>
>>>>>I'm noticing there doesn't seem to be any difference in speed
>>>>
>>>>between having
>>>>
>>>>>*.qix files vs. not having them. I'm wondering if I'm doing this
>>>
>>>right.
>>>
>>>>>My data is from tiger and is in shapefiles. I'm only dealing with
>>>
>>>county
>>>
>>>>>outlines right now. They are each in a separate folder, each state's
>>>>>counties are in one state folder. There's a tileindex on
>>
>>them all at
>>
>>>the
>>>
>>>>>root of the tree.
>>>>
>>>>John,
>>>>
>>>>The impression I get is that you have exactly one county polygon
>>>>per shapefile.
>>>>Is that right?  A .qix won't help because it is used to more
>>>>quickly select
>>>>the subset of shapes in a shapefile based on the extents.  If you
>>>>only have
>>>>one shape per file then it can't help.
>>>>
>>>>Of course, the tileindex should allow you to quickly select the county
>>>>files you want.  But a more efficient approach would likely
>>
>>be to have
>>
>>>one
>>>
>>>>shapefile with a .qix with all your county outlines in it.
>>>>
>>>>Best regards,
>>>>--
>>>>---------------------------------------+--------------------------
>>>>------------
>>>>I set the clouds in motion - turn up   | Frank Warmerdam,
>>>>warmerdam at pobox.com
>>>>light and sound - activate the windows | http://pobox.com/~warmerdam
>>>>and watch the world go round - Rush    | Geospatial
>>
>>Programmer for Rent
>>
>>>>
>>_________________________________________________________________
>>FREE pop-up blocking with the new MSN Toolbar – get it now!
>>http://toolbar.msn.click-url.com/go/onm00200415ave/direct/01/
>>
>>
>
>