Attributes in tile indexes?

[Stephen Woodbridge]

Jeff,

For tiger data I use the counties as that is the way the data is 
organized so I have 3300 counties in the US and each county is a 
directory with all the layers in that directory.

For Navteq data, I use shp2tile with the quad tree option and limit the 
maximum number of entities in a tile to 8000. This seems to work 
reasonably well, but I haven't actual tested variations on that to see 
what is better.

You also should build mapserver with the debug option in ./configure and 
then add DEBUG ON in all you layers and look at the apache error log for 
the timing info on the layers. One of the big killers is the labelcache 
processing time.

<soapbox>
I have discussed a new algorithm on the dev list regarding the 
labelcache processing that should change the it from O(n*(n-1)) to O(n) 
complexity problem. I'm hoping I can get someone to champion the 
development because when there are a lot of labels the current 
labelcache processing can take nearly 50% to the map rendering time.
</soapbox>

If the labelcache is you problem, try to thin the number of objects that 
could potentially be labeled at the zoom scale in question. You might 
try splitting the shapefile based on road class into major and minor 
roads and only labeling the major roads until you are zoomed in even more.

-Steve

Jeff Dege wrote:
> In my particular case, I'm only using this layer when zoomed in tightly.
> 
> The shapefile in question contains street data.  There are millions of
> features in the original shapefile.  But I have very much smaller
> shapefiles containing only the major roads, only the highways, and only
> the interestates, that I use at wider zoom levels.  It's only at the
> tight zooms that I will draw the individual streets, so trying to read
> large numbers of tiles is not an issue.
> 
> What's surprised me in this case is that it's the first time where only
> drawing detail on close-in zooms, and adding spatial indexes to the
> shapefiles with shptree wasn't enough to get reasonable performance.
> 
> I'm currently running shp2tile with different arguments, trying to see
> how speed vs. number of files trade off against one another.  But with a
> half-gig shapefile, each trial is taking a while.
> 
> My current fastest attempt cut the time to draw to 1/4 of what I
> originally had.  Which isn't fast enough.
> 
>> -----Original Message-----
>> From: UMN MapServer Users List 
>> [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On Behalf Of Ed McNierney
>> Sent: Monday, April 23, 2007 1:05 PM
>> To: MAPSERVER-USERS at LISTS.UMN.EDU
>> Subject: Re: [UMN_MAPSERVER-USERS] Attributes in tile indexes?
>>
>> Jeff -
>>
>> In addition to Steve's suggestion, please keep in mind that 
>> what you are
>> trying to do is reduce the amount of data MapServer needs to 
>> read in order
>> to retrieve the data you need.  Think about your data 
>> carefully and see
>> whether you're achieving that goal.
>>
>> First, splitting the shapefile into multiple files and 
>> indexing them with a
>> TILEINDEX helps MapServer select a subset of that data to be 
>> used for a
>> given map draw.  If MapServer has to open most or all of 
>> those shapefiles
>> anyway, then the TILEINDEX isn't going to do a thing!
>>
>> Since your method of producing shapefile tiles will minimize 
>> overlap between
>> files, you should also check the relationship of your map's 
>> spatial extent
>> with those tiles.  Ideally, each tile should be (roughly) 
>> somewhat larger
>> than the extent of the map view.  With rectangularly-tiled 
>> data, you cannot
>> avoid situations where the map view is centered on a corner 
>> and you need to
>> open four different input shapefiles.  If each rectangular 
>> tile is, however,
>> slightly larger than the map image size, then you will never 
>> get a situation
>> that is WORSE than four input files.  If you render a zoomed-out view
>> covering dozens of input files, your TILEINDEX isn't going to 
>> help much.
>>
>> Think about how your TILEINDEX and split shapefiles are 
>> behaving.  Have you
>> substantially eliminated most of your input data from 
>> consideration in a
>> map-drawing request?  If not, then you need to go back and 
>> think about how
>> to restructure your tiling and indexing, since that's the 
>> whole goal of
>> those techniques.
>>
>>      - Ed
>>
>> -- 
>> Ed McNierney
>> President and Chief Mapmaker
>> Maps a la carte, Inc. / TopoZone.com
>> 73 Princeton Street, Suite 305
>> North Chelmsford, MA  01863
>> Phone: (978) 251-4242
>> Fax: (978) 251-1396
>> ed at topozone.com
>>
>>> From: Stephen Woodbridge <woodbri at SWOODBRIDGE.COM>
>>> Reply-To: Stephen Woodbridge <woodbri at SWOODBRIDGE.COM>
>>> Date: Mon, 23 Apr 2007 11:53:56 -0400
>>> To: <MAPSERVER-USERS at LISTS.UMN.EDU>
>>> Subject: Re: [UMN_MAPSERVER-USERS] Attributes in tile indexes?
>>>
>>> Jeff,
>>>
>>> Make SURE all your tiles have a spatial index *.qix and the 
>> tileindex
>>> also has a spatial index. These can be built with
>>>
>>>    shptree file.shp
>>>
>>> -Steve W.
>>>
>>> Jeff Dege wrote:
>>>> OK - I knew I had to be doming something stupid.
>>>>
>>>> To tell a layer to draw using a tileindex shapefile, you 
>> include the
>>>> commands:
>>>>
>>>>    TILEINDEX "tileindex"
>>>>    TILEITEM "LOCATION"
>>>>
>>>> Using:
>>>>
>>>>    DATA "tileindex"
>>>>
>>>> won't work.
>>>>
>>>> Now, after all of that, drawing with the split shapefiles 
>> is slower than
>>>> with the original.  So there's still some playing to do.
>>>>  
>>>>
>>>>> -----Original Message-----
>>>>> From: UMN MapServer Users List
>>>>> [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On Behalf Of Jeff Dege
>>>>> Sent: Monday, April 23, 2007 9:53 AM
>>>>> To: MAPSERVER-USERS at LISTS.UMN.EDU
>>>>> Subject: Re: [UMN_MAPSERVER-USERS] Attributes in tile indexes?
>>>>>
>>>>> Well, I've been looking at this in more detail, and 
>> finding that the
>>>>> tileindex layer isn't drawing at all.  With all other layers
>>>>> turned off,
>>>>> when I use shp2img with an extent that causes the tile layer
>>>>> to draw, I
>>>>> get a blank image.  When I change the layer to use the original
>>>>> shapefile, I get the data displayed.  (And the layer takes 40
>>>>> seconds to
>>>>> draw, even after I've run shptree on the shapefile, which 
>> is why I've
>>>>> been playing around with tiling.)
>>>>>
>>>>> I identified the individual tile shapefile that contains 
>> the area I'm
>>>>> trying to draw and used it directly, and it works fine.  So the
>>>>> splitting apart seems to be working.  But the tileindex
>>>>> shapefile seems
>>>>> to be broken.
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: UMN MapServer Users List
>>>>>> [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On Behalf Of Stephen
>>>>> Woodbridge
>>>>>> Sent: Saturday, April 21, 2007 7:15 AM
>>>>>> To: MAPSERVER-USERS at LISTS.UMN.EDU
>>>>>> Subject: Re: [UMN_MAPSERVER-USERS] Attributes in tile indexes?
>>>>>>
>>>>>> Jeff,
>>>>>>
>>>>>> This should just work. There is nothing special you should
>>>>>> have to do. I 
>>>>>> do it all the time with my mapfiles. I use shp2tile to break
>>>>>> mine up but 
>>>>>> that should not make a difference.
>>>>>>
>>>>>> First I would try changing your layer to use just one of
>>>>> the smaller 
>>>>>> shape files without the tileindex and see if that works and
>>>>>> that you can 
>>>>>> display the labels.
>>>>>>
>>>>>> You might also want to post info about what OS, version of
>>>>> mapserver, 
>>>>>> and the LAYER block in question.
>>>>>>
>>>>>> -Steve W
>>>>>>
>>>>>> Jeff Dege wrote:
>>>>>>> I have a shape file that's too big to draw maps with -
>>>>> takes far too
>>>>>>> long.
>>>>>>>
>>>>>>> In an attempt to speed things up, I split it into 16
>>>>>> smaller shapefiles,
>>>>>>> using:
>>>>>>>
>>>>>>>    ogr2ogr -f "ESRI Shapefile" -spat xmin ymin xmax 
>> ymax newXX.shp
>>>>>>> old.shp
>>>>>>>
>>>>>>> Then did a shptree on each of these new files, then a
>>>>> shptinidex to
>>>>>>> create a single tile index shapefile on them, and tried to
>>>>>> build a map.
>>>>>>> It works fine, so long as I don't try to display labels.
>>>>>> But if I do, I
>>>>>>> get errors:
>>>>>>>
>>>>>>>    msDBFGetItemIndex(): Item 'NAME' not found.
>>>>>>>
>>>>>>> Where "NAME" is the attribute I'm trying to use for the
>>>>>> label.  It looks
>>>>>>> like the tileindex shapefile doesn't expose the 
>> attributes of the
>>>>>>> underlying shape tiles.  I've done an ogrinfo on the
>>>>>> shapefiles that I'd
>>>>>>> generated, and the attributes are there.
>>>>>>>
>>>>>>> Can I expose the attributes through the tileindex?
>>>>>>>
>>>>>>> How?