Attributes in tile indexes?

[Ed McNierney]

Jeff -

And how does the geographic extent of the tiled shapefiles you're using
compare to the geographic extent of the map requests you're using in your
test?

     - Ed

-- 
Ed McNierney
President and Chief Mapmaker
Maps a la carte, Inc. / TopoZone.com
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
Phone: (978) 251-4242
Fax: (978) 251-1396
ed at topozone.com

> From: Jeff Dege <jdege at KORTERRA.COM>
> Reply-To: Jeff Dege <jdege at KORTERRA.COM>
> Date: Mon, 23 Apr 2007 14:19:20 -0500
> To: <MAPSERVER-USERS at LISTS.UMN.EDU>
> Conversation: [UMN_MAPSERVER-USERS] Attributes in tile indexes?
> Subject: Re: [UMN_MAPSERVER-USERS] Attributes in tile indexes?
> 
> In my particular case, I'm only using this layer when zoomed in tightly.
> 
> The shapefile in question contains street data.  There are millions of
> features in the original shapefile.  But I have very much smaller
> shapefiles containing only the major roads, only the highways, and only
> the interestates, that I use at wider zoom levels.  It's only at the
> tight zooms that I will draw the individual streets, so trying to read
> large numbers of tiles is not an issue.
> 
> What's surprised me in this case is that it's the first time where only
> drawing detail on close-in zooms, and adding spatial indexes to the
> shapefiles with shptree wasn't enough to get reasonable performance.
> 
> I'm currently running shp2tile with different arguments, trying to see
> how speed vs. number of files trade off against one another.  But with a
> half-gig shapefile, each trial is taking a while.
> 
> My current fastest attempt cut the time to draw to 1/4 of what I
> originally had.  Which isn't fast enough.
> 
>> -----Original Message-----
>> From: UMN MapServer Users List
>> [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On Behalf Of Ed McNierney
>> Sent: Monday, April 23, 2007 1:05 PM
>> To: MAPSERVER-USERS at LISTS.UMN.EDU
>> Subject: Re: [UMN_MAPSERVER-USERS] Attributes in tile indexes?
>> 
>> Jeff -
>> 
>> In addition to Steve's suggestion, please keep in mind that
>> what you are
>> trying to do is reduce the amount of data MapServer needs to
>> read in order
>> to retrieve the data you need.  Think about your data
>> carefully and see
>> whether you're achieving that goal.
>> 
>> First, splitting the shapefile into multiple files and
>> indexing them with a
>> TILEINDEX helps MapServer select a subset of that data to be
>> used for a
>> given map draw.  If MapServer has to open most or all of
>> those shapefiles
>> anyway, then the TILEINDEX isn't going to do a thing!
>> 
>> Since your method of producing shapefile tiles will minimize
>> overlap between
>> files, you should also check the relationship of your map's
>> spatial extent
>> with those tiles.  Ideally, each tile should be (roughly)
>> somewhat larger
>> than the extent of the map view.  With rectangularly-tiled
>> data, you cannot
>> avoid situations where the map view is centered on a corner
>> and you need to
>> open four different input shapefiles.  If each rectangular
>> tile is, however,
>> slightly larger than the map image size, then you will never
>> get a situation
>> that is WORSE than four input files.  If you render a zoomed-out view
>> covering dozens of input files, your TILEINDEX isn't going to
>> help much.
>> 
>> Think about how your TILEINDEX and split shapefiles are
>> behaving.  Have you
>> substantially eliminated most of your input data from
>> consideration in a
>> map-drawing request?  If not, then you need to go back and
>> think about how
>> to restructure your tiling and indexing, since that's the
>> whole goal of
>> those techniques.
>> 
>>      - Ed
>> 
>> -- 
>> Ed McNierney
>> President and Chief Mapmaker
>> Maps a la carte, Inc. / TopoZone.com
>> 73 Princeton Street, Suite 305
>> North Chelmsford, MA  01863
>> Phone: (978) 251-4242
>> Fax: (978) 251-1396
>> ed at topozone.com
>> 
>>> From: Stephen Woodbridge <woodbri at SWOODBRIDGE.COM>
>>> Reply-To: Stephen Woodbridge <woodbri at SWOODBRIDGE.COM>
>>> Date: Mon, 23 Apr 2007 11:53:56 -0400
>>> To: <MAPSERVER-USERS at LISTS.UMN.EDU>
>>> Subject: Re: [UMN_MAPSERVER-USERS] Attributes in tile indexes?
>>> 
>>> Jeff,
>>> 
>>> Make SURE all your tiles have a spatial index *.qix and the
>> tileindex
>>> also has a spatial index. These can be built with
>>> 
>>>    shptree file.shp
>>> 
>>> -Steve W.
>>> 
>>> Jeff Dege wrote:
>>>> OK - I knew I had to be doming something stupid.
>>>> 
>>>> To tell a layer to draw using a tileindex shapefile, you
>> include the
>>>> commands:
>>>> 
>>>>    TILEINDEX "tileindex"
>>>>    TILEITEM "LOCATION"
>>>> 
>>>> Using:
>>>> 
>>>>    DATA "tileindex"
>>>> 
>>>> won't work.
>>>> 
>>>> Now, after all of that, drawing with the split shapefiles
>> is slower than
>>>> with the original.  So there's still some playing to do.
>>>>  
>>>> 
>>>>> -----Original Message-----
>>>>> From: UMN MapServer Users List
>>>>> [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On Behalf Of Jeff Dege
>>>>> Sent: Monday, April 23, 2007 9:53 AM
>>>>> To: MAPSERVER-USERS at LISTS.UMN.EDU
>>>>> Subject: Re: [UMN_MAPSERVER-USERS] Attributes in tile indexes?
>>>>> 
>>>>> Well, I've been looking at this in more detail, and
>> finding that the
>>>>> tileindex layer isn't drawing at all.  With all other layers
>>>>> turned off,
>>>>> when I use shp2img with an extent that causes the tile layer
>>>>> to draw, I
>>>>> get a blank image.  When I change the layer to use the original
>>>>> shapefile, I get the data displayed.  (And the layer takes 40
>>>>> seconds to
>>>>> draw, even after I've run shptree on the shapefile, which
>> is why I've
>>>>> been playing around with tiling.)
>>>>> 
>>>>> I identified the individual tile shapefile that contains
>> the area I'm
>>>>> trying to draw and used it directly, and it works fine.  So the
>>>>> splitting apart seems to be working.  But the tileindex
>>>>> shapefile seems
>>>>> to be broken.
>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: UMN MapServer Users List
>>>>>> [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On Behalf Of Stephen
>>>>> Woodbridge
>>>>>> Sent: Saturday, April 21, 2007 7:15 AM
>>>>>> To: MAPSERVER-USERS at LISTS.UMN.EDU
>>>>>> Subject: Re: [UMN_MAPSERVER-USERS] Attributes in tile indexes?
>>>>>> 
>>>>>> Jeff,
>>>>>> 
>>>>>> This should just work. There is nothing special you should
>>>>>> have to do. I
>>>>>> do it all the time with my mapfiles. I use shp2tile to break
>>>>>> mine up but 
>>>>>> that should not make a difference.
>>>>>> 
>>>>>> First I would try changing your layer to use just one of
>>>>> the smaller 
>>>>>> shape files without the tileindex and see if that works and
>>>>>> that you can
>>>>>> display the labels.
>>>>>> 
>>>>>> You might also want to post info about what OS, version of
>>>>> mapserver, 
>>>>>> and the LAYER block in question.
>>>>>> 
>>>>>> -Steve W
>>>>>> 
>>>>>> Jeff Dege wrote:
>>>>>>> I have a shape file that's too big to draw maps with -
>>>>> takes far too
>>>>>>> long.
>>>>>>> 
>>>>>>> In an attempt to speed things up, I split it into 16
>>>>>> smaller shapefiles,
>>>>>>> using:
>>>>>>> 
>>>>>>>    ogr2ogr -f "ESRI Shapefile" -spat xmin ymin xmax
>> ymax newXX.shp
>>>>>>> old.shp
>>>>>>> 
>>>>>>> Then did a shptree on each of these new files, then a
>>>>> shptinidex to
>>>>>>> create a single tile index shapefile on them, and tried to
>>>>>> build a map.
>>>>>>> It works fine, so long as I don't try to display labels.
>>>>>> But if I do, I
>>>>>>> get errors:
>>>>>>> 
>>>>>>>    msDBFGetItemIndex(): Item 'NAME' not found.
>>>>>>> 
>>>>>>> Where "NAME" is the attribute I'm trying to use for the
>>>>>> label.  It looks
>>>>>>> like the tileindex shapefile doesn't expose the
>> attributes of the
>>>>>>> underlying shape tiles.  I've done an ogrinfo on the
>>>>>> shapefiles that I'd
>>>>>>> generated, and the attributes are there.
>>>>>>> 
>>>>>>> Can I expose the attributes through the tileindex?
>>>>>>> 
>>>>>>> How?
>>