Attributes in tile indexes?

[Jeff Dege]

In my particular case, I'm only using this layer when zoomed in tightly.

The shapefile in question contains street data.  There are millions of
features in the original shapefile.  But I have very much smaller
shapefiles containing only the major roads, only the highways, and only
the interestates, that I use at wider zoom levels.  It's only at the
tight zooms that I will draw the individual streets, so trying to read
large numbers of tiles is not an issue.

What's surprised me in this case is that it's the first time where only
drawing detail on close-in zooms, and adding spatial indexes to the
shapefiles with shptree wasn't enough to get reasonable performance.

I'm currently running shp2tile with different arguments, trying to see
how speed vs. number of files trade off against one another.  But with a
half-gig shapefile, each trial is taking a while.

My current fastest attempt cut the time to draw to 1/4 of what I
originally had.  Which isn't fast enough.

> -----Original Message-----
> From: UMN MapServer Users List 
> [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On Behalf Of Ed McNierney
> Sent: Monday, April 23, 2007 1:05 PM
> To: MAPSERVER-USERS at LISTS.UMN.EDU
> Subject: Re: [UMN_MAPSERVER-USERS] Attributes in tile indexes?
> 
> Jeff -
> 
> In addition to Steve's suggestion, please keep in mind that 
> what you are
> trying to do is reduce the amount of data MapServer needs to 
> read in order
> to retrieve the data you need.  Think about your data 
> carefully and see
> whether you're achieving that goal.
> 
> First, splitting the shapefile into multiple files and 
> indexing them with a
> TILEINDEX helps MapServer select a subset of that data to be 
> used for a
> given map draw.  If MapServer has to open most or all of 
> those shapefiles
> anyway, then the TILEINDEX isn't going to do a thing!
> 
> Since your method of producing shapefile tiles will minimize 
> overlap between
> files, you should also check the relationship of your map's 
> spatial extent
> with those tiles.  Ideally, each tile should be (roughly) 
> somewhat larger
> than the extent of the map view.  With rectangularly-tiled 
> data, you cannot
> avoid situations where the map view is centered on a corner 
> and you need to
> open four different input shapefiles.  If each rectangular 
> tile is, however,
> slightly larger than the map image size, then you will never 
> get a situation
> that is WORSE than four input files.  If you render a zoomed-out view
> covering dozens of input files, your TILEINDEX isn't going to 
> help much.
> 
> Think about how your TILEINDEX and split shapefiles are 
> behaving.  Have you
> substantially eliminated most of your input data from 
> consideration in a
> map-drawing request?  If not, then you need to go back and 
> think about how
> to restructure your tiling and indexing, since that's the 
> whole goal of
> those techniques.
> 
>      - Ed
> 
> -- 
> Ed McNierney
> President and Chief Mapmaker
> Maps a la carte, Inc. / TopoZone.com
> 73 Princeton Street, Suite 305
> North Chelmsford, MA  01863
> Phone: (978) 251-4242
> Fax: (978) 251-1396
> ed at topozone.com
> 
> > From: Stephen Woodbridge <woodbri at SWOODBRIDGE.COM>
> > Reply-To: Stephen Woodbridge <woodbri at SWOODBRIDGE.COM>
> > Date: Mon, 23 Apr 2007 11:53:56 -0400
> > To: <MAPSERVER-USERS at LISTS.UMN.EDU>
> > Subject: Re: [UMN_MAPSERVER-USERS] Attributes in tile indexes?
> > 
> > Jeff,
> > 
> > Make SURE all your tiles have a spatial index *.qix and the 
> tileindex
> > also has a spatial index. These can be built with
> > 
> >    shptree file.shp
> > 
> > -Steve W.
> > 
> > Jeff Dege wrote:
> >> OK - I knew I had to be doming something stupid.
> >> 
> >> To tell a layer to draw using a tileindex shapefile, you 
> include the
> >> commands:
> >> 
> >>    TILEINDEX "tileindex"
> >>    TILEITEM "LOCATION"
> >> 
> >> Using:
> >> 
> >>    DATA "tileindex"
> >> 
> >> won't work.
> >> 
> >> Now, after all of that, drawing with the split shapefiles 
> is slower than
> >> with the original.  So there's still some playing to do.
> >>  
> >> 
> >>> -----Original Message-----
> >>> From: UMN MapServer Users List
> >>> [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On Behalf Of Jeff Dege
> >>> Sent: Monday, April 23, 2007 9:53 AM
> >>> To: MAPSERVER-USERS at LISTS.UMN.EDU
> >>> Subject: Re: [UMN_MAPSERVER-USERS] Attributes in tile indexes?
> >>> 
> >>> Well, I've been looking at this in more detail, and 
> finding that the
> >>> tileindex layer isn't drawing at all.  With all other layers
> >>> turned off,
> >>> when I use shp2img with an extent that causes the tile layer
> >>> to draw, I
> >>> get a blank image.  When I change the layer to use the original
> >>> shapefile, I get the data displayed.  (And the layer takes 40
> >>> seconds to
> >>> draw, even after I've run shptree on the shapefile, which 
> is why I've
> >>> been playing around with tiling.)
> >>> 
> >>> I identified the individual tile shapefile that contains 
> the area I'm
> >>> trying to draw and used it directly, and it works fine.  So the
> >>> splitting apart seems to be working.  But the tileindex
> >>> shapefile seems
> >>> to be broken.
> >>> 
> >>>> -----Original Message-----
> >>>> From: UMN MapServer Users List
> >>>> [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On Behalf Of Stephen
> >>> Woodbridge
> >>>> Sent: Saturday, April 21, 2007 7:15 AM
> >>>> To: MAPSERVER-USERS at LISTS.UMN.EDU
> >>>> Subject: Re: [UMN_MAPSERVER-USERS] Attributes in tile indexes?
> >>>> 
> >>>> Jeff,
> >>>> 
> >>>> This should just work. There is nothing special you should
> >>>> have to do. I 
> >>>> do it all the time with my mapfiles. I use shp2tile to break
> >>>> mine up but 
> >>>> that should not make a difference.
> >>>> 
> >>>> First I would try changing your layer to use just one of
> >>> the smaller 
> >>>> shape files without the tileindex and see if that works and
> >>>> that you can 
> >>>> display the labels.
> >>>> 
> >>>> You might also want to post info about what OS, version of
> >>> mapserver, 
> >>>> and the LAYER block in question.
> >>>> 
> >>>> -Steve W
> >>>> 
> >>>> Jeff Dege wrote:
> >>>>> I have a shape file that's too big to draw maps with -
> >>> takes far too
> >>>>> long.
> >>>>> 
> >>>>> In an attempt to speed things up, I split it into 16
> >>>> smaller shapefiles,
> >>>>> using:
> >>>>> 
> >>>>>    ogr2ogr -f "ESRI Shapefile" -spat xmin ymin xmax 
> ymax newXX.shp
> >>>>> old.shp
> >>>>> 
> >>>>> Then did a shptree on each of these new files, then a
> >>> shptinidex to
> >>>>> create a single tile index shapefile on them, and tried to
> >>>> build a map.
> >>>>> It works fine, so long as I don't try to display labels.
> >>>> But if I do, I
> >>>>> get errors:
> >>>>> 
> >>>>>    msDBFGetItemIndex(): Item 'NAME' not found.
> >>>>> 
> >>>>> Where "NAME" is the attribute I'm trying to use for the
> >>>> label.  It looks
> >>>>> like the tileindex shapefile doesn't expose the 
> attributes of the
> >>>>> underlying shape tiles.  I've done an ogrinfo on the
> >>>> shapefiles that I'd
> >>>>> generated, and the attributes are there.
> >>>>> 
> >>>>> Can I expose the attributes through the tileindex?
> >>>>> 
> >>>>> How?
>