[Fwd: RE: [UMN_MAPSERVER-USERS] shp2tile question]

Wed Feb 15 12:54:45 PST 2006

-------- Original Message --------
Subject: RE: [UMN_MAPSERVER-USERS] shp2tile question
Date: Wed, 15 Feb 2006 12:37:22 -0800
From: Eric Shuman <erics at ameri-title.com>
Reply-To: <erics at ameri-title.com>
To: 'Stephen Woodbridge' <woodbri at swoodbridge.com>

Hi Steve,

So I have finally gotten DEBUG info working, and have made the shptree index
for my shapefiles. I have also gotten PostGIS working and have imported a
shapefile to the DB.  I have created indexes for the geometry field as well
as the OID, and a TAXLOT attribute field, and then Vacuumed/Analyzed the DB.

I did see an improvement in draw time after creating the shptree index, but
I don't see much difference between an attribute query based on a shapefile
vs. postGIS.  My shapefile has about 88000 records. The .shp is about 43Megs
and the .dbf is about 53Megs in size.  What is "Really Huge"?

What I am trying to do is this:
1. Query "Lots" shapefile for a particular lot number.
2. Find all the lots that are within a distance of the lot in question.
3. Gather attribute info about the near-by lots.
4. Zoom map extent to the area in question.

I am either hitting this large shapefile or the postGIS table (or some combo
of the two) 3 times in the MAP file to get everything symbolized properly.
Total draw times are running around 9 seconds.

Is there any other techniques to speed things up?  Am I missing anything?

Thanks,
~Eric

-----Original Message-----
From: Stephen Woodbridge [mailto:woodbri at swoodbridge.com]
Sent: Thursday, February 02, 2006 5:11 PM
To: Eric Shuman
Cc: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] shp2tile question

Eric,

Here is how you should optimize mapserver:

1) every *.shp file should have a shptree index, (ie: a corresponding
*.qix) if you haven't done this then don't think about anything else.
This includes on all TILEINDEX shp files also.
2) If you have large numbers of objects that you need to do attribute
queries on consider using postgis for just those layers.
3) I will let others comment on rasters and how to optimize them
4) Set "DEBUG ON" in the mapfile and create a LOG file, then do a
typical mapdraw and look at how long each layer took to draw. Then tell
us about that if it is not obvious how to fix that performance issue.
5) As a last resort, if you have REALLY HUGE shapefiles and they are
drawing very slowly then consider using shp2tile, but talk about here on
the list first.

Let us know if this helps and what your problems are after you have
tried this.

-Steve W.
   http://imaptools.com/

Eric Shuman wrote:
> I also have a similar problem. Don, did you come up with a good solution?
> I'm still new to mapserver and have not delved into the indexing
> capabilities yet. Can someone enlighten me?
>
> Here is my situation:
> I have a map set up that displays cadastral data as well as other layers
> such as streets and points of interest.  From PHP I query the map by
> attribute for a taxlot ID.  The map layer itself is set to the full extent
> of the data, and uses the parcel layer multiple times to show the query
> results, as well as for other symbology purposes.  When the lot is found
the
> map zooms to the extent of the results.  The data extent covers an entire
> county so the layers are fairly large, and the map takes awhile to draw.
>
> The questions:
> In all cases the final extent is pretty small, maybe 1 square mile max.
> Should I be tiling my parcels layer with the shp2tile/ogrtindex combo and
> then creating a shptree on the resulting index? OR Should I just create a
> shptree on the parcels layer as well as all the other layers?  Will the
> shptree help when doing an attribute query, or do I need to find a
different
> solution for speeding up the attribute query?  Any recomendations on how
to
> speed things up?
>
> In addition I am serving up aerial photos.  I have a 1Gig MrSid, but don't
> really want to get into building mapserver from scratch to add the
> functionality.  So my though was to convert the SID to 2500 8Meg TIFF's,
and
> then run a gdaltindex on them, then maybe a shptree on the index. Is this
a
> sound plan? Other options? Does anyone have experience with the speed
> differences of MrSid vs. TIFF?
>
> Thanks for the help!
>
> ~Eric Shuman
>
>
>
> On Mon, 18 Apr 2005 10:20:23 -0400, Don Deskins <dldeskins at GMAIL.COM>
wrote:
>
>
>>Steve,
>>
>>Thank you for your reply.
>>
>>I have experimented with different numbers and after reading a bit
>>more, I have discovered that it make no difference (in my case) how I
>>divide it up.  As a matter of fact, my application performs SLOWER the
>>more files that I have.
>>
>>What I am trying to do is to display an area with parcels of land.  I
>>have two identical layers... one to display all of the parcels and the
>>other to only display the selected parcel.  It is extremely slow when
>>querying for by parcel id.  Because of this, I need to put the second
>>layer (at least) in a database so that the parcelid field can be
>>indexed.
>>
>>Thanks,
>>
>>Don
>>
>>On 4/11/05, Stephen Woodbridge <woodbri at swoodbridge.com> wrote:
>>
>>>Don,
>>>
>>>Like I said there is no magic number. If you are most showing all your
>>>parcels most of the time then there is not value in having it tiled. One
>>>of the MOST important performance issues is to make sure all your *.shp
>>>files have a *.qix file.
>>>
>>>Try a different set of tiling parameters and test it again.
>>>
>>>Did you have *.qix on you original shapefile? If not go back to that and
>>>add one (shptree file.shp) and see how it performs untiled.
>>>
>>>-Steve
>>>
>>>Don Deskins wrote:
>>>
>>>>Bart and Steve,
>>>>
>>>>My shapefiles are showing up now... thanks!
>>>>
>>>>Now I have a time problem... It seems to take more time to generate
>>>>the image than it did before.  I am checking for errors now.
>>>>
>>>>Thanks again.
>>>>
>>>>Don
>>>>
>>>>
>>>>On Apr 11, 2005 3:28 PM, Bart van den Eijnden <bartvde at xs4all.nl> wrote:
>>>>
>>>>
>>>>>Hi Don,
>>>>>
>>>>>check the dbf of your generated tileindex (the LOCATION field). I
usually
>>>>>use absolute paths in there. Are there absolute or relative paths to
the
>>>>>files in there?
>>>>>
>>>>>Also use TYPE POLYGON. Are you zoomed in enough to have your layer
showing
>>>>>up, since you are using a MAXSCALE setting?
>>>>>
>>>>>Best regards,
>>>>>Bart
>>>>>
>>>>>On Mon, 11 Apr 2005 15:24:45 -0400, Don Deskins <dldeskins at gmail.com>
>>>>>wrote:
>>>>>
>>>>>
>>>>>
>>>>>>Bart (and Steve),
>>>>>>
>>>>>>Thanks.
>>>>>>
>>>>>>I have my files created and tried to reference this index in my map
file:
>>>>>>
>>>>>> ####################################
>>>>>> # parcels
>>>>>> LAYER
>>>>>>   NAME 'tileindex'
>>>>>>   TYPE TILEINDEX
>>>>>>   DATA '/path/to/tileindex'
>>>>>>   MAXSCALE 40000
>>>>>> END
>>>>>>
>>>>>> LAYER
>>>>>>   NAME 'SCAP_Cert2004_ParcelsUTM16M'
>>>>>>   STATUS ON
>>>>>>   TYPE POLYGON
>>>>>>   TILEINDEX '/path/to/tileindex'
>>>>>>   #TILEITEM 'LOCATION'
>>>>>>   MAXSCALE 40000
>>>>>> END
>>>>>> ####################################
>>>>>>
>>>>>>I get no error but my shapes are not showing up... do you see a
>>>>>>problem with the above map?
>>>>>>
>>>>>>Don
>>>>>>
>>>>>>On Apr 11, 2005 2:58 PM, Bart van den Eijnden <bartvde at xs4all.nl>
wrote:
>>>>>>
>>>>>>
>>>>>>>Hi,
>>>>>>>
>>>>>>>It is tile4ms, not tile2ms, probably a typo by Steve.
>>>>>>>
>>>>>>>It is a Mapserver utility to make a tileindex. Alternatively, you can
>>>>>>>also
>>>>>>>use ogrtindex for vector layers and gdaltindex for raster layers.
>>>>>>>
>>>>>>>Best regards,
>>>>>>>Bart
>>>>>>>
>>>>>>
>>>>>--
>>>>>Using Opera's revolutionary e-mail client: http://www.opera.com/m2/
>>>>>
>>>>
>>>>
>>>>
>>>>--
>>>>Don Deskins
>>>>
>>>
>>>
>>
>>--
>>Don Deskins
>
>