Contours again.

[Ed McNierney]

Bob -

If things got 25% faster, the index is being used - you don't need to do anything other than create it.  Do you have a feel for what level of bandwidth you're getting from the NAS server?  Roughly what percentage of the vectors in the shapefile are being used to draw your test image?

Don't mess around with different values yet - just use the default.  It sounds like you're suffering from either excessively-detailed data (needs simplification for the scales you're using), a test that draws a large percentage of the vectors in the file, and/or poor throughput to the file server.  A simple test would be to copy the shapefile to the local disk just to see if there's a difference.

     - Ed

----- Original Message -----
From: Bob Basques <bob.basques at CI.STPAUL.MN.US>
To: MAPSERVER-USERS at LISTS.UMN.EDU
Sent: Fri, 15 Oct 2004 11:01:17 -0400
Subject: Re: [UMN_MAPSERVER-USERS] Contours again.


> Ed McNierney wrote:
> 
> >Bob -
> >
> >After you run shptree you will see a new file with a .qix extension.  You
> should then rerun your tests and see a difference.
> >
> >
> It shaved off about 30 sec for the search.  Is there a way to verify
> whether the new index is even being used ?  I also tried with some
> different tree values (24, 30) and they didn't seem to make a difference
> either way., this is one reason I would like to verify if the index is
> even being used.  Does it need to be specified anywhere in a MAPfile or ???
> 
> bobb
> 
> >     - Ed
> >
> >----- Original Message -----
> >From: Bob Basques <bob.basques at ci.stpaul.mn.us>
> >To: Ed McNierney <ed at TOPOZONE.COM>
> >Sent: Thu, 14 Oct 2004 11:05:02 -0400
> >Subject: Re: Contours again.
> >
> >
> >
> >
> >>Ed McNierney wrote:
> >>
> >>
> >>
> >>>Bob -
> >>>
> >>>I'm not entirely sure from your reply that you understood that Armin's f=
> >>>irst suggestion is to simply use shptree to create a spatial index for y=
> >>>our single shapefile.  In the directory where your shapefile is stored, =
> >>>simply use the command
> >>>
> >>>   shptree shapefile.shp
> >>>
> >>>
> >>>
> >>>
> >>I tried this and nothing is produced, the prompt sits there (crewating
> >>index of new  LSB format) for a while then just returns.
> >>
> >>I'm running on REDHAT Fedora Core.
> >>
> >>Is there some special syntax to look out for?
> >>
> >>Some notes:  I have the data on a NAS mounted on the WebServer.  I'm
> >>running the SHPTREE command on the Server into the home directory of the
> >>login.  Don't know if this might be causing write problems or not.
> >>
> >>
> >>
> >>>and a file with the same base name and the extension .qix will be create=
> >>>d.  Then you can run your test again - creating the index will just take=
> >>>a few moments and you don't have to do anything with your data organiza=
> >>>tion.
> >>>
> >>>I'm assuming, by the way, that your test request is actually displaying =
> >>>only a small portion of the entire shapefile.
> >>>
> >>>
> >>>
> >>Yup, I plan on making a couple of generalized layers for the wider area
> >>views.
> >>
> >>
> >>
> >>> If you're trying to draw =
> >>>the entire 550 MB shapefile in a single image, you have different proble=
> >>>ms <g>.
> >>>
> >>>
> >>>
> >>>
> >>Nope.  Just trying to see if this is going to be a performance problem
> >>or not right now.
> >>
> >>bobb
> >>
> >>
> >>
> >>>    - Ed
> >>>
> >>>Ed McNierney
> >>>TopoZone.com
> >>>
> >>>----- Original Message -----
> >>>From: Bob Basques <bob.basques at CI.STPAUL.MN.US>
> >>>To: MAPSERVER-USERS at LISTS.UMN.EDU
> >>>Sent: Wed, 13 Oct 2004 17:56:00 -0400
> >>>Subject: Re: [UMN=5FMAPSERVER-USERS] Contours again.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>>Armin Burger wrote:
> >>>>=20
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>Bob,
> >>>>>
> >>>>>you could also try just creating a spatial index for the shapefile us=
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>ing s=3D
> >>>
> >>>
> >>>
> >>>
> >>>>>hptree. 2 minutes looks like there is no existing spatial index and m=
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>ore o=3D
> >>>
> >>>
> >>>
> >>>
> >>>>>r less all features of the shapefile are read all the time.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>Yes I think this is correct.
> >>>>=20
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>Maybe this ind=3D
> >>>>>ex already speeds up things sufficiently.
> >>>>>
> >>>>>Instead of splitting up the contour lines into seperate shapefiles th=
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>at ha=3D
> >>>
> >>>
> >>>
> >>>
> >>>>>ve then to be put together with a shapeindex, it might be an alternat=
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>ive t=3D
> >>>
> >>>
> >>>
> >>>
> >>>>>o intersect the contour lines with a regular grid. E.g. like the shee=
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>ts fo=3D
> >>>
> >>>
> >>>
> >>>
> >>>>>r topographic maps or a rectangular one with arbitrary extent you cre=
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>ate o=3D
> >>>
> >>>
> >>>
> >>>
> >>>>>n your own. That way very long lines with huge amount of vertices are=
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>spli=3D
> >>>
> >>>
> >>>
> >>>
> >>>>>t up into smaller chunks.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>I actually started out with something like that in the AutoCAD world.
> >>>>Each tile (1/2 sq. mile) was brought together into a single coverage. =
> >>>>
> >>>>
> >>>>
> >>>>
> >>>I
> >>>
> >>>
> >>>
> >>>
> >>>>didn't do any joins on the linework, I just let them be seperate
> >>>>entities.  I figured that was better than joining them anyway.
> >>>>=20
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>The time to read such features is much less than=3D
> >>>>>for the large ones, but you still have everything in a single shapef=
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>ile.
> >>>
> >>>
> >>>
> >>>
> >>>>>If you want just display the values as integer, it might be the easie=
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>st wa=3D
> >>>
> >>>
> >>>
> >>>
> >>>>>y to add a new integer column in the dbf file and copy the values fro=
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>m the=3D
> >>>
> >>>
> >>>
> >>>
> >>>>>float column. And then use the new integer column  for the labeling.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>Good idea, although I think I'll just get rid of the old column
> >>>>afterwards.  I tried some things with the data writer(AutoCAD Map) but
> >>>>nothing seemed to help with the labelling.
> >>>>=20
> >>>>I think your idea will work though.  I can load the DBF up and add a
> >>>>column easy enough.
> >>>>=20
> >>>>Thanks for the suggestions
> >>>>=20
> >>>>bobb
> >>>>=20
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>armin
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>All,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>Well I tried out a composite file before asking this question.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>I just ran a 550+meg SHP file for our Contour data.  There is only o=
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>ne
> >>>
> >>>
> >>>
> >>>
> >>>>>>attribute in the DBF, which is the elevation.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>a Typical request takes around 120 sec to complete.  so I'm going to=
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>try
> >>>
> >>>
> >>>
> >>>
> >>>>>>and tile the shp file out into smaller version and use a tile index =
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>on
> >>>
> >>>
> >>>
> >>>
> >>>>>>top of that.  Are there any pitfalls I should watch out for during t=
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>he
> >>>
> >>>
> >>>
> >>>
> >>>>>>process of tiling a SHP file=3F
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>Also, how do I label with a INTEGER vs a REAL for the elevations.  I=
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>'m
> >>>
> >>>
> >>>
> >>>
> >>>>>>getting  14 decimal places in the labels.  :c)
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>Thanks
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>bobb
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>=20
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >
> >
> >
> >
>