Contours again.

[Ed McNierney]

Bob -

If you are looking for a 1600x1300-foot area and producing an 800x650-pixel map, something's slowing you down.  There's no reason that request should take that long, and I share Frank's suspicion that the problem's somewhere else.

Can you try my earlier suggestion of copying the data to a non-network drive, just for comparison?  The TILEINDEX route should help you, but as Frank says your data appears to be well-structured for handling this kind of display request quite easily.

     - Ed

----- Original Message -----
From: Bob Basques <bob.basques at CI.STPAUL.MN.US>
To: MAPSERVER-USERS at LISTS.UMN.EDU
Sent: Fri, 15 Oct 2004 15:15:17 -0400
Subject: Re: [UMN_MAPSERVER-USERS] Contours again.


> Ed McNierney wrote:
> 
> >Bob -
> >
> >Thanks for the info - here are some thoughts:
> >
> >1. Is there a particular reason you need to merge all those tiles into a=
> > single shapefile=3F  Since you're not merging the polylines inside that=
> > shapefile, there isn't much benefit (and a bit of harm) in merging them=
> >.  If you keep each file as a separate tile, you can use a TILEINDEX to =
> >combine all the tiles into a single logical layer - as if they were a si=
> >ngle shapefile.  You can then use shptree on each of those individual ti=
> >les to create a spatial index for it.  The combination bascially provide=
> >s a two-level index that can be very efficient.
> >
> >
> Actaully that was my first thought.  I do this already with Aerial Photo
> layers.  This was really an experiment to guage performance.  It's still
> impressive (to me) that the request comes back as quick as it does,
> considering the data amounts.
> 
> >2. The contour interval has an effect, but so does the granularity of th=
> >e data.  If your source data is at a higher resolution than the output i=
> >mage then you can often end up drawing multiple polyline segments all of=
> > which live inside a single output pixel!
> >
> That's not an issue at the test viewings.  I'm also planning on
> generating multiple resolutions of the contours data for just this
> reason.  Doing a Contour map of the whole City for example.
> 
> > This is where generalizing th=
> >e data to a level appropriate to the output can help.
> >
> Understood.  I do this now for the aerial photo's too. where I have
> multiple tile sets for the same LAYER at different source resaolutions
> that are used for different viewing scales.   I've not tried this same
> technique with the SHP file though.  Are there any pitfalls to look out for?
> 
> > If your data has =
> >a much higher resolution than the output image, it will simply slow thin=
> >gs down - 100 polylines in one pixel looks exactly the same as only 1 po=
> >lyline in that same pixel.  There's no improvement in quality.
> >
> >3. What is the physical image size (in pixels) and the geographic extent=
> > (in feet) of your test image=3F
> >
> >
> 1600 x 1300 feet (extents) and 800 x 650 pixels.   Right in the correct
> neighborhood for the source data.  I'm also labelling the lines, but I
> don't think that's a significant overhead overall.
> 
> The biggest issue seems to be the fact that the Layer resided on a
> network mounted device.  (This is the same for the Aerial photo datasets
> as well.)  The smaller I can make the initial loading the better I
> think.  So the TileIndex seems to be the way to go, in my case anyways.
> 
> bobb
> 
> >     - Ed
> >
> >----- Original Message -----
> >From: Bob Basques <bob.basques at CI.STPAUL.MN.US>
> >To: MAPSERVER-USERS at LISTS.UMN.EDU
> >Sent: Fri, 15 Oct 2004 13:56:30 -0400
> >Subject: Re: [UMN=5FMAPSERVER-USERS] Contours again.
> >
> >
> >
> >
> >>Ed McNierney wrote:
> >>=20
> >>
> >>
> >>>Frank -
> >>>
> >>>Thanks for chiming in!  You're right, of course; Bob made the comment=
> >>>
> >>>
> > ea=3D
> >
> >
> >>>rlier:
> >>>
> >>>"I actually started out with something like that in the AutoCAD world=
> >>>
> >>>
> >.
> >
> >
> >>>Each tile (1/2 sq. mile) was brought together into a single coverage.=
> >>>
> >>>
> > I
> >
> >
> >>>didn't do any joins on the linework, I just let them be seperate
> >>>entities. I figured that was better than joining them anyway."
> >>>
> >>>
> >>>
> >>>from which I concluded that these were a lot of small, independent ob=
> >>
> >>
> >jec=3D
> >
> >
> >>>ts.  But I'm now not sure that's really the case, so, Bob, can you co=
> >>>
> >>>
> >nfi=3D
> >
> >
> >>>rm the arrangement of your data=3D3F
> >>>
> >>>
> >>>
> >>>
> >>Yes, the source data was all organized into 1/2 x 1/2 mile tiles (2000=
> >>
> >>
> > x
> >
> >
> >>2000 ft).   When assembled into a single SHP file they were NOT joined
> >>together.  They are still polylines though.  I'm starting to wonder no=
> >>
> >>
> >w
> >
> >
> >>though about exploding them into their respective line segments.  The
> >>data size will double though if I do that with all the extra end point=
> >>
> >>
> >s
> >
> >
> >>to take manage.
> >>=20
> >>Even though the contours are chopped up by 1/2 mile increments, they a=
> >>
> >>
> >re
> >
> >
> >>still quite large polylines, with hundreds of vertices each.   I could
> >>chop them up further, down to 500 foot grid tile perhaps.   The big
> >>concern for me was labelling.  I didn't want to end up with seperate
> >>segments that were labelled to often.   I could get away with a single
> >>label per object with the tiling method of aggregation.  Seperating
> >>every segment out seems like going backwards for some reason.  :c)
> >>=20
> >>I don't recall now if I mentioned this or not, but the contours are at
> >>one foot increments, quite dense and 560+ megs for the *.SHP file.  Th=
> >>
> >>
> >e
> >
> >
> >>qix file was 4+ megs.  As a matter of fact, this is the densest datase=
> >>
> >>
> >t
> >
> >
> >>that I need to publish ( so far )  I have the DEM next that was used t=
> >>
> >>
> >o
> >
> >
> >>generate the Contours.  :c)
> >>=20
> >>bobb
> >>=20
> >>
> >>
> >>>I ran into a similar situation a few years ago with US street map dat=
> >>>
> >>>
> >a, =3D
> >
> >
> >>>which was originally organized as one shapefile per layer type per st=
> >>>
> >>>
> >ate=3D
> >
> >
> >>>.  I was surprised at how noticeable the improvement was when I switc=
> >>>
> >>>
> >hed=3D
> >
> >
> >>>to one shapefile per layer type per COUNTY, as it helped prune large=
> >>>
> >>>
> > ob=3D
> >
> >
> >>>jects (interstates, rivers).
> >>>
> >>>    - Ed
> >>>
> >>>----- Original Message -----
> >>>From: Frank Warmerdam <warmerdam at pobox.com>
> >>>To: Ed McNierney <ed at TOPOZONE.COM>
> >>>Cc: MAPSERVER-USERS at LISTS.UMN.EDU
> >>>Sent: Fri, 15 Oct 2004 12:45:34 -0400
> >>>Subject: Re: [UMN=3D5FMAPSERVER-USERS] Contours again.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>>Ed McNierney wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>Bob -
> >>>>>=3D20
> >>>>>If things got 25% faster, the index is being used - you don't need =
> >>>>>
> >>>>>
> >t=3D
> >
> >
> >>>>>
> >>>>>
> >>>o do
> >>>
> >>>
> >>>
> >>>
> >>>>anything other than create it.  Do you have a feel for what level of=
> >>>>
> >>>>
> > b=3D
> >
> >
> >>>>
> >>>>
> >>>andwidth
> >>>
> >>>
> >>>
> >>>
> >>>>you're getting from the NAS server=3D3F  Roughly what percentage of =
> >>>>
> >>>>
> >the =3D
> >
> >
> >>>>
> >>>>
> >>>vectors in
> >>>
> >>>
> >>>
> >>>
> >>>>the shapefile are being used to draw your test image=3D3F
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>=3D20
> >>>>>Don't mess around with different values yet - just use the default.=
> >>>>>
> >>>>>
> > =3D
> >
> >
> >>>>>
> >>>>>
> >>>It sounds
> >>>
> >>>
> >>>
> >>>
> >>>>like you're suffering from either excessively-detailed data (needs
> >>>>simplification for the scales you're using), a test that draws a lar=
> >>>>
> >>>>
> >ge
> >
> >
> >>>>percentage of the vectors in the file, and/or poor throughput to the=
> >>>>
> >>>>
> > f=3D
> >
> >
> >>>>
> >>>>
> >>>ile
> >>>
> >>>
> >>>
> >>>
> >>>>server.  A simple test would be to copy the shapefile to the local d=
> >>>>
> >>>>
> >is=3D
> >
> >
> >>>>
> >>>>
> >>>k just to
> >>>
> >>>
> >>>
> >>>
> >>>>see if there's a difference.
> >>>>=3D20
> >>>>=3D20
> >>>>Guys,
> >>>>=3D20
> >>>>I haven't followed this discussion very closely, so I may have misse=
> >>>>
> >>>>
> >d
> >
> >
> >>>>something.  But one issue I have encountered with contours is that t=
> >>>>
> >>>>
> >he=3D
> >
> >
> >>>>
> >>>>
> >>>y
> >>>
> >>>
> >>>
> >>>
> >>>>often end up being very large polygons.  For instance, a "zero eleva=
> >>>>
> >>>>
> >ti=3D
> >
> >
> >>>>
> >>>>
> >>>on"
> >>>
> >>>
> >>>
> >>>
> >>>>coastal contour might well be continental sized.  Really big "rings"=
> >>>>
> >>>>
> > g=3D
> >
> >
> >>>>
> >>>>
> >>>et
> >>>
> >>>
> >>>
> >>>
> >>>>pushed way up the the spatial index tree and will have to be read fr=
> >>>>
> >>>>
> >om
> >
> >
> >>>>disk and rendered even if the current map view is just somewhere ins=
> >>>>
> >>>>
> >id=3D
> >
> >
> >>>>
> >>>>
> >>>e
> >>>
> >>>
> >>>
> >>>
> >>>>the contour ring such that the contour won't actually be visible, bu=
> >>>>
> >>>>
> >t
> >
> >
> >>>>the bounding rectangles of the contour and the view area will overla=
> >>>>
> >>>>
> >p.
> >
> >
> >>>>=3D20
> >>>>If this is Bob's problem, he will really be much better off to split=
> >>>>
> >>>>
> > t=3D
> >
> >
> >>>>
> >>>>
> >>>he
> >>>
> >>>
> >>>
> >>>
> >>>>contours into shortish line segments instead of keeping as massive s=
> >>>>
> >>>>
> >in=3D
> >
> >
> >>>>
> >>>>
> >>>gle
> >>>
> >>>
> >>>
> >>>
> >>>>geometries as this would allow the spatial index to localize things
> >>>>much better.
> >>>>=3D20
> >>>>I didn't speak up about this before because it wasn't clear if this =
> >>>>
> >>>>
> >ha=3D
> >
> >
> >>>>
> >>>>
> >>>s
> >>>
> >>>
> >>>
> >>>
> >>>>already been done.
> >>>>=3D20
> >>>>Best regards,
> >>>>--=3D20
> >>>>---------------------------------------+----------------------------=
> >>>>
> >>>>
> >--=3D
> >
> >
> >>>>
> >>>>
> >>>--------
> >>>
> >>>
> >>>
> >>>
> >>>>I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam@=
> >>>>
> >>>>
> >po=3D
> >
> >
> >>>>
> >>>>
> >>>box.com
> >>>
> >>>
> >>>
> >>>
> >>>>light and sound - activate the windows | http://pobox.com/~warmerdam
> >>>>and watch the world go round - Rush    | Geospatial Programmer for R=
> >>>>
> >>>>
> >en=3D
> >
> >
> >>>>
> >>>>
> >>>t
> >>>
> >>>
> >>>
> >>>
> >>>>=3D20
> >>>>=3D20
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>=20
> >>
> >>
> >
> >
> >
>