[Mapserver-users] optimizing shapefiles with the mapserver utilities?

Tue Feb 11 13:57:42 PST 2003

--------------7D1EACC80EA19CDF20F996E1
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

I can't speak from experience, but I can on where I'm heading.

I work in NYC and often need to render street data at great detail. I
found an ArcView script that added 2 fields to my shapefile's .dbf file.
midpoint_x and midpoint_y. This populated the two fields with the
midpoint longitude and latitude for each polyline in the shape.

When I review the data, my needs are best met by having the data for the
downstate NYC area at the top of the shapefile. So, I used shpsort to
sort midpoint_y (the latitude) in ascending order.

Like I say, I'm not done yet, but when I open the shapefile in ArcView,
roads are displayed south to north, which is what I expected. I guess
then that using shptree will improve any performance gain.

When I'm done with other data issues, I'll put this all together in a
mapfile. I guess I'll find out THEN if it was worth the time!

Kieran

Avenue script header info follows:

' Name: PolyLineMidPoint
'
' Date: 04.01.2001
'
' Updated:
'
' Author: Nick Bauer
'         CRC Freshwater Ecology
'         University of Canberra
'         Australia
'         email: bauer at lake.canberra.edu.au
'
' Title:  MidPoint of a PolyLine
'
' Topics: Themes, PolyLine, MidPoint,
'
' Description: Returns the co-ordinates of the midpoint (center) of a
polyline. These are added
'              as two new fields to the attribute table of the polyline
theme.
'
' Requires: The view theme to be the active doc. The polyline as the
active theme.
'
' Self:  nil
'
' Returns: nil

Eric Davies wrote:

>
> I've got some fairly large shapefiles (> 100M) which I'm asking
> mapserver to display.
> I'd like to improve the speed at which it processes these large files.
>
> I've already done the level-of-detail stuff (setting the minimum and
> maximum scales at
> which a layer is displayed), and I've built up the quadtree index with
> shptree. However,
> the latter didn't make much change to the speed.
>
> My suspicion is that my shapefiles are relatively unordered, so that
> elements for any particular small region are likely scattered across
> the entire shapefile. What I'd like to do is order the shapefile so
> that geometries that are geographically close together are more likely
> to be close together in the shapefile.
>
> The functionality to do perform this ordering is already present in
> the utilities: shptree maps geometry locations to a linear key (the
> tile number), and sortshp can reorder a shapefile based on a key.
> However, I don't see any way to make the two talk to each other.
>
> It shouldn't be too difficult for me modify sortshp to read in the
> shptree produced index file, but before  I do, I should ask if there
> is a way to do this type of thing out of the box, without writing more
> code.
>
> Thank you.
> Eric.
>
> **********************************************
>
> Eric Davies, M.Sc.
> Barrodale Computing Services Ltd.
> Tel: (250) 472-4372 Fax: (250) 472-4373
> Web: http://www.barrodale.com
> Email: eric at barrodale.com
> **********************************************
> Mailing Address:
> P.O. Box 3075 STN CSC
> Victoria BC Canada V8W 3W2
>
> Shipping Address:
> Hut R, McKenzie Avenue
> University of Victoria
> Victoria BC Canada V8W 3W2
> **********************************************
>
>

--------------7D1EACC80EA19CDF20F996E1
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: 7bit

<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
<html>
I can't speak from experience, but I can on where I'm heading.
I work in NYC and often need to render street data at great detail.
I found an ArcView script that added 2 fields to my shapefile's .dbf file.
midpoint_x and midpoint_y. This populated the two fields with the midpoint
longitude and latitude for each polyline in the shape.
When I review the data, my needs are best met by having the data for
the downstate NYC area at the top of the shapefile. So, I used shpsort
to sort midpoint_y (the latitude) in ascending order.
Like I say, I'm not done yet, but when I open the shapefile in ArcView,
roads are displayed south to north, which is what I expected. I guess then
that using shptree will improve any performance gain.
When I'm done with other data issues, I'll put this all together in
a mapfile. I guess I'll find out THEN if it was worth the time!
Kieran
Avenue script header info follows:
' Name: PolyLineMidPoint
 '
 ' Date: 04.01.2001
 '
 ' Updated:
 '
 ' Author: Nick Bauer
 '         CRC Freshwater Ecology
 '         University of Canberra
 '         Australia
 '         email: bauer at lake.canberra.edu.au
 '
 ' Title:  MidPoint of a PolyLine
 '
 ' Topics: Themes, PolyLine, MidPoint,
 '
 ' Description: Returns the co-ordinates of the midpoint (center) of
a polyline. These are added
 '             
as two new fields to the attribute table of the polyline theme.
 '
 ' Requires: The view theme to be the active doc. The polyline as the
active theme.
 '
 ' Self:  nil
 '
 ' Returns: nil
  
  
  
  
Eric Davies wrote:
<blockquote TYPE=CITE> 
 I've got some fairly large shapefiles (> 100M) which I'm asking mapserver
to display.
 I'd like to improve the speed at which it processes these large files.
 I've already done the level-of-detail stuff (setting the minimum and
maximum scales at
 which a layer is displayed), and I've built up the quadtree index with
shptree. However,
 the latter didn't make much change to the speed.
My suspicion is that my shapefiles are relatively unordered, so that
elements for any particular small region are likely scattered across the
entire shapefile. What I'd like to do is order the shapefile so that geometries
that are geographically close together are more likely to be close together
in the shapefile.
The functionality to do perform this ordering is already present in
the utilities: shptree maps geometry locations to a linear key (the tile
number), and sortshp can reorder a shapefile based on a key. However, I
don't see any way to make the two talk to each other.
It shouldn't be too difficult for me modify sortshp to read in the shptree
produced index file, but before  I do, I should ask if there is a
way to do this type of thing out of the box, without writing more code.
Thank you.
 Eric.
**********************************************
 <x-sigsep>
</x-sigsep>Eric Davies, M.Sc.
 Barrodale Computing Services Ltd.
 Tel: (250) 472-4372 Fax: (250) 472-4373
 Web:
<a href="http://www.barrodale.com/" eudora="autourl">http://www.barrodale.com</a>
 Email: eric at barrodale.com
 **********************************************
 Mailing Address:
 P.O. Box 3075 STN CSC
 Victoria BC Canada V8W 3W2
Shipping Address:
 Hut R, McKenzie Avenue
 University of Victoria
 Victoria BC Canada V8W 3W2
 **********************************************
  
  </blockquote>
</html>

--------------7D1EACC80EA19CDF20F996E1--