map performance

Wed Dec 6 11:20:39 PST 2006

What if you created a shapefile like I describe ahead of time (could use
MapScript to do it)? Algorithm might look like (for say counties):

1) make a copy of the base shapefile (shp/shx) files using system cp
command (or whatever) since your not screwing with the geometries
2) create a new DBF table based on the contents in the base shapefile
and add a column for each variable you want to map (number, no
decimals)
3) for each variable you want to map:
  a) do your mysql query
  b) loop through the results and find the right DBF record (based on
FIPS or whatever) and set the appropriate column value to, say, 1,2,3,4
(where those are your status values)

It might take a couple of minutes to build the data but it can be done
offline.

I've done this with species occurance data in Minnesota townships. For
each township I know if a species is present or not and can create range
maps very
fast. You're doing something similar but have more states than simply
yes/no. Same idea though.

In my case the mapfile is super simple since the layer definition is
the same for each variable, you just change the variable being mapped at
runtime.

Steve

>>> boice tomlin <boice at RUNSKIP.COM> 12/6/2006 12:23:41 PM >>>

The data comes from a MySQL database.  The primary function of the site
is to collect the data from users.  It is a PHP/MqSQL application.

The map is rendered each time for only a specific layer such as
orthos.

So I go to the database(MySQL) and grab all distinct county ids that
have a status of complete for that layer.

I build the regex string from that array of county ids and then use
phpmapscript to set the expression for the county layer.

I understand the data structure you explained below.  But I don't
understand how you would implement it.  Do you propose connecting to a
data source directly from the mapfile?

At this point the only way I know how to color a county green on the
map is by using a regular expression.  But I'm open to any method. 
Especially if its faster.

thanks,

boice

On Dec 6, 2006, at 12:39 PM, Steve Lime wrote:
Ok, I looked back through another email and saw the output. I think
the
bottleneck is in the large regular expressions. You'd be much better
off
with a data structure like I outlined below. How do you determine, for
example, which counties are do be drawn a certain way. That is, how do
you build the regex? Using attributes in the shapefile? From another
source?

Steve

boice tomlin <boice at RUNSKIP.COM> 12/6/2006 10:55:52 AM >>>

Hi Steve,

Assuming I was using static class items.  Do you think my method of
using a regular expression to specify which geographies to display is
a
good method?

Thanks for the tip on saving the map file.  Very cool.  It looks as I
expected or at least hoped.  Basically my original map file with a few
extra classes and regular expressions for the layers I display.

Someone else suggested a method using postgis that would pull in shape
data with less polygons for certain extents.

Couldn't I use a version of the shape files I have now with less
detail/polygons when rendering a map at larger extents?

I actually want to explore every method I can for optimization.  My
goal is a fairly descriptive national view in 1 tenth of a second.  Is
that too lofty?

thanks,

boice

On Dec 5, 2006, at 6:32 PM, Steve Lime wrote:
Might be interesting to see what the resulting mapfile (after your
dynamic work) looks like (use $map->save(...)).

We'd really need to know more about the data, how you're doing
classifications and such to comment more. There may be lots of ways
to make things go faster. From the looks of it for each geography
(city, county, tribe, state, nation) you have a bunch of variables
that
indicate
if a data theme is available. One could organize that data like so
(for
example state level data):

State  Hydrography  Watersheds ...
MN     0                    1 
WI      3                    2
IA       3                    0
FL       1                    3

Where 0 means no responce, 1 complete and so on. So in that case your
class definitions would always be the same you'd
be simply changing the variable you're mapping on (e.g. CLASSITEM),
and
you wouldn't need dynamic classes (or even MapScript for that
matter).  Just thinking out loud...

Steve

boice tomlin <boice at RUNSKIP.COM> 12/4/2006 1:24:58 PM >>>

Hello users,

The map located here;

http://gisinventory.net/status_maps.html 

takes a while to load.  At least at the national view where there is a
lot of area to render.

I am looking for alternative ways to generate the map that will
significantly improve performance.

Currently I am using php and looping through data and turning on
layers
as I go.  The PHP part is lightning fast.  But after I get the map
ready
mapserver takes several seconds to generate it.

I'm curious about alternative ways to handle this problem and wonder
if
anyone had comments on any of them.

1) modifying the shape files in some way so the layer information is
in
those files so that all mapserver has to do is load those files and
not
depend on the map files.
2) using a db such as postgres with postgis so that layer information
is available all in one compact source.

Right now I have to generate a bunch of dynamic classes in PHP using
the general method below.

$lyr = $this->ramona_map->getLayerByName("state_yes");
                $cla = $lyr->getClass(0);
                $cla->setExpression("/".$expression."/");
                $lyr->set("status", MS_ON);

I have to do this several hundred times to represent all of the data. 
The time it takes to generate the map seem proportionate to the amount
of layers I make visible.  And again this is on the mapserver side and
not PHP.  PHP does its part of the operation in thousandths of a
second.

anyone's thoughts are greatly appreciated.

-boice tomlin

////////////////////////////
Run Skip
http://runskip.com/ 

boice tomlin

boice at runskip.com 

503-528-6204

////////////////////////////
Run Skip
http://runskip.com/ 

boice tomlin

boice at runskip.com 

503-528-6204

////////////////////////////
Run Skip
http://runskip.com/ 

boice tomlin

boice at runskip.com 

503-528-6204