tileindexed layer crashing...

[Jeff Portwine]

So far I have not used any spatial database like postgis or anything like
that...  and in fact I was trying to avoid that because it's another level
of complexity adding to my learning curve here.    I've just been using
whatever shapefiles I could find on the internet that would suit my neads
and displaying them and up until now that has worked just fine...  Then I
plot points on the map (representing customers) by doing a query on our
customer tables and getting the latitudes and longitudes of the results of
hte query and projecting those points onto the map...

I'm not sure that I'd benefit from the complexity of something like postGIS
even though i'm sure it would technically be "better".   I'll never be
getting a ton of hits on my mapserver app though so I don't think it's too
important.

----- Original Message -----
From: "Ethan Alpert" <ealpert at DIGITALGLOBE.COM>
To: <MAPSERVER-USERS at LISTS.UMN.EDU>
Sent: Friday, March 18, 2005 1:08 PM
Subject: Re: [UMN_MAPSERVER-USERS] tileindexed layer crashing...


Yep it's very important if you're using shapefiles to have one class of
feature per shapefile.

However if you're putting these in a DB it makes sense to combine them,
provide appropriate indexes and leverage the spatial indexing.

-e

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
Behalf Of Ed McNierney
Sent: Friday, March 18, 2005 10:47 AM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] tileindexed layer crashing...


Jeff -

I would get rid of the expression classification for now.  Try to get
one thing working at a time - if you do three things and once and it
breaks, it's hard to tell where the problem is.

However, even if there were no roads visible in your first map,
MapServer didn't know that unless it processed your layer.  It's
reasonable to think that MapServer worked correctly.  Your TILEINDEX
helped MapServer select only the TIGER file(s) it needed to read, and
ignore the rest.

Please realize that if you use an EXPRESSION to select features,
MapServer still has to read *every single feature* in the input files
and test them against your expression.  If you try to do that for the
whole Northeast, MapServer is still being asked to read *every* road in
those states even if it ends up selecting and drawing only two of them.

Once you get things working, you should look at what you're asking
MapServer to do.  If you are going to be only looking at a small subset
of roads that match certain attributes, don't do that on every map
request - preprocess your data into separate files, selecting out the
parts you want.  It doesn't make sense to compel MapServer to repeatedly
parse input data you don't want to see!

        - Ed

Ed McNierney
President and Chief Mapmaker
TopoZone.com / Maps a la carte, Inc.
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396
ed at topozone.com

-----Original Message-----
From: Jeff Portwine [mailto:jdport at veritime.com]
Sent: Friday, March 18, 2005 12:30 PM
To: Ed McNierney; MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] tileindexed layer crashing...

Sorry for not mentioning that it didn't have a problem when I first
reloaded
the map.   There were no roads visible in that view, and so  I thought
that
was probably why it was ok until I zoomed out again.

I did forget to put in a min/max scale line in there... but I wouldn't
have thought that would matter as I was only showing roads that matched
the expression [CFCC] = A15... and it is only for a selected area of the
US, not
the whole country.   My map is of the NE part of the US basically from
Pennsylvania to Maine.

As I said when I originally added the layer, my map was in a zoomed in
state and there were no roads visible (i'm not sure whether there should
have been or not)..  I then zoomed to the full extent of the area so
that I could see
if any roads were being drawn at all.   I never tried panning or
scrolling
around to see if they were drawing because at that point I had the crash
and I won't really be able to experiment any more until i'm able to get
mapserver installed someplace where I can't cause any problems if things
go badly.

-Jeff

----- Original Message -----
From: "Ed McNierney" <ed at topozone.com>
To: "Jeff Portwine" <jdport at VERITIME.COM>;
<MAPSERVER-USERS at LISTS.UMN.EDU>
Sent: Friday, March 18, 2005 11:38 AM
Subject: RE: [UMN_MAPSERVER-USERS] tileindexed layer crashing...


Jeff -

Thanks for the confirmation.  And thank you for adding that very
valuable information you didn't mention the first time <g>.  If your map
worked fine the first time, ignore my entire previous email - your index
and shapefiles seem to be working fine.  If you have no MINSCALE or
MAXSCALE settings on your layers, is your attempt to view "the entire
area" really a map request to draw every single road in the United
States?  You should not be surprised if that effort is a little
overwhelming for your server.  If you can scroll around at your original
zoom level and see streets correctly, then your setup should be fine and
you need to move on to wisely classifying and selecting your data so
each map draw uses a reasonable and appropriate subset of your data.

You're misunderstanding my comments about your TILEINDEX.  Each of your
individual TIGER files is a shapefile.  A TILEINDEX file is ANOTHER
shapefile made up of rectangular polygons, each describing the extent of
one of your TIGER files.  The index is used to quickly determine which
source TIGER files overlap the requested output map area at all - those
are the only ones that need to be opened to try to draw the final map.

- Ed

Ed McNierney
President and Chief Mapmaker
TopoZone.com / Maps a la carte, Inc.
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396
ed at topozone.com

________________________________

From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
Behalf Of Jeff Portwine
Sent: Friday, March 18, 2005 11:25 AM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] tileindexed layer crashing...


First, some nit-picking.  I presume you did something first to create a
"data_dir.in" file, right?  And there's a space between xargs and "-n",
correct?  The latter shouldn't make much difference.

Yea, there is a space between xargs and -n, i mistyped in the email. The
data_dir.in file was just a listing of all the tiger data files, created
with  "find /homedir/tiger -name "*.shp" > data_dir.in".

Does your tiger-roads shapefile look OK?  Do the filesizes look
plausible?  Can you view it in a GIS application (it's just a regular
shapefile)?

The file sizes looked reasonable to me... but I don't really know how
big they should be.   I don't have any other GIS applications atm, but I
could try to download something to try to view the shapefile.

As a diagnostic step, I'd suggest changing TILEINDEX to DATA in your
LAYER definition, and the TYPE to POLYGON.  The tile index is a polygon
shapefile with a bunch of rectangles, each describing the bounding box
of one input file.  Use this technique to see if the tile index file
itself looks reasonable.  If it does, then your problem's likely to be
with your TIGER data - otherwise, the TILEINDEX file needs fixing.

 I am not entirely sure what you mean here ... how to diagnose the tile
index.   I actually thought I had to use TILEINDEX to view a shapefile
created with this method.    If I can just use it like any other
shapefile I probably don't even need to tile it... i'm not trying to do
roads for the entire country or anything, though in theory the tiling is
probably better.



When I first added the layer, my map was in a zoomed in state and it
seemed ok... it was when I hit the "recenter" button and it tried to go
back to view the entire area that it blew up.   Perhaps tiling isn't so
good for that situation since i'm looking at the whole area anyway...



-Jeff