[mapserver-users] problem with processquerytemplate and large selection sets

[Lime, Steve D (MNIT)]

I mean did you add timing statements to the PHP code or did you muck about in the MapServer/MapScript source. It's possible that PHP is slow, not the specific MapServer calls.

From: Bistrais, Bob [mailto:Bob.Bistrais at maine.gov]
Sent: Tuesday, April 23, 2013 12:48 PM
To: Lime, Steve D (MNIT); mapserver-users at lists.osgeo.org
Subject: RE: [mapserver-users] problem with processquerytemplate and large selection sets

Not sure what you meant by internal/external timings?

I do know that we are running MapServer in a FastCGI mode.  I'm also in a GeoMoose framework, so I'm not sure how to run it outside of MapScript (open for suggestions).  But one thing I do know is that the QueryByAttributes occurs in a reasonable time (a couple seconds on 1100 records).  Where it's bogging down is in a for or a while loop, iterating through the result set.

From: Lime, Steve D (MNIT) [mailto:Steve.Lime at state.mn.us]
Sent: Tuesday, April 23, 2013 1:04 PM
To: Bistrais, Bob; mapserver-users at lists.osgeo.org<mailto:mapserver-users at lists.osgeo.org>
Subject: RE: [mapserver-users] problem with processquerytemplate and large selection sets

Hmmm... so this is just a "TEMPLATE 'thisfile'" setting in the mapfile and not using the new-style templates. It really should be quite efficient. Are your timings internal or external to MapServer? I'm curious if this could be a PHP issue.

Can you do your query against the CGI/FastCGI and by-pass MapScript altogether.

Steve

From: Bistrais, Bob [mailto:Bob.Bistrais at maine.gov]
Sent: Tuesday, April 23, 2013 11:57 AM
To: Lime, Steve D (MNIT); mapserver-users at lists.osgeo.org<mailto:mapserver-users at lists.osgeo.org>
Subject: RE: [mapserver-users] problem with processquerytemplate and large selection sets

Here it is:


<!-- MapServer Template -->


<b>SiteID:</b> [SiteID]<br/>
<b>Basic Structure</b> [BasicStruc]<br/>
<b>Barrier Class:</b> [BarrierCla]<br/>
<b>Site Status:</b> [SiteStatus]<br/>
<b>Date:</b> [Date_]<br/>
<b>Stream:</b> [Stream]<br/>
<b>Town:</b> [Town]<br/>
<b>UTM East:</b> [UTMEast]<br/>
<b>UTM North:</b> [UTMNorth]<br/>
<b>Road Name:</b> [RoadName]<br/>
<b>Road Type:</b> [RoadType]<br/>
<b>Road Class:</b> [RoadClass]<br/>
<b>Number of Culverts:</b> [NumberOfCu]<br/>
<b>Material:</b> [Material]<br/>
<b>Specific Struct Type:</b> [SpecificSt]<br/>
<b>Inlet Cond:</b> [InletCondi]<br/>
<b>Blocked:</b> [Blocked]<br/>
<b>Inlet Water Depth:</b> [InletWater]<br/>
<b>Inlet Span:</b> [InletSpan]<br/>
<b>Inlet Clearance:</b> [InletClear]<br/>
<b>Crossing Structure Length:</b> [CrossingSt]<br/>
<b>Outlet Cond:</b> [OutletCond]<br/>
<b>Outlet Water Depth:</b> [OutletWate]<br/>
<b>Outlet Drop:</b> [OutletDrop]<br/>
<b>Outlet Span:</b> [OutletSpan]<br/>
<b>OutletClearace:</b> [OutletClea]<br/>
<b>Crossing Substrate:</b> [CrossingSu]<br/>
<b>Substrate Cond:</b> [SubstrateC]<br/>
<b>Condition:</b> [Condition]<br/>
<b>Fill Height:</b> [FillHeight]<br/>
<b>Bankfull Width:</b> [BankfullWi]<br/>
<b>BAT Count US:</b> [batCountUS]<br/>
<b>BAT Count DS:</b> [batCountDS]<br/>
<b>Sum Habitat Units:</b> [Sum_HabUni]<br/>
<b>US Blocked Miles:</b> [USBlockedM]<br/>
<b>US Blocked Alwife Pond Ac:</b> [USBlockedA]<br/>
<b>Comments: </b>[Comments]<br/>

<b>Photos:</b><span style='text-decoration:underline;color:#0000FF;'><a onClick="window.open('php/barrierPhotos.php?SiteID=[SiteID]&layer=barriers','Stream Viewer Photos','width=900,height=790,scrollbars=yes,dependent=yes,toolbar=no,location=no,status=no,directories=no,menubar=no,status=no,resizable=yes');return false;">[SiteID]</a></span><br/>

<hr/>

From: Lime, Steve D (MNIT) [mailto:Steve.Lime at state.mn.us]
Sent: Tuesday, April 23, 2013 12:56 PM
To: Bistrais, Bob; mapserver-users at lists.osgeo.org<mailto:mapserver-users at lists.osgeo.org>
Subject: RE: [mapserver-users] problem with processquerytemplate and large selection sets

What does the template look like? --Steve

From: mapserver-users-bounces at lists.osgeo.org<mailto:mapserver-users-bounces at lists.osgeo.org> [mailto:mapserver-users-bounces at lists.osgeo.org] On Behalf Of Bistrais, Bob
Sent: Friday, April 19, 2013 3:29 PM
To: mapserver-users at lists.osgeo.org<mailto:mapserver-users at lists.osgeo.org>
Subject: [mapserver-users] problem with processquerytemplate and large selection sets

I've run into a problem when running queries with a large number of results (in my test case, over 1000 records, point locations, shape file).  After some testing, I found the problem wasn't the query itself- I did a queryByAttributes and it told me the records were selected within a couple seconds.  On further investigation, the problem seems to be when calling the processquerytemplate method.  The process takes a long time, often timing out beyond 30 seconds.  I am also suspicious that the result set is getting processed in such a way that we are reiterating over records.

Has anyone had a similar problem, and are there any suggestions ?

Here is a code excerpt, I hope it's enough to help...

$layer->queryByAttributes($queryItem,$queryExp,MS_MULTIPLE);
$numResults = $layer->getNumResults();
$layer->open();

for($i=0;$i<$layer->getNumResults();$i++){
    $shape=$layer->getShape($layer->getResult($i));
    $foundShapes[] = $shape;
    $fields = array();
    array_push($fields, 'wkt_geometry text');
    foreach($attributes as $attribute) {
                    array_push($fields, $attribute.' text');
    }
    array_push($results,$map->processquerytemplate(array(), false));

$layer->close();


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20130423/d3d010ca/attachment.htm>