<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">I mean did you add timing statements to the PHP code or did you muck about in the MapServer/MapScript source. It’s possible that PHP is slow, not the specific MapServer calls.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Bistrais, Bob [mailto:Bob.Bistrais@maine.gov]
<br>
<b>Sent:</b> Tuesday, April 23, 2013 12:48 PM<br>
<b>To:</b> Lime, Steve D (MNIT); mapserver-users@lists.osgeo.org<br>
<b>Subject:</b> RE: [mapserver-users] problem with processquerytemplate and large selection sets<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Not sure what you meant by internal/external timings?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I do know that we are running MapServer in a FastCGI mode. I’m also in a GeoMoose framework, so I’m not sure how to run it outside of MapScript (open for suggestions). But one thing I do know is that the QueryByAttributes
occurs in a reasonable time (a couple seconds on 1100 records). Where it’s bogging down is in a for or a while loop, iterating through the result set.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Lime, Steve D (MNIT) [<a href="mailto:Steve.Lime@state.mn.us">mailto:Steve.Lime@state.mn.us</a>]
<br>
<b>Sent:</b> Tuesday, April 23, 2013 1:04 PM<br>
<b>To:</b> Bistrais, Bob; <a href="mailto:mapserver-users@lists.osgeo.org">mapserver-users@lists.osgeo.org</a><br>
<b>Subject:</b> RE: [mapserver-users] problem with processquerytemplate and large selection sets<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Hmmm… so this is just a “TEMPLATE ‘thisfile’” setting in the mapfile and not using the new-style templates. It really should be quite efficient. Are your timings internal or external to MapServer? I’m curious
if this could be a PHP issue.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Can you do your query against the CGI/FastCGI and by-pass MapScript altogether.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Steve<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Bistrais, Bob [<a href="mailto:Bob.Bistrais@maine.gov">mailto:Bob.Bistrais@maine.gov</a>]
<br>
<b>Sent:</b> Tuesday, April 23, 2013 11:57 AM<br>
<b>To:</b> Lime, Steve D (MNIT); <a href="mailto:mapserver-users@lists.osgeo.org">
mapserver-users@lists.osgeo.org</a><br>
<b>Subject:</b> RE: [mapserver-users] problem with processquerytemplate and large selection sets<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Here it is:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><!-- MapServer Template --><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>SiteID:</b> [SiteID]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Basic Structure</b> [BasicStruc]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Barrier Class:</b> [BarrierCla]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Site Status:</b> [SiteStatus]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Date:</b> [Date_]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Stream:</b> [Stream]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Town:</b> [Town]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>UTM East:</b> [UTMEast]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>UTM North:</b> [UTMNorth]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Road Name:</b> [RoadName]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Road Type:</b> [RoadType]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Road Class:</b> [RoadClass]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Number of Culverts:</b> [NumberOfCu]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Material:</b> [Material]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Specific Struct Type:</b> [SpecificSt]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Inlet Cond:</b> [InletCondi]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Blocked:</b> [Blocked]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Inlet Water Depth:</b> [InletWater]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Inlet Span:</b> [InletSpan]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Inlet Clearance:</b> [InletClear]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Crossing Structure Length:</b> [CrossingSt]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Outlet Cond:</b> [OutletCond]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Outlet Water Depth:</b> [OutletWate]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Outlet Drop:</b> [OutletDrop]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Outlet Span:</b> [OutletSpan]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>OutletClearace:</b> [OutletClea]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Crossing Substrate:</b> [CrossingSu]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Substrate Cond:</b> [SubstrateC]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Condition:</b> [Condition]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Fill Height:</b> [FillHeight]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Bankfull Width:</b> [BankfullWi]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>BAT Count US:</b> [batCountUS]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>BAT Count DS:</b> [batCountDS]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Sum Habitat Units:</b> [Sum_HabUni]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>US Blocked Miles:</b> [USBlockedM]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>US Blocked Alwife Pond Ac:</b> [USBlockedA]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Comments: </b>[Comments]<br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><b>Photos:</b><span style='text-decoration:underline;color:#0000FF;'><a onClick="window.open('php/barrierPhotos.php?SiteID=[SiteID]&layer=barriers','Stream Viewer Photos','width=900,height=790,scrollbars=yes,dependent=yes,toolbar=no,location=no,status=no,directories=no,menubar=no,status=no,resizable=yes');return
false;">[SiteID]</a></span><br/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><hr/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Lime, Steve D (MNIT) [<a href="mailto:Steve.Lime@state.mn.us">mailto:Steve.Lime@state.mn.us</a>]
<br>
<b>Sent:</b> Tuesday, April 23, 2013 12:56 PM<br>
<b>To:</b> Bistrais, Bob; <a href="mailto:mapserver-users@lists.osgeo.org">mapserver-users@lists.osgeo.org</a><br>
<b>Subject:</b> RE: [mapserver-users] problem with processquerytemplate and large selection sets<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">What does the template look like? --Steve<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a href="mailto:mapserver-users-bounces@lists.osgeo.org">mapserver-users-bounces@lists.osgeo.org</a> [<a href="mailto:mapserver-users-bounces@lists.osgeo.org">mailto:mapserver-users-bounces@lists.osgeo.org</a>]
<b>On Behalf Of </b>Bistrais, Bob<br>
<b>Sent:</b> Friday, April 19, 2013 3:29 PM<br>
<b>To:</b> <a href="mailto:mapserver-users@lists.osgeo.org">mapserver-users@lists.osgeo.org</a><br>
<b>Subject:</b> [mapserver-users] problem with processquerytemplate and large selection sets<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’ve run into a problem when running queries with a large number of results (in my test case, over 1000 records, point locations, shape file). After some testing, I found the problem wasn’t the query itself- I did a queryByAttributes and
it told me the records were selected within a couple seconds. On further investigation, the problem seems to be when calling the processquerytemplate method. The process takes a long time, often timing out beyond 30 seconds. I am also suspicious that the
result set is getting processed in such a way that we are reiterating over records.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Has anyone had a similar problem, and are there any suggestions ?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here is a code excerpt, I hope it’s enough to help…<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">$layer->queryByAttributes($queryItem,$queryExp,MS_MULTIPLE);<o:p></o:p></p>
<p class="MsoNormal">$numResults = $layer->getNumResults();<o:p></o:p></p>
<p class="MsoNormal">$layer->open();<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">for($i=0;$i<$layer->getNumResults();$i++){<o:p></o:p></p>
<p class="MsoNormal"> $shape=$layer->getShape($layer->getResult($i));<o:p></o:p></p>
<p class="MsoNormal"> $foundShapes[] = $shape;<o:p></o:p></p>
<p class="MsoNormal"> $fields = array();<o:p></o:p></p>
<p class="MsoNormal"> array_push($fields, 'wkt_geometry text');<o:p></o:p></p>
<p class="MsoNormal"> foreach($attributes as $attribute) {<o:p></o:p></p>
<p class="MsoNormal"> array_push($fields, $attribute.' text');<o:p></o:p></p>
<p class="MsoNormal"> }<o:p></o:p></p>
<p class="MsoNormal"> array_push($results,$map->processquerytemplate(array(), false));<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">$layer->close();<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>