[postgis-devel] Something wrong with build

[Kevin Neufeld]

  Ok, after some troubleshooting, I've traced the problem to this commit.

http://trac.osgeo.org/postgis/changeset/5466/trunk/doc/postgis.xml

If I revert this change,
   make chunked-html jumps from 20 minutes to 1.
   make comments     jumps from 15 minutes to 0.
   make pdf          jumps from 22 minutes to 8.

Perhaps the corresponding issue (#471) should be reopened and different 
solution found.

-- Kevin


On 10/13/2010 1:57 PM, Paragon Corporation wrote:
> Have any idea how long its been going on.  I could take out some of the
> sections I added to  postgis_aggs_mm.xml.xsl  to see if that improves
> things, but the documentation however you look at it is significantly larger
> in 2.0.  Just the Raster section alone adds about 50 pages.
>
> Though like I mentioned on our local doesn't take that long (probably about
> 2 minutes at most), but we don't have the doc book dtd etc included so
> perhaps the extra dtd validation is eating up resources.
>
> -----Original Message-----
> From: Kevin Neufeld [mailto:kneufeld at refractions.net]
> Sent: Wednesday, October 13, 2010 4:51 PM
> To: Paragon Corporation
> Cc: 'PostGIS Development Discussion'
> Subject: Re: Something wrong with build
>
>    In PostGIS 1.5, the whole thing builds in 10 minutes.
>
> The time differences are in the xsltproc command.  IE., the "make
> chunked-html" command does this:
>
> /usr/bin/xsltproc ./xsl/postgis_aggs_mm.xml.xsl postgis.xml>
> postgis_aggs_mm.xml
>
> This runs practically instantaneously in 1.5, but takes 10 minutes in trunk.
> Since building the docs makes several calls to xsltproc, it all adds up to
> about an hour in trunk.
>
> Either the postgis.xml file is significantly larger in trunk and it's
> exceeding some memory threshold in xsltproc, or the xml file is malformed
> and xsltproc is confused (though no errors/warnings are issued).
>
> Separate volumes may not be such a bad idea.
>
> -- Kevin
>
> On 10/13/2010 11:59 AM, Paragon Corporation wrote:
>> Kevin,
>>
>> For what its worth, doing the
>>
>> Make comments
>>
>> On our Xp box takes about 2 minutes.  We are running a newer version
>> of xsltproc than you.
>> -- ours is --
>> Using libxml 20703, libxslt 10124 and libexslt 813 xsltproc was
>> compiled against libxml 20701, libxslt 10124 and libexslt 813 libxslt
>> 10124 was compiled against libxml 20701 libexslt 813 was compiled
>> against libxml 20701
>> --- end --
>>
>> When I cheat by copying in a blank  postgis_aggs_mm.xml   -- it takes
> about
>> 15-30 seconds to build both the postgis_comments.sql and
>> postgis_raster.sql
>>
>> --
>>
>> I suppose this may be my fault :( -- the postgis_aggs_mm.xml.xsl is
>> doing a tad bit more than it used to in PostGIS 1.5.
>>
>> 2 more columns for the function matrix (so much for my dreams of
>> having a raster function matrix - though that wouldn't be quite as
>> useful anyway) Another 50 someodd functions and an additional what's
>> new section (hmm a what's new broken out into two because we have too
>> many new functions and enhanced functions)
>>
>> Our manual for PostGIS 2.0 is already about 100 pages longer than the
>> PostGIS 1.5.  At this rate, we might have to start breaking it out
>> into volumes :)
>>
>>
>>    How does the speed compare to building PostGIS 1.5 by the way?  I
>> imagine its much longer.
>>
>> Thanks,
>> Regina
>>
>> -----Original Message-----
>> From: Kevin Neufeld [mailto:kneufeld at refractions.net]
>> Sent: Wednesday, October 13, 2010 1:32 PM
>> To: Paragon Corporation
>> Cc: PostGIS Development Discussion
>> Subject: Re: Something wrong with build
>>
>>     Right, so the current autobuild process is taking about 60 minutes
>> to complete.
>>
>> The build itself happens within minutes.  Building the documentation
>> on the other hand is what is dragging the process out.
>>
>> # takes about 20 minutes
>> make -e chunked-html 2>   doc-errors.log
>>
>> # takes about 15 minutes
>> make comments 2>>   doc-errors.log
>>
>> # takes about 22 minutes
>> make pdf
>>
>>
>> During chuck-html task, it looks like we're stalling on this command
>> (stalls for about 10 minutes, idle cpu, i/o):
>> /usr/bin/xsltproc ./xsl/postgis_aggs_mm.xml.xsl postgis.xml>
>> postgis_aggs_mm.xml
>>
>> and this (stalls here too for about 10 minutes):
>> /usr/bin/xsltproc \
>>      --param section.autolabel 1 \
>>      --param section.label.includes.component.label 1 \
>>      --param chunk.section.depth 0 \
>>      --param generate.section.toc.level 1 \
>>      --param funcsynopsis.style ansi \
>>      --param admon.graphics 1 \
>>      --param admon.textlabel 0 \
>>      --param simplesect.in.toc 0 \
>>      --param use.id.as.filename 1 \
>>      --param chunk.quietly 1 \
>>      --stringparam html.stylesheet style.css  \
>>      --output html/ \
>>      --stringparam saxon.character.representation decimal \
>>      /usr/share/sgml/docbook/xsl-stylesheets/html/chunk.xsl \
>>      postgis-out.xml
>>
>> Then, on the comments task (taking about 15 minutes):
>> /usr/bin/xsltproc ./xsl/postgis_comments.sql.xsl postgis.xml>
>> postgis_comments.sql /usr/bin/xsltproc ./xsl/raster_comments.sql.xsl
>> postgis.xml>   raster_comments.sql
>>
>> The pdf task used to take only 5-10 minutes, now we're up to 22
>> minutes, stalling on the dblatex command.
>>
>> Anyone have ideas why xsltproc is getting slower and slower?  The
>> build machine is a standalone PostGIS build box ...  load average is
> minimal.
>> Is it just the configuration / software versions on this box?  Are
>> others able to build the docs in a reasonable amount of time?
>>
>> For reference:
>> $ xsltproc --version
>> Using libxml 20626, libxslt 10117 and libexslt 813 xsltproc was
>> compiled against libxml 20626, libxslt 10117 and libexslt 813 libxslt
>> 10117 was compiled against libxml 20626 libexslt 813 was compiled
>> against libxml 20626
>>
>>
>> Cheers,
>> Kevin
>>
>> On 10/12/2010 2:06 PM, Paragon Corporation wrote:
>>> Kevin,
>>>
>>> Can you take a look and see what is taking the build so long.
>>>
>>> http://office.refractions.net:1500/view/PostGIS%20Trunk/job/PostGIS-t
>>> r
>>> unk-bu
>>> ild/
>>>
>>> Last  change was trying to include raster_comments build in the
>>>
>>> make comments
>>>
>>> I think it broke last time I tried that too so had to roll back the
>> change.
>>> Not giving an error just stalling as far as I can tell.
>>>
>>>
>