[mapguide-commits] r8401 - in trunk/Tools/Vagrant: . centos/x64 centos/x86 packer scripts sources

svn_mapguide at osgeo.org svn_mapguide at osgeo.org
Tue Oct 14 06:03:03 PDT 2014 

Author: jng
Date: 2014-10-14 06:03:03 -0700 (Tue, 14 Oct 2014)
New Revision: 8401

Added:
   trunk/Tools/Vagrant/README.txt
   trunk/Tools/Vagrant/make_boxes.bat
   trunk/Tools/Vagrant/packer/
   trunk/Tools/Vagrant/packer/centos6-amd64.json
   trunk/Tools/Vagrant/packer/centos6-i386.json
Modified:
   trunk/Tools/Vagrant/centos/x64/Vagrantfile
   trunk/Tools/Vagrant/centos/x86/Vagrantfile
   trunk/Tools/Vagrant/scripts/build.sh
   trunk/Tools/Vagrant/scripts/build_fdo.sh
   trunk/Tools/Vagrant/scripts/mginstallcentos.sh
   trunk/Tools/Vagrant/sources/README.txt
Log:
Vagrant build system updates:
 - Check in packer configurations and scripts. This can be used to build the vagrant base boxes we use to build MapGuide for the various distros. CentOS base boxes have been verified to be a suitable build environment for MapGuide. Ubuntu support will come later.
 - Add a new make_boxes.bat that builds all the vagrant base boxes with packer and register said boxes with vagrant.
 - Update CentOS vagrantfiles to assume the base boxes are produced from the above packer configurations.
 - #2487: Use xz compression for our tarballs. The CS-Map changes have resulted in a much heavier installation footprint, xz compression will help offset some of these gains, giving us a ~260MB tarball instead of a >300MB one with gzip.

Added: trunk/Tools/Vagrant/README.txt
===================================================================
--- trunk/Tools/Vagrant/README.txt	                        (rev 0)
+++ trunk/Tools/Vagrant/README.txt	2014-10-14 13:03:03 UTC (rev 8401)
@@ -0,0 +1,61 @@
+Vagrant Build Environment README
+================================
+
+This directory contains configurations and scripts to do the following:
+
+ 1. Create/register the necessary vagrant base boxes
+ 2. Build MapGuide through vagrant-provisioned VMs
+
+Requirements
+============
+
+ * Vagrant (http://www.vagrantup.com/)
+ * packer (http://www.packer.io/)
+ * Build host assumed to be Windows
+ * Read sources/README.txt for additional requirements
+
+Building base boxes
+===================
+
+Run make_boxes.bat. This will run packer and create the following vagrant base boxes
+
+ * CentOS 6.5 32-bit
+ * CentOS 6.5 64-bit
+ * Ubuntu 12.04 32-bit
+ * Ubuntu 12.04 64-bit
+
+These boxes will also be registered in vagrant as:
+
+ * centos6-i386
+ * centos6-amd64
+ * ubuntu12-i386
+ * ubuntu12-amd64
+
+Building MapGuide and FDO
+=========================
+
+Run build.bat, this will build MapGuide and FDO for:
+
+ * CentOS 6.5 32-bit
+ * CentOS 6.5 64-bit
+ * Ubuntu 12.04 32-bit
+ * Ubuntu 12.04 64-bit
+
+When each build completes, the build artifacts and assorted log files and unit test result XML files will be copied to these directories:
+
+ * centos/x86/build
+ * centos/x64/build
+ * ubuntu/x86/build
+ * ubuntu/x64/build
+
+For CentOS, the build will produce tarballs of FDO and MapGuide along with the necessary install scripts
+For Ubuntu, the build will produce debian package files of FDO and MapGuide along with the necessary install scripts
+
+At the end of the run, updated MapGuide and FDO source tarballs will be in sources/updated. You can move these tarballs up one level and
+overwrite the existing copies to ensure you have the most updated checkouts for the next build run.
+
+Troubleshooting
+===============
+
+If a given build does not produce the expected build artifacts, always check the build logs for any errors.
+

Modified: trunk/Tools/Vagrant/centos/x64/Vagrantfile
===================================================================
--- trunk/Tools/Vagrant/centos/x64/Vagrantfile	2014-10-10 09:03:25 UTC (rev 8400)
+++ trunk/Tools/Vagrant/centos/x64/Vagrantfile	2014-10-14 13:03:03 UTC (rev 8401)
@@ -10,7 +10,7 @@
   # please see the online documentation at vagrantup.com.
 
   # Every Vagrant virtual environment requires a box to build off of.
-  config.vm.box = "vagrant-centos64-x64"
+  config.vm.box = "centos6-amd64"
 
   # The url from where the 'config.vm.box' box will be fetched if it
   # doesn't already exist on the user's system.
@@ -183,8 +183,6 @@
     sudo mkdir -p /usr/include/asm
     sudo cp /mapguide_sources/atomic.h /usr/include/asm
 fi
-#echo [provision]: Checking all tools and libs are in place
-#yum install -y gcc-* gd-* automake bison byacc flex doxygen expat expat-devel libtool libjpeg-devel libpng libpng-devel libxml2 libxml2-devel openssl curl curl-devel libxslt libxslt-devel subversion java-1.7.0-openjdk java-1.7.0-openjdk-devel ant dos2unix openssh-server openldap-devel alsa-lib-devel pcre-devel unixODBC-devel libcom_err-devel krb5-devel openssl-devel mysql-devel postgresql-devel unixODBC
 echo [provision]: Clean line endings
 sudo dos2unix /vagrant/*.sh
 sudo dos2unix /vagrant/*.pl
@@ -293,7 +291,7 @@
     cd $SCRIPT_ROOT/mgdev/Server/src/Core
     sudo make unittest.sh
     # Run the test suites individually except for TileService/ResourceService for reasons already stated
-    for comp in CoordinateSystem FeatureService Geometry KmlService LogManager MdfModel Misc RenderingService ServerAdminService ServerManager ServiceManager SiteManager SiteService ProfilingService TransformMesh
+    for comp in CoordinateSystem FeatureService Geometry KmlService LogManager MappingService MdfModel Misc Performance RenderingService ServerAdminService ServerManager ServiceManager SiteManager SiteService ProfilingService TransformMesh
     do
         sudo -E ./mgserver test $comp UnitTestResults_${comp}.xml 2>&1 | tee ~/mapguide_${comp}_unit_test.log
         check_test
@@ -308,8 +306,8 @@
 echo [provision]: Copy log files to output dir
 sudo mv -f ~/*.log /vagrant/build
 echo [provision]: Copy tarballs to output dir
-sudo cp bin/*.tar.gz /vagrant/build
-sudo cp fdosdk-centos6-i386-${FDO_VER_FULL}_${FDO_REV}.tar.gz /vagrant/build
+sudo cp bin/*.tar.xz /vagrant/build
+sudo cp fdosdk-centos6-amd64-${FDO_VER_FULL}_${FDO_REV}.tar.xz /vagrant/build
 if [ $MAKE_FDO_SDK -eq 1 ]; then
     echo [provision]: Copy FDO SDK for Ubuntu builds
     # Copy SDK also to sources, so Ubuntu can pick it up

Modified: trunk/Tools/Vagrant/centos/x86/Vagrantfile
===================================================================
--- trunk/Tools/Vagrant/centos/x86/Vagrantfile	2014-10-10 09:03:25 UTC (rev 8400)
+++ trunk/Tools/Vagrant/centos/x86/Vagrantfile	2014-10-14 13:03:03 UTC (rev 8401)
@@ -10,7 +10,7 @@
   # please see the online documentation at vagrantup.com.
 
   # Every Vagrant virtual environment requires a box to build off of.
-  config.vm.box = "vagrant-centos64-x86"
+  config.vm.box = "centos6-i386"
 
   # The url from where the 'config.vm.box' box will be fetched if it
   # doesn't already exist on the user's system.
@@ -293,7 +293,7 @@
     cd $SCRIPT_ROOT/mgdev/Server/src/Core
     sudo make unittest.sh
     # Run the test suites individually except for TileService/ResourceService for reasons already stated
-    for comp in CoordinateSystem FeatureService Geometry KmlService LogManager MdfModel Misc RenderingService ServerAdminService ServerManager ServiceManager SiteManager SiteService ProfilingService TransformMesh
+    for comp in CoordinateSystem FeatureService Geometry KmlService LogManager MappingService MdfModel Misc Performance RenderingService ServerAdminService ServerManager ServiceManager SiteManager SiteService ProfilingService TransformMesh
     do
         sudo -E ./mgserver test $comp UnitTestResults_${comp}.xml 2>&1 | tee ~/mapguide_${comp}_unit_test.log
         check_test
@@ -308,7 +308,7 @@
 echo [provision]: Copy log files to output dir
 sudo mv -f ~/*.log /vagrant/build
 echo [provision]: Copy tarballs to output dir
-sudo cp bin/*.tar.gz /vagrant/build
+sudo cp bin/*.tar.xz /vagrant/build
 sudo cp fdosdk-centos6-i386-${FDO_VER_FULL}_${FDO_REV}.tar.gz /vagrant/build
 if [ $MAKE_FDO_SDK -eq 1 ]; then
     echo [provision]: Copy FDO SDK for Ubuntu builds

Added: trunk/Tools/Vagrant/make_boxes.bat
===================================================================
--- trunk/Tools/Vagrant/make_boxes.bat	                        (rev 0)
+++ trunk/Tools/Vagrant/make_boxes.bat	2014-10-14 13:03:03 UTC (rev 8401)
@@ -0,0 +1,35 @@
+ at echo off
+SET BUILD_UBUNTU_32=0
+SET BUILD_UBUNTU_64=0
+SET BUILD_CENTOS_32=1
+SET BUILD_CENTOS_64=1
+pushd packer
+if "%BUILD_CENTOS_64%" == "1" (
+echo [packer]: Build CentOS 64-bit base box
+packer build -force centos6-amd64.json
+echo [vagrant]: Export centos6-amd64 base box
+rem call vagrant package --base "centos6-amd64" --output centos6-amd64.box
+call vagrant box add "centos6-amd64" centos6-amd64.box --force
+)
+if "%BUILD_CENTOS_32%" == "1" (
+echo [packer]: Build CentOS 32-bit base box
+packer build -force centos6-i386.json
+echo [vagrant]: Export centos6-i386 base box
+rem call vagrant package --base "centos6-i386" --output centos6-i386.box
+call vagrant box add "centos6-i386" centos6-i386.box --force
+)
+if "%BUILD_UBUNTU_64%" == "1" (
+echo [packer]: Build Ubuntu 64-bit base box
+packer build -force ubuntu12-amd64.json
+echo [vagrant]: Export ubuntu12-amd64 base box
+rem call vagrant package --base "ubuntu12-amd64" --output ubuntu12-amd64.box
+call vagrant box add "ubuntu12-amd64" ubuntu12-amd64.box --force
+)
+if "%BUILD_UBUNTU_32%" == "1" (
+echo [packer]: Build Ubuntu 32-bit base box
+packer build -force ubuntu12-i386.json
+echo [vagrant]: Export ubuntu12-i386 base box
+rem call vagrant package --base "ubuntu12-i386" --output ubuntu12-i386.box
+call vagrant box add "ubuntu12-i386" ubuntu12-i386.box --force
+)
+popd
\ No newline at end of file


Property changes on: trunk/Tools/Vagrant/packer
___________________________________________________________________
Added: svn:ignore
   + *.box
http
scripts
packer_cache


Added: trunk/Tools/Vagrant/packer/centos6-amd64.json
===================================================================
--- trunk/Tools/Vagrant/packer/centos6-amd64.json	                        (rev 0)
+++ trunk/Tools/Vagrant/packer/centos6-amd64.json	2014-10-14 13:03:03 UTC (rev 8401)
@@ -0,0 +1,54 @@
+{
+  "builders": [{
+    "type": "virtualbox-iso",
+    "guest_os_type": "RedHat_64",
+    "iso_url": "{{user `mirror`}}/6.5/isos/x86_64/CentOS-6.5-x86_64-minimal.iso",
+    "iso_checksum": "0d9dc37b5dd4befa1c440d2174e88a87",
+    "iso_checksum_type": "md5",
+    "output_directory": "output-centos-6.5-x86_64",
+    "vm_name": "packer-centos-6.5-x86_64",
+    "disk_size": "{{user `disk_size`}}",
+    "headless": "{{user `headless`}}",
+    "http_directory": "http",
+    "boot_wait": "5s",
+    "boot_command": [
+      "<esc>",
+      "<wait>",
+      "linux ks=http://{{.HTTPIP}}:{{.HTTPPort}}/centos-6.5/anaconda-ks.cfg",
+      "<enter>"
+    ],
+    "ssh_wait_timeout": "60m",
+    "ssh_username": "vagrant",
+    "ssh_password": "vagrant",
+    "shutdown_command": "sudo poweroff",
+    "vboxmanage": [
+      ["modifyvm", "{{.Name}}", "--memory", "{{user `memory`}}"],
+      ["modifyvm", "{{.Name}}", "--cpus", "{{user `cpus`}}"]
+    ]
+  }],
+  "provisioners": [{
+    "type": "shell",
+    "scripts": [
+      "scripts/centos-6.5/base.sh",
+      "scripts/centos/virtualbox.sh",
+      "scripts/common/vagrant.sh",
+      "scripts/common/sshd.sh",
+      "scripts/mapguide/centos64.sh",
+      "scripts/centos-6.5/cleanup.sh",
+      "scripts/common/minimize.sh"
+    ]
+  }],
+  "post-processors": [{
+    "type": "vagrant",
+    "compression_level": "{{user `compression_level`}}",
+    "output": "centos6-amd64.box"
+  }],
+  "variables": {
+    "compression_level": 1,
+    "cpus": "2",
+    "disk_size": 25000,
+    "headless": false,
+    "memory": "1024",
+    "mirror": "http://mirror.internode.on.net/pub/centos"
+  }
+}

Added: trunk/Tools/Vagrant/packer/centos6-i386.json
===================================================================
--- trunk/Tools/Vagrant/packer/centos6-i386.json	                        (rev 0)
+++ trunk/Tools/Vagrant/packer/centos6-i386.json	2014-10-14 13:03:03 UTC (rev 8401)
@@ -0,0 +1,54 @@
+{
+  "builders": [{
+    "type": "virtualbox-iso",
+    "guest_os_type": "RedHat",
+    "iso_url": "{{user `mirror`}}/6.5/isos/i386/CentOS-6.5-i386-minimal.iso",
+    "iso_checksum": "a4f27ab51d0d2c9d36b68c56b39f752b",
+    "iso_checksum_type": "md5",
+    "output_directory": "output-centos-6.5-i386",
+    "vm_name": "packer-centos-6.5-i386",
+    "disk_size": "{{user `disk_size`}}",
+    "headless": "{{user `headless`}}",
+    "http_directory": "http",
+    "boot_wait": "5s",
+    "boot_command": [
+      "<esc>",
+      "<wait>",
+      "linux ks=http://{{.HTTPIP}}:{{.HTTPPort}}/centos-6.5/anaconda-ks.cfg",
+      "<enter>"
+    ],
+    "ssh_wait_timeout": "60m",
+    "ssh_username": "vagrant",
+    "ssh_password": "vagrant",
+    "shutdown_command": "sudo poweroff",
+    "vboxmanage": [
+      ["modifyvm", "{{.Name}}", "--memory", "{{user `memory`}}"],
+      ["modifyvm", "{{.Name}}", "--cpus", "{{user `cpus`}}"]
+    ]
+  }],
+  "provisioners": [{
+    "type": "shell",
+    "scripts": [
+      "scripts/centos-6.5/base.sh",
+      "scripts/centos/virtualbox.sh",
+      "scripts/common/vagrant.sh",
+      "scripts/common/sshd.sh",
+      "scripts/mapguide/centos32.sh",
+      "scripts/centos-6.5/cleanup.sh",
+      "scripts/common/minimize.sh"
+    ]
+  }],
+  "post-processors": [{
+    "type": "vagrant",
+    "compression_level": "{{user `compression_level`}}",
+    "output": "centos6-i386.box"
+  }],
+  "variables": {
+    "compression_level": 1,
+    "cpus": "2",
+    "disk_size": 25000,
+    "headless": false,
+    "memory": "1024",
+    "mirror": "http://mirror.internode.on.net/pub/centos"
+  }
+}

Modified: trunk/Tools/Vagrant/scripts/build.sh
===================================================================
--- trunk/Tools/Vagrant/scripts/build.sh	2014-10-10 09:03:25 UTC (rev 8400)
+++ trunk/Tools/Vagrant/scripts/build.sh	2014-10-14 13:03:03 UTC (rev 8401)
@@ -226,7 +226,7 @@
         mkdir -p bin
     fi
 
-    tar -zcf bin/mapguideopensource-${BUILDNUM}.${REVISION}.${MGCPUPLATFORM}.tar.gz ${INSTALLROOT} ${LOCKFILEDIR}
+    tar -Jcf bin/mapguideopensource-${BUILDNUM}.${REVISION}.${MGCPUPLATFORM}.tar.xz ${INSTALLROOT} ${LOCKFILEDIR}
 fi
 echo "Build complete!"
 echo Main build execution: `expr $end_time - $start_time` s

Modified: trunk/Tools/Vagrant/scripts/build_fdo.sh
===================================================================
--- trunk/Tools/Vagrant/scripts/build_fdo.sh	2014-10-10 09:03:25 UTC (rev 8400)
+++ trunk/Tools/Vagrant/scripts/build_fdo.sh	2014-10-14 13:03:03 UTC (rev 8401)
@@ -543,7 +543,7 @@
     FDO_BUILD_COMPONENT="Make tarball"
     # Create a binary tar ball for FDO
     cd ${FDO_INST}
-    tar -zcf ${BUILDROOT}/fdosdk-centos6-${FDO_BUILD_CPU}-${FDO_VER_FULL}_${REVISION}.tar.gz *
+    tar -Jcf ${BUILDROOT}/fdosdk-centos6-${FDO_BUILD_CPU}-${FDO_VER_FULL}_${REVISION}.tar.xz *
     check_build
 
     if [ ${UBUNTU} -eq 1 ];

Modified: trunk/Tools/Vagrant/scripts/mginstallcentos.sh
===================================================================
--- trunk/Tools/Vagrant/scripts/mginstallcentos.sh	2014-10-10 09:03:25 UTC (rev 8400)
+++ trunk/Tools/Vagrant/scripts/mginstallcentos.sh	2014-10-14 13:03:03 UTC (rev 8401)
@@ -13,8 +13,8 @@
 MG_ARCH=i386
 MGVER_MAJOR_MINOR_REV=${MGVER_MAJOR_MINOR}.${MGVER_POINT}
 MGVER=${MGVER_MAJOR_MINOR_REV}.${MGVER_REV}
-FDO_TARBALL=fdosdk-centos6-${FDO_ARCH}-${FDOVER}.tar.gz
-MG_TARBALL=mapguideopensource-${MGVER}.${MG_ARCH}.tar.gz
+FDO_TARBALL=fdosdk-centos6-${FDO_ARCH}-${FDOVER}.tar.xz
+MG_TARBALL=mapguideopensource-${MGVER}.${MG_ARCH}.tar.xz
 MG_URL=${URL}/${MG_TARBALL}
 FDO_URL=${URL}/${FDO_TARBALL}
 
@@ -31,14 +31,14 @@
 #tar -C / -zxvf ${FDO_TARBALL}
 mkdir -p /usr/local/fdo-${FDOVER_MAJOR_MINOR_REV}
 echo "[install]: Extracting FDO"
-tar -C /usr/local/fdo-${FDOVER_MAJOR_MINOR_REV}/ -zxf ${FDO_TARBALL}
+tar -C /usr/local/fdo-${FDOVER_MAJOR_MINOR_REV}/ -Jxf ${FDO_TARBALL}
 
 if [ ! -f ${MG_TARBALL} ]; then
 wget -N ${MG_URL}
 fi
 
 echo "[install]: Extracting MapGuide"
-tar -C / -zxf ${MG_TARBALL}
+tar -C / -Jxf ${MG_TARBALL}
 
 # All of this is to make SELinux happy
 pushd /usr/local/fdo-${FDOVER_MAJOR_MINOR_REV}/lib

Modified: trunk/Tools/Vagrant/sources/README.txt
===================================================================
--- trunk/Tools/Vagrant/sources/README.txt	2014-10-10 09:03:25 UTC (rev 8400)
+++ trunk/Tools/Vagrant/sources/README.txt	2014-10-14 13:03:03 UTC (rev 8401)
@@ -2,4 +2,23 @@
 
 fdo_rdbms_thirdparty.tar.gz
 fdo-checkout.tar.gz
-mapguide-checkout.tar.gz
\ No newline at end of file
+mapguide-checkout.tar.gz
+
+fdo_rdbms_thirdparty.tar.gz is a tarball containing the following directory structure:
+
+    fdo_rdbms_thirdparty
+        mysql
+            x86 [Put 32-bit MySQL ConnectorC SDK here]
+            x64 [Put 64-bit MySQL ConnectorC SDK here]
+        oracle
+            x86 [Put 32-bit Oracle Instant Client SDK here]
+            x64 [Put 64-bit Oracle Instant Client SDK here]
+        pgsql [Put libpq SDK here]
+
+fdo-checkout.tar.gz is a tarball containing the following directory structure:
+
+    trunk [svn checkout of http://svn.osgeo.org/fdo/trunk]
+
+mapguide-checkout.tar.gz is tarball containing the following directory structure:
+
+    MgDev [svn checkout of http://svn.osgeo.org/mapguide/trunk/MgDev]
\ No newline at end of file

More information about the mapguide-commits mailing list