<br><font size=2 face="sans-serif">IMO:</font>
<br>
<br>
<br><font size=2 face="sans-serif">Hi Randy,</font>
<br>
<br><font size=2 face="sans-serif">Thank you for your informative post.
It has given me a lot to follow up on and think about.</font>
<br>
<br><font size=2 face="sans-serif">I can see an immediate need that this
type of solution could well be used for. I like it.</font>
<br>
<br><font size=2 face="sans-serif">I suspect that in many larger corporate
types of environments, it could well be used effectively for 'pilot' and
'pre-production' type tasks. </font>
<br>
<br><font size=2 face="sans-serif">For 'production' type environments,
there would be issues of integrating an external service hosting spatial
data with internal services hosting corporate aspatial data sources and
applications.</font>
<br>
<br>
<br>
<br><font size=2 face="sans-serif">with regards to storing imagery in a
database:</font>
<br>
<br><font size=2 face="sans-serif"><rant> (and
not directed at you)</font>
<br>
<br><font size=2 face="sans-serif">I've also seen a lot of reports suggesting
that image management should be file based.</font>
<br>
<br><font size=2 face="sans-serif">My personal preference is to use a database
if possible, so that I can take advantage of corporate data management
facilities, backups, point in time restores etc.</font>
<br>
<br><font size=2 face="sans-serif">I've managed 70 GB orthophoto mosaics
in ArcSDE / Oracle before with minimal problems. I found performance and
response times to be comparable with other image web server options on
the market that use file based solutions for storing data.</font>
<br>
<br><font size=2 face="sans-serif">Ideally, I'm looking to manage state
wide mosaics with a consistant look and feel that can be treated as a single
'layer' by client GIS / Remote Sensing applications (data integrity issues
allowing). </font>
<br>
<br><font size=2 face="sans-serif">One potential use is 'best available'
data mosaics could undergo regular updates as more imagery is flown or
captured. A database makes it easier to manage and deliver such data.</font>
<br>
<br><font size=2 face="sans-serif">My definition of 'imagery' goes beyond
aerial photographs and includes multi or hyper-spectral imagery; various
geophysics data sources such as aeromagnetics, gravity, radiometrics; radar
data etc.</font>
<br>
<br><font size=2 face="sans-serif">Typically this data is required for
digital image analysis purposes using a remote sensing application, so
the integrity of 'the numbers' that make up the image is very important.</font>
<br>
<br><font size=2 face="sans-serif">Many of today's image based solutions
use a (lossy) wavelet compression that can corrupt the integrity of 'the
numbers' describing the radiometric data in the image.</font>
<br>
<br><font size=2 face="sans-serif">When we consider the big picture issues
facing us today, such as Climate Change, I think that it is important to
protect our definitive image libraries from such corruption as they will
be invaluable sources of data for future multi-temporal analysis.</font>
<br>
<br><font size=2 face="sans-serif">That said, if the end use is just for
a picture, then a wavelet compression is a good option. Just protect the
source data for future use.</font>
<br>
<br><font size=2 face="sans-serif"></rant> </font>
<br>
<br>
<br><font size=2 face="sans-serif">So, does anyone know of a good open
source spatial solution for storing and accessing (multi and hyperspectral)
imagery in a database? ;-)</font>
<br>
<br><font size=2 face="sans-serif">WMS 1.3 and WCS are showing promise
for serving imagery, including multi and hyperspectral data.</font>
<br>
<br>
<br>
<br><font size=2 face="sans-serif">Bruce Bannerman</font>
<br>
<br>
<br>
<br>
<br>
<br><font size=2><tt>discuss-bounces@lists.osgeo.org wrote on 20/02/2008
10:09:28 AM:<br>
<br>
> Hi Ivan,<br>
> <br>
> The most common advice I've seen says to leave raster
out of the DB.<br>
> Of course footprints and meta data could be there, but you would want
to<br>
> point Geoserver coverage to the image/image pyramid url somewhere
in the<br>
> directory hierarchy.<br>
> <br>
> Brent has a nice writeup here:<br>
> http://docs.codehaus.org/display/GEOSDOC/Load+NASA+Blue+Marble+Data<br>
> <br>
> In an AWS sense my idea is to Java proxy the Geoserver Coverage Data
URL to<br>
> S3 buckets and park the imagery over on the S3 side to take advantage
of<br>
> stability and replication. Performance, though, might not be as good
as a<br>
> local directory. Maybe a one time cache to a local directory would
work<br>
> better.<br>
> <br>
> Note: Amazon doesn't charge for inside AWS data transfers.<br>
> <br>
> So in theory:<br>
> PostGIS holds the footprint geometry + metadata<br>
> EC2 Geoserver WFS handles footprint queries into an Svg/Xaml
client, just<br>
> stick it on top of something like JPL BMNG. Once a user picks a coverage<br>
> switch to the Geoserver WMS/WCS service for zooming around in the
selected<br>
> image pyramid<br>
> S3 buckets contain the tiffs, pyramids ...<br>
> EC2 Geoserver handles WMS/WCS service<br>
> EC2 proxy pulls the imagery from the S3 side as needed<br>
> <br>
> Sorry I haven't had time to try this so it is just theoretical. Of
course<br>
> you can go traditional and just keep the coverage imagery files on
the local<br>
> instance avoiding the S3 proxy idea. The reason I don't like that
idea is<br>
> the imagery has to be loaded with every instance creation while an
S3<br>
> approach would need only one copy.<br>
> <br>
> <br>
> randy<br>
> <br>
> -----Original Message-----<br>
> From: Lucena, Ivan [mailto:ivan.lucena@pmldnet.com] <br>
> Sent: Tuesday, February 19, 2008 2:59 PM<br>
> To: rkgeorge@cadmaps.com; OSGeo Discussions<br>
> Subject: Re: [OSGeo-Discuss] OS Spatial environment 'sizing'<br>
> <br>
> Hi Randy, Bruce,<br>
> <br>
> That is a nice piece of advise Randy. I am sorry to intrude the <br>
> conversation but I would like to ask how that "heavy raster"
<br>
> manipulation would be treated by PostgreSQL/PostGIS, managed or unmanaged?<br>
> <br>
> Best regards,<br>
> <br>
> Ivan<br>
> <br>
> Randy George wrote:<br>
> > Hi Bruce,<br>
> > <br>
> > <br>
> > <br>
> > On the
"scale relatively quickly" front, you should look <br>
> > at Amazon's EC2/S3 services. I've recently worked with it and
find it an <br>
> > attractive platform for scaling http://www.cadmaps.com/gisblog<br>
> > <br>
> > <br>
> > <br>
> > The stack I like is Ubuntu+Java+ Postgresql/PostGIS + Apache2
mod_jk <br>
> > Tomcat + Geoserver + custom SVG or XAML clients run out of Tomcat<br>
> > <br>
> > <br>
> > <br>
> > If you
use the larger instances the cost is higher but <br>
> > it sounds like you plan on some heavy raster services (WMS,WCS)
and lots <br>
> > of memory will help.<br>
> > <br>
> > Small EC2 instance provides $0.10/hr:<br>
> > <br>
> > 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2
Compute <br>
> > Unit), 160 GB of instance storage, 32-bit platform<br>
> > <br>
> > <br>
> > <br>
> > Large EC2 instances provide $0.40/hr:<br>
> > <br>
> > 7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2
EC2 <br>
> > Compute Units each), 850 GB of instance storage, 64-bit platform<br>
> > <br>
> > <br>
> > <br>
> > Extra large EC2 instances $0.80/hr:<br>
> > <br>
> > 15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2
EC2 Compute <br>
> > Units each), 1690 GB of instance storage, 64-bit platform<br>
> > <br>
> > <br>
> > <br>
> > Note: that the instances do not need to be permanent. Some people
<br>
> > (WeoGeo) have been using a couple of failover small instances
and then <br>
> > starting new large instances for specific requirements. The idea
is to <br>
> > start and stop instances as required rather than having ongoing
<br>
> > infrastructure costs. It only takes a minute or so to start an
ec2 <br>
> > instance. If you are running a corporate service there may be
parts of <br>
> > the day with very little use so you just schedule your heavy
duty <br>
> > instances for peak times. If you can connect your raster to S3
buckets <br>
> > rather than instance storage you have built in replicated backup.<br>
> > <br>
> > <br>
> > <br>
> > I know that Java JAI can easily eat up memory and is core to
Geoserver <br>
> > WMS/WCS so you probably want to look at large memory footprint
for any <br>
> > platform with lots of raster service. I'm partial to Geoserver
because <br>
> > of its Java foundation. I think I would try to keep the
Apache2 mod_jk <br>
> > Tomcat Geoserver on a separate server instance from PostGIS.
This might <br>
> > avoid problems for instance startup since your database would
need to be <br>
> > loaded separately. The instance ami resides in a 10G partition
the <br>
> > balance of data will probably reside on a /mnt partition separate
from <br>
> > ec2-run-instances. You may be able to avoid datadir problems
by adding <br>
> > something like Elastra to the mix. Elastra beta is a wrapper
for <br>
> > PostgreSql that puts the datadir on S3 rather than local to an
instance. <br>
> > I suppose they still keep indices(GIST et al) on the local instance.<br>
> > <br>
> > (I still think it an interesting exercise to see what could be
done <br>
> > connecting PostGIS to AWS SimpleDB services.)<br>
> > <br>
> > <br>
> > <br>
> > So thinking out loud here is a possible architecture-<br>
> > <br>
> > Basic permanent setup<br>
> > <br>
> > put raster in S3 - this may require some customization of Geoserver,<br>
> > <br>
> > build a datadir in a PostGIS and backup to S3<br>
> > <br>
> > create a private ami for Postgresql/PostGIS<br>
> > <br>
> > create a private ami for the load balancer instance<br>
> > <br>
> > create a private ami with your service stack for both a small
and large <br>
> > instance for flexibility,<br>
> > <br>
> > Startup services<br>
> > <br>
> > start a balancer instance<br>
> > <br>
> > point your DNS CNAME to this balancer instance<br>
> > <br>
> > start a PostGis instance (you could have more than one if necessary
but <br>
> > it would be easier to just scale to a larger instance type if
the load <br>
> > demands it)<br>
> > <br>
> > have a scripted download from an S3 BU to your PostGIS datadir
(I'm <br>
> > assuming a relatively static data resource)<br>
> > <br>
> > Variable services<br>
> > <br>
> > start service stack instance and connect to PostGIS<br>
> > <br>
> > update balancer to see new instance - this could be tricky<br>
> > <br>
> > repeat previous two steps as needed<br>
> > <br>
> > at night scale back - cron scaling for a known cycle or use a
controller <br>
> > like weoceo to detect and respond to load fluctuation<br>
> > <br>
> > <br>
> > <br>
> > By the way the public AWS ami with the best resources that I
have found <br>
> > is Ubuntu 7.10 Gutsy. The debian dependency tools are much easier
to use <br>
> > and the resources are plentiful.<br>
> > <br>
> > <br>
> > <br>
> > I've been toying with using an AWS stack adapted for serving
some larger <br>
> > Postgis vector sets such as fully connected census demographic
data and <br>
> > block polygons here in US. The idea would be to populate the
data <br>
> > directly from the census SF* and TIGER with a background Java
bot. There <br>
> > are some potentially novel 3D viewing approaches possible with
xaml. <br>
> > Anyway lots of fun to have access to virtual systems like this.<br>
> > <br>
> > <br>
> > <br>
> > As you can see I'm excited anyway.<br>
> > <br>
> > <br>
> > <br>
> > randy<br>
> > <br>
> > <br>
> > <br>
> > <br>
> > <br>
> > *From:* discuss-bounces@lists.osgeo.org <br>
> > [mailto:discuss-bounces@lists.osgeo.org] *On Behalf Of <br>
> > *Bruce.Bannerman@dpi.vic.gov.au<br>
> > *Sent:* Monday, February 18, 2008 6:35 PM<br>
> > *To:* OSGeo Discussions<br>
> > *Subject:* [OSGeo-Discuss] OS Spatial environment 'sizing'<br>
> > <br>
> > <br>
> > <br>
> > <br>
> > IMO:<br>
> > <br>
> > <br>
> > Hello everyone,<br>
> > <br>
> > I'm trying to get a feel for server 'sizing' for a **hypothetical**
<br>
> > Corporate environment to support OS Spatial apps.<br>
> > <br>
> > <br>
> > <br>
> > Assume that:<br>
> > <br>
> > - this is a dedicated environment to allow the use of OS Spatial
<br>
> > applications to serve Corporate OGC Services.<br>
> > <br>
> > - the applications of interest are GeoServer, Deegree, GeoNetwork,
<br>
> > MapServer, MapGuide and Postgres/PostGIS.<br>
> > <br>
> > - the environment may need to scale relatively quickly.<br>
> > <br>
> > - it will be required to serve in the vicinty of 5 to 10 TB of
data <br>
> > initially (WMS, WFS, WCS).<br>
> > <br>
> > <br>
> > <br>
> > Can anyone shed some light on the following questions please?<br>
> > <br>
> > - I'm assuming a Linux installation (SLES, Redhat or Debian)
or possibly <br>
> > Intel Solaris. Has anyone experienced any issues in these (or
other) <br>
> > environments that they'd like to share?<br>
> > <br>
> > - Are there any recommendations as to dedicated network bandwidth
that <br>
> > should be allocated?<br>
> > <br>
> > - Has anyone done any work with load balancing and would like
to share <br>
> > their experiences?<br>
> > <br>
> > - Of the above OS Spatial products, which ones could co-exist
on the <br>
> > same server (excluding Postgres/PostGIS)?<br>
> > <br>
> > <br>
> > Any thoughts are appreciated.<br>
> > <br>
> > <br>
> > Bruce Bannerman<br>
> > Australia<br>
> > <br>
> > Notice:<br>
> > This email and any attachments may contain information that is
personal, <br>
> > confidential,<br>
> > legally privileged and/or copyright. No part of it should be
reproduced, <br>
> > adapted or communicated without the prior written consent of
the <br>
> > copyright owner.<br>
> > <br>
> > It is the responsibility of the recipient to check for and remove
viruses.<br>
> > <br>
> > If you have received this email in error, please notify the sender
by <br>
> > return email, delete it from your system and destroy any copies.
You are <br>
> > not authorised to use, communicate or rely on the information
contained <br>
> > in this email.<br>
> > <br>
> > Please consider the environment before printing this email.<br>
> > <br>
> > <br>
> > <br>
> > <br>
> > <br>
> > <br>
> > <br>
> > <br>
> > ------------------------------------------------------------------------<br>
> > <br>
> > _______________________________________________<br>
> > Discuss mailing list<br>
> > Discuss@lists.osgeo.org<br>
> > http://lists.osgeo.org/mailman/listinfo/discuss<br>
> <br>
> _______________________________________________<br>
> Discuss mailing list<br>
> Discuss@lists.osgeo.org<br>
> http://lists.osgeo.org/mailman/listinfo/discuss<br>
</tt></font>