Architecture for Enterprise Backend

[Simon Haddon]

On Tue, 21 Mar 2006 00:06:12 +1100, Shoaib Burq <hydromap at GMAIL.COM> wrote:

>Hi
>
>I have a question which I am have difficulty putting in words but I'll
>try anyway...
>
>Can anyone suggest how large data vendors organise their raster and
>vector data with the view of having the flexibility to update,
>reorganise, manage and search/query. How can this process be automated
>using the various OS GIS components (e.g. PostGIS, Mapserver, etc).
>Are data sets divide into categories. How does one deal with temporal
>variability over the same extents? Is their an ideal way of organising
>the data on disk? How should the various mapserver components like
>Mapfiles be organised internally and on disk? Understandably it could
>get very ugly as data increases.

At Bureau of Regional Sciences we are currently building a site for drought
assessment which involved a large number of data sets.  Data sets are
updated at different intervals depending on the type of data and are stored
in PostGIS and on the file system depending on the type and putpose of the
data.  We have ended up building an application database (as separate the
the data sets) to drive the many layers and the time series data which has
given us alot of flexibility on where and how we store the data.  A simple
database structure that links products->layers->datasets means that we can
have many datasets that make up a layer and can also give us the location of
the dataset at runtime so we can alter the data acording to date parameters
entered by the user.

This format also means that we can use the dataset table as a reference to
the actual datasets stored in the database and how to perform data queries
on them without knowing in advance what columns they have.  We use the same
table for the file system.  Each time we get a new dataset then a new entry
goes in the dataset table with a relative path to the data on the file
system or a reference to the data in a table.

The categorisation has been treated as a 1->many link to the layers table as
we use categories to help display the layers in some sort of meanful context
but that is all it does.

Cheers,
Simon Haddon

>[snipped]
>
>Then having some way of mapping their spatial locations and locations
>on disk along with metadata info into PostGIS tables.
>
>I'd like to get some feedback from anyone who has had similar experience
>
>thanks in advance
>shoaib