Fwd: [UMN_MAPSERVER-USERS] Application Data

[James Cradock]

Giridhar,

CSV is text. PostgreSQL will hold the PostGIS data, and text (delimited
text or SQL (INSERT) commands) can be easily loaded into PostgreSQL. Look
at the docs for psql for loading data from SQL script commends. Look here:

http://techdocs.postgresql.org/techdocs/usingcopy.php

...to see how to load data from delimited files.

Performance *may* fall as the table holding this data gets large, but the
speed of the query to return results can be improved by using EXPLAIN to
examine how PostgreSQL thinks it ought to run queries (you'll see where
you likely need to add or remove indices, cast data types, etc. I think
you'll want to use EXPLAIN anyway to get a better handle on your data and
to see how PostgreSQL is going to use it.

http://www.postgresql.org/docs/7.4/interactive/sql-explain.html

The database will need to be setup to use PostGIS:

http://postgis.refractions.net/docs/

Look also at the sections on storing lon/lat data. You can do so in a
geometry column or you can use PostGIS to transform the lon and lat data
into geometry data in the mapfile.

After you've loaded the data and are satisfied with the structure of the
database objects and have some optimized queries, look at the MapServer
section for making that data available as layers in your map application.

Jim

> ---------- Forwarded message ----------
> From: Giridhar Manepalli <gmanepal at gmail.com>
> Date: Thu, 17 Mar 2005 17:59:29 -0500
> Subject: Re: [UMN_MAPSERVER-USERS] Application Data
> To: warmerdam at pobox.com
>
>
> Thanks Frank,
> But how is it possible to convert that CSV data into a shapefile or
> POSTGIS supportive table, in order for me to use in a MAPSERVER.
>
> Condition being : data < 1MB
>
> Regards,
> Giridhar
>
>
> On Thu, 17 Mar 2005 17:52:02 -0500, Frank Warmerdam
> <fwarmerdam at gmail.com> wrote:
>> On Thu, 17 Mar 2005 17:41:24 -0500, Giridhar Manepalli
>> <gmanepal at gmail.com> wrote:
>> > I would like to know, if it is possible to convert data produced from
>> > a third party application to a shapefile or PostGIS table.
>> >
>> > Hypothetically speaking, the data from the third party application
>> > will be in the following format.
>> >
>> > SSN, Lat, Long, Age etc
>> >
>> > Imagine that the above data is huge (as large as 100GB), so I need to
>> > dump this information to a database.
>> >
>> > I would like to know, how to use this data from a table to Map it onto
>> > a MAPServer.
>> >
>> > and process queries like:
>> >
>> > *.)  How many people are at a given location
>>
>> Giridhar,
>>
>> It is certainly possible in principle to translate this data to PostGIS
>> or into a point shapefile, but if it is 100GB of data in that
>> compact comma seperated value format, it is going to be alot
>> larger in PostGIS or in a shapefile.
>>
>> A traditional approach to handling this sort of dataset, if you
>> only need summary information is to aggregate it somehow.
>> Otherwise you are going to have alot of performance and
>> query optimization issues.
>>
>> Good luck,
>> --
>> ---------------------------------------+--------------------------------------
>> I set the clouds in motion - turn up   | Frank Warmerdam,
>> warmerdam at pobox.com
>> light and sound - activate the windows | http://pobox.com/~warmerdam
>> and watch the world go round - Rush    | Geospatial Programmer for Rent
>>
>



-----
James Cradock, jcradock at me3.com