fastest way to convert lon-lat to points

[Pagurek,Debbie [NCR]]

Yes, I'm following you. My suggestion to test out the OVF in a mapfile
was just to make sure that the OVF is working.

Funny you should mention all this point in polygon search stuff, because
that is exactly what I'm working on right now, but I'm using a .csv
file, WFS requests with a DWITHIN filter, and adding the results to the
original .csv as new fields. I don't think the performance is going to
be very good though ;)

Is your work all to be done 'on the fly'? Repetitively? If not, I would
just use Quantum GIS to load that textfile and create a shapefile...

Hopefully some others on the list will have some suggestions for you /
us!

Debbie 

-----Original Message-----
From: punk.kish at gmail.com [mailto:punk.kish at gmail.com] On Behalf Of P
Kishor
Sent: Thursday, July 20, 2006 10:37 AM
To: Pagurek,Debbie [NCR]
Cc: MAPSERVER-USERS at lists.umn.edu
Subject: Re: [UMN_MAPSERVER-USERS] fastest way to convert lon-lat to
points

Hey Debbie, Thanks for your reply. A few clarifications...

On 7/20/06, Pagurek,Debbie [NCR] <Debbie.Pagurek at ec.gc.ca> wrote:


> I am working on some similar work with .csv files and OGR right now.
> I don't know why your field is being changed like that...
>

Yup, this is strange, and unacceptable, so I need a workaround for this.

> As for debugging, aside from the ogrinfo test, you might want to embed

> your <OGRVRTDataSource> into a mapfile and then try the shp2img tool. 
> If this produces an image for you, all the mapserver stuff is fine. 
> But you might be having permissions problems on your server, which is 
> where MY problem lies right now. I'm using IIS5, not apache... So I 
> have some work ahead of me!

I am not doing anything with the web at all here. This is pure command
line stuff. The complete process I want to accomplish is as follows --

1. convert lon-lat into shapefile points.
2. overlay (point-in-poly) #1 shapefile with a shapefile polys to
determine, for each point, what poly it falls in... so, if #1 shapefile
has cities, and the polys shapefile has states, I want to create an
additional attribute for each city and fill it with the states value.

This is all command line, and for quite a bit of data (upward of 5
million), so speed is an issue, although CPU and memory are not an issue
(good hardware). I am using Perl for all the automation.

I would like to keep everything in the realm of text files and dbf files
as much as possible, but I am not averse to using SQLite or Berkeley DB.
If push-comes-to-shove, I might be open to experimenting with PostGIS,
but only if push really comes to shove.


Many thanks, (and hope to see you again soon somewhere... maybe I will
have to make a trip up North).


>
> You might also want to write a little PHP script to make sure that 
> your ODBC DSN (a system DSN, right?) is working ok as well. It's when 
> I did this that I realized that my problem has nothing to do with 
> mapserver, ogrinfo etc.
>
> Good luck and let us know what you discover, Debbie Pagurek
>
> -----Original Message-----
> From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] 
> On Behalf Of P Kishor
> Sent: Thursday, July 20, 2006 8:19 AM
> To: MAPSERVER-USERS at LISTS.UMN.EDU
> Subject: [UMN_MAPSERVER-USERS] fastest way to convert lon-lat to 
> points
>
> Folks,
>
> I have seen this question asked in the past, and have a similar
problem.
> I want to convert many lon-lat to a shapefile of points quickly. The 
> lon-lat are in a csv text file (values.txt) like so
>
> foo,bar,baz,long,lat,qux
> 1, bar1, baz1, -87.796341, 41.907504, s10 ...
> ...
>
> So, I installed FW_Tools 1.05, created a DSN out of the text file 
> (this is and has to be on a Windows box), and ogrinfo happily gave me 
> information that I wanted, but with a twist...
>
> OGRFeature(values.txt):1056
>   foo (Integer) = 1
>   bar(String) = ba1
>   baz(String) = baz1
>   long (Real) = -87.796341
>   lat (Real) = 41.907504
>   qux(String) = 10.0000
>
> So, what's with interpreting qux? It converted my "s10" into a string 
> of value "10.0000".
>
> Anyway, then I created an OVF file (values.ovf) like so
>
> <OGRVRTDataSource>
>   <OGRVRTLayer  name="values">
>   <SrcDataSource>ODBC:values</SrcDataSource>
>   <SrcLayer>values.txt</SrcLayer>
>   <GeometryField encoding="PointFromColumns" x="Long" y="Lat"/>
>   <GeometryType>wkbPoint</GeometryType>
>   </OGRVRTLayer>
> </OGRVRTDataSource>
>
> well,
>
> ogr2ogr -f "ESRI Shapefile" . values.ovf values
>
> but while that seemed to do something without any error, I got no 
> output.
>
> So, is what I am doing the right way of going about this? Is there a 
> better or another way to try out and benchmark? Oh, did I mention? -- 
> The values.txt file has about 5.25 million rows, so speed is
important.
>



--
Puneet Kishor http://punkish.eidesis.org/ Nelson Inst. for Env. Studies,
UW-Madison http://www.ies.wisc.edu/ Open Source Geospatial Foundation
https://edu.osgeo.org/