fastest way to convert lon-lat to points

[P Kishor]

Hey Debbie, Thanks for your reply. A few clarifications...

On 7/20/06, Pagurek,Debbie [NCR] <Debbie.Pagurek at ec.gc.ca> wrote:


> I am working on some similar work with .csv files and OGR right now.
> I don't know why your field is being changed like that...
>

Yup, this is strange, and unacceptable, so I need a workaround for this.

> As for debugging, aside from the ogrinfo test, you might want to embed
> your <OGRVRTDataSource> into a mapfile and then try the shp2img tool. If
> this produces an image for you, all the mapserver stuff is fine. But you
> might be having permissions problems on your server, which is where MY
> problem lies right now. I'm using IIS5, not apache... So I have some
> work ahead of me!

I am not doing anything with the web at all here. This is pure command
line stuff. The complete process I want to accomplish is as follows --

1. convert lon-lat into shapefile points.
2. overlay (point-in-poly) #1 shapefile with a shapefile polys to
determine, for each point, what poly it falls in... so, if #1
shapefile has cities, and the polys shapefile has states, I want to
create an additional attribute for each city and fill it with the
states value.

This is all command line, and for quite a bit of data (upward of 5
million), so speed is an issue, although CPU and memory are not an
issue (good hardware). I am using Perl for all the automation.

I would like to keep everything in the realm of text files and dbf
files as much as possible, but I am not averse to using SQLite or
Berkeley DB. If push-comes-to-shove, I might be open to experimenting
with PostGIS, but only if push really comes to shove.


Many thanks, (and hope to see you again soon somewhere... maybe I will
have to make a trip up North).


>
> You might also want to write a little PHP script to make sure that your
> ODBC DSN (a system DSN, right?) is working ok as well. It's when I did
> this that I realized that my problem has nothing to do with mapserver,
> ogrinfo etc.
>
> Good luck and let us know what you discover,
> Debbie Pagurek
>
> -----Original Message-----
> From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
> Behalf Of P Kishor
> Sent: Thursday, July 20, 2006 8:19 AM
> To: MAPSERVER-USERS at LISTS.UMN.EDU
> Subject: [UMN_MAPSERVER-USERS] fastest way to convert lon-lat to points
>
> Folks,
>
> I have seen this question asked in the past, and have a similar problem.
> I want to convert many lon-lat to a shapefile of points quickly. The
> lon-lat are in a csv text file (values.txt) like so
>
> foo,bar,baz,long,lat,qux
> 1, bar1, baz1, -87.796341, 41.907504, s10 ...
> ...
>
> So, I installed FW_Tools 1.05, created a DSN out of the text file (this
> is and has to be on a Windows box), and ogrinfo happily gave me
> information that I wanted, but with a twist...
>
> OGRFeature(values.txt):1056
>   foo (Integer) = 1
>   bar(String) = ba1
>   baz(String) = baz1
>   long (Real) = -87.796341
>   lat (Real) = 41.907504
>   qux(String) = 10.0000
>
> So, what's with interpreting qux? It converted my "s10" into a string of
> value "10.0000".
>
> Anyway, then I created an OVF file (values.ovf) like so
>
> <OGRVRTDataSource>
>   <OGRVRTLayer  name="values">
>   <SrcDataSource>ODBC:values</SrcDataSource>
>   <SrcLayer>values.txt</SrcLayer>
>   <GeometryField encoding="PointFromColumns" x="Long" y="Lat"/>
>   <GeometryType>wkbPoint</GeometryType>
>   </OGRVRTLayer>
> </OGRVRTDataSource>
>
> well,
>
> ogr2ogr -f "ESRI Shapefile" . values.ovf values
>
> but while that seemed to do something without any error, I got no
> output.
>
> So, is what I am doing the right way of going about this? Is there a
> better or another way to try out and benchmark? Oh, did I mention? --
> The values.txt file has about 5.25 million rows, so speed is important.
>



-- 
Puneet Kishor http://punkish.eidesis.org/
Nelson Inst. for Env. Studies, UW-Madison http://www.ies.wisc.edu/
Open Source Geospatial Foundation https://edu.osgeo.org/