[OSGeo-Discuss] Batch geocoding

Sat Feb 5 13:24:32 PST 2011

Thanks Agustin. I started with that, but the address geocoder is just a
front-end to Google or Yahoo, and it also uses your IP address and is
subject to the same limitations as any other method [
http://www.gpsvisualizer.com/geocoder/]:

How this page works

Google and Yahoo each provide a geocoding "API" -- a way for other programs
to quickly and easily access their services. But they only allow a certain
number of queries per day, based on your IP address. This form uses
JavaScript-On-Demand (JSON) code that causes *your* Web browser to be the
one making the request (rather than gpsvisualizer.com), which means your
queries don't count against my server's limit. (And thanks are due to the
developers of the batch geocoder at
BatchGeocode.com<http://www.gpsvisualizer.com/misc/link.cgi?url=http://www.batchgeocode.com/>
for
the Yahoo JSON info.)

With geopy I could do the same thing, but I could automate the extraction of
the addresses from my database and writing back the coordinates, so it
actually saved me a lot of work instead of having to parse the gpsvisualizer
results. For 50,000 addresses, that is a big drag.

Cheers,
JP

On Sat, Feb 5, 2011 at 8:46 PM, Agustin Diez Castillo <adiez at uv.es> wrote:

> gpsvisualizer can do both yahoo and google geocoding.
> http://www.gpsvisualizer.com/geocoding.html
>
> On Feb 5, 2011, at 6:52 PM, JP Glutting wrote:
>
> Hola Alvaro,
>
> No, I wasn't aware of Cartociudad. It looks very interesting. Now I have so
> many choices, I don't know what to do.
>
> Thanks!
> JP
>
>
> On Sat, Feb 5, 2011 at 6:44 PM, Alvaro Zabala <azabala at gmail.com> wrote:
>
>> Hi,
>>
>> Do you know cartociudad project, from spanish IGN?
>>
>> It have web services related with spanish street map, and one of this is
>> geocoding service:
>>
>> http://www.cartociudad.es/portal/pdf/CARTOCIUDAD_Direcciones.pdf
>>
>>
>> Best regards
>>
>>
>> On 4 February 2011 21:27, JP Glutting <jpglutting at gmail.com> wrote:
>>
>>> Hi Wladimir,
>>>
>>> That is sort of what I thought. Too bad. I was just in Barcelona
>>> yesterday, at the CSIC, to get data. Where do you work?
>>>
>>> I am on the geoinquiets list too, but I am afraid I am on so many lists
>>> and in so many projects that I don't keep close track of them all. I should
>>> hop over and introduce myself, though.
>>>
>>> Fins aviat,
>>> JP
>>>
>>> Public Health - GIS - Bioinformatics
>>> http://www.bmeisis.com
>>> <http://www.bmeisis.com/>http://www.jpglutting.com
>>>
>>>
>>>
>>>
>>> On Fri, Feb 4, 2011 at 7:18 PM, Wladimir Szczerban <bolosig at gmail.com>wrote:
>>>
>>>> Hi JP,
>>>>
>>>> I live in Barcelona and unfortunately there is no open geocoding
>>>> service in Barcelona. In the group of geoinquietos (
>>>> http://www.geoinquiets.cat/ or
>>>> http://wiki.osgeo.org/wiki/Geoinquiets_Barcelona) we are trying to get
>>>> the ICC (http://www.icc.cat/) the official agency open the geocoding
>>>> service they have.
>>>>
>>>> Cheers,
>>>> BOLO
>>>>
>>>> 2011/2/4 Stark Hans-Jörg <hansjoerg.stark at fhnw.ch>
>>>>
>>>> Any initiative for OA is appreciated highly ;o)
>>>>>
>>>>>
>>>>> Best,
>>>>>
>>>>> hj
>>>>>
>>>>>
>>>>> *Von:* discuss-bounces at lists.osgeo.org [mailto:
>>>>> discuss-bounces at lists.osgeo.org] *Im Auftrag von *JP Glutting
>>>>> *Gesendet:* Freitag, 4. Februar 2011 16:17
>>>>>
>>>>> *An:* OSGeo Discussions
>>>>> *Betreff:* Re: [OSGeo-Discuss] Batch geocoding
>>>>>
>>>>>
>>>>>
>>>>> There might be a data sharing initiative, there are several different
>>>>> actors (city, regional government, etc.). I don't know all of what there is
>>>>> out there, and the data was originally collected for another project.
>>>>>
>>>>>
>>>>> I think I will be Ok with Yahoo. If it works out, it shouldn't be
>>>>> difficult to export the dataset for openaddresses. It isn't my data, though,
>>>>> so I will have to ask permission first. It seems like a good idea, though.
>>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>> JP
>>>>>
>>>>> On Fri, Feb 4, 2011 at 3:49 PM, Stark Hans-Jörg <
>>>>> hansjoerg.stark at fhnw.ch> wrote:
>>>>>
>>>>>
>>>>> *Von:* discuss-bounces at lists.osgeo.org [mailto:
>>>>> discuss-bounces at lists.osgeo.org] *Im Auftrag von *JP Glutting
>>>>> *Gesendet:* Freitag, 4. Februar 2011 15:34
>>>>>
>>>>>
>>>>> *An:* OSGeo Discussions
>>>>> *Betreff:* Re: [OSGeo-Discuss] Batch geocoding
>>>>>
>>>>>
>>>>> Hi Hans-Jörg,
>>>>>
>>>>>
>>>>> I am not aware that Barcelona (the city?) has a very open geodata
>>>>> strategy, but that certainly would be nice. If you know of anyone in the
>>>>> area who might know more, I am more than willing to contact them.
>>>>>
>>>>> *[shj] my mistake. I thought I had read something on this some time
>>>>> ago. Maybe also because Spain has strong support in the FOSS domain.*
>>>>>
>>>>>
>>>>> I have the addresses, do you mean the coordinates? The address format
>>>>> is not ideal, but I am pretty happy with them, as they seem to geocode well
>>>>> in the tests I have done.
>>>>>
>>>>> *[shj] yes, addresses along with co-ordinates.*
>>>>>
>>>>>
>>>>> The Yahoo API allows geocoding of 50k addresses a day, which is plenty
>>>>> for what I need. I am going to try to use that.
>>>>>
>>>>> *[shj] that’ll certainly do.*
>>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>> JP
>>>>>
>>>>>
>>>>> On Fri, Feb 4, 2011 at 3:26 PM, Stark Hans-Jörg <
>>>>> hansjoerg.stark at fhnw.ch> wrote:
>>>>>
>>>>> Hi JP
>>>>>
>>>>>
>>>>> Barcelona is unfortunately not well covered yet in OA. But I thought
>>>>> that Spain has a very “open” strategy in terms of providing geodata. If you
>>>>> manage to get Barcelona addresses (perhaps from council or any other
>>>>> “official body”) the OA team will insert these into OA and then you can use
>>>>> the REST service.
>>>>>
>>>>>
>>>>> Good luck!
>>>>>
>>>>> -hj
>>>>>
>>>>>
>>>>> *Von:* discuss-bounces at lists.osgeo.org [mailto:
>>>>> discuss-bounces at lists.osgeo.org] *Im Auftrag von *JP Glutting
>>>>> *Gesendet:* Freitag, 4. Februar 2011 15:24
>>>>> *An:* OSGeo Discussions
>>>>> *Betreff:* Re: [OSGeo-Discuss] Batch geocoding
>>>>>
>>>>>
>>>>> Thanks for all the responses! I will track them all down and see how
>>>>> they work.
>>>>>
>>>>>
>>>>> Stark, I have 146,472 addresses in the city of Barcelona. Many of them
>>>>> are duplicates, but at the very least I have 31,514 that need to be coded,
>>>>> although that would leave out many that I want to use. I am in the process
>>>>> of developing filters to clean out apartment numbers, etc., to be able to
>>>>> pull a single coordinate for a whole set of addresses, and there are at
>>>>> least 6,000 that are invalid (from the original 146k, so 140k). It is a lot.
>>>>> I will take a look at openaddresses and do some testing.
>>>>>
>>>>>
>>>>> Thanks!
>>>>>
>>>>> JP
>>>>>
>>>>> On Fri, Feb 4, 2011 at 3:10 PM, Stark Hans-Jörg <
>>>>> hansjoerg.stark at fhnw.ch> wrote:
>>>>>
>>>>> The OpenAddresses project (www.openaddresses.org) is supposed to solve
>>>>> exactly your problem.
>>>>>
>>>>> You can use the provided geocoding services (
>>>>> http://code.google.com/p/openaddresses/wiki/RESTService)
>>>>>
>>>>>
>>>>> OpenAddresses has some regions where data was donated – there you will
>>>>> get high-quality results. Unfortunately this is not yet globally available…
>>>>>
>>>>>
>>>>> Hth
>>>>>
>>>>> -hj
>>>>>
>>>>>
>>>>> *Von:* discuss-bounces at lists.osgeo.org [mailto:
>>>>> discuss-bounces at lists.osgeo.org] *Im Auftrag von *JP Glutting
>>>>> *Gesendet:* Freitag, 4. Februar 2011 14:34
>>>>> *An:* discuss at lists.osgeo.org
>>>>> *Betreff:* [OSGeo-Discuss] Batch geocoding
>>>>>
>>>>>
>>>>> Hello,
>>>>>
>>>>>
>>>>> I have a large set of addresses (around 150k) that I need to geocode
>>>>> for a study (my Masters thesis on heat-related mortality). I am looking into
>>>>> different solutions, but I can't find anything that seems like it would work
>>>>> properly.
>>>>>
>>>>>
>>>>> I could script a solution using Google's map API, but there is a limit
>>>>> of 2,500 addreses per day (I can get around them with a little patience).
>>>>>
>>>>>
>>>>> Right now the best solution I am looking at geopy for geocoding
>>>>> addresses (http://code.google.com/p/geopy/). It seems like a good
>>>>> system, I think I can use it to pull addresses out of my database and write
>>>>> back coordinates. There is one thing that I am not sure, about, though, is
>>>>> whether I am actually allowed to use the Google API without my use being
>>>>> liked to a specific web page. The terms of service and form for getting a
>>>>> Google API key require a URL linked to a Google account. In fact, it looks
>>>>> like the API can only be used through a web site:
>>>>>
>>>>>
>>>>> "5.2 *Account Key*. After supplying Google with your account
>>>>> information and the URL of your Maps API Implementation, and accepting the
>>>>> Terms, you will be issued an alphanumeric key assigned to you by Google that
>>>>> is uniquely associated with your Google Account and the URL of your Maps API
>>>>> Implementation. Your Maps API Implementation must import the Google Maps
>>>>> APIs using this key as described in the Maps APIs Documentation<http://code.google.com/apis/maps/documentation/>,
>>>>> and Google will block requests with an invalid key or invalid URL. You may
>>>>> only obtain and use a key in accordance with these Terms and the Maps
>>>>> APIs Documentation <http://code.google.com/apis/maps/documentation/>."
>>>>>
>>>>>
>>>>>
>>>>> So it looks like I can't even get it to work without a URL.
>>>>>
>>>>>
>>>>> I can always write a script that loops through results extracted from
>>>>> the database, creates URLs and parses the XML results one at a time, but
>>>>> that seems like a fairly inelegant solution.
>>>>>
>>>>>
>>>>> Does anyone have any good ideas about how to geocode a few thousand
>>>>> addresses?
>>>>>
>>>>>
>>>>> Many thanks,
>>>>>
>>>>> JP
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Discuss mailing list
>>>>> Discuss at lists.osgeo.org
>>>>> http://lists.osgeo.org/mailman/listinfo/discuss
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Discuss mailing list
>>>>> Discuss at lists.osgeo.org
>>>>> http://lists.osgeo.org/mailman/listinfo/discuss
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Discuss mailing list
>>>>> Discuss at lists.osgeo.org
>>>>> http://lists.osgeo.org/mailman/listinfo/discuss
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Discuss mailing list
>>>>> Discuss at lists.osgeo.org
>>>>> http://lists.osgeo.org/mailman/listinfo/discuss
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Discuss mailing list
>>>> Discuss at lists.osgeo.org
>>>> http://lists.osgeo.org/mailman/listinfo/discuss
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Discuss mailing list
>>> Discuss at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/discuss
>>>
>>>
>>
>>
>> --
>> Alvaro Zabala Ordóñez
>> azabala[en]gmail[punto]com
>> alvaro.zabala[en]juntadeandalucia[punto]es
>> Tlf: 954 995 572
>> Gabinete de Normalización y Calidad
>> Servicio de Coordinación y Desarrollo de Sistemas Horizontales.
>> D.G. de Innovación y Administraciones Públicas.
>> Consejería de Innovación, Ciencia y Empresa, Junta de Andalucía
>>
>>
>>
>> _______________________________________________
>> Discuss mailing list
>> Discuss at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/discuss
>>
>>
> _______________________________________________
> Discuss mailing list
> Discuss at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/discuss
>
>
>
> _______________________________________________
> Discuss mailing list
> Discuss at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/discuss/attachments/20110205/c26d97af/attachment-0002.html>