[postgis-users] Multi-server, multi-core reverse_geocode processing

[David Jantzen]

Hi Barry,

Seems like a good approach from a scaling perspective. One risk is multiple processes fetching the same file from S3 and doing unnecessary work, or worse, creating duplicate records. Do you have a locking mechanism of some kind? 

Also:
1) Have you instrumented the system so that you can identify bottlenecks? For example, how much time is spent in network calls to S3 versus the database query? How fast is the round trip to the database and should requests be batched somehow? This will be helpful information when you’re debating whether you need to add more nodes to your cluster or optimize your process.
2) Can you recover from failure? You want to be able to restart a long running batch process without re-doing a ton of work. Do failed reverse lookups need to be recorded and reported to someone?
3) Can you monitor the system? Will you have any warning about exceeding the system’s capacity before it tanks? What happens when someone asks what percentage of the points you’re processing result in valid reverse lookups? Or how many come back with multiple possibiliities?
4) Do you have automated tests around your components? While it seems pretty simple on the surface, when you combine multiple machines, possible locking issues, network requests and database queries, there’s actually a lot that can go wrong. It will be much easier to debug and maintain if you have clearly delineated responsibiliities in the codebase and tests around each part.

Hope that helps,
David

On Dec 22, 2014, at 9:49 AM, Barry McCall <barry.mccall+postgis at gmail.com> wrote:

> I was just wondering if anyone was utilizing multiple servers with a large amount of cores to harness multi-threading for batch processing? 
> 
> I'm currently running a farm of 4 systems with 32 cores per system. A number of bash scripts handle grabbing files from an s3 bucket when files exist. The design is able to scale up as needed by simply spinning up a new image. After doing a POC this weekend I've obtained some benchmark numbers. I was able to successfully reverse geocode 46,826,673 lon/lats in an average of 33 hours. (Approx 353,834 records per hour, per-machine, 1,415,337 per hour across the farm)
> 
> Was just curious if anyone else had designed something like this and what your design methodologies were?!
> 
> I've only posted a few questions but wanted to contribute what I had going on :-)
> 
> Thanks again for all the previous help!
> 
> -Barry
> _______________________________________________
> postgis-users mailing list
> postgis-users at lists.osgeo.org
> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users