[Landsat-pds] Status Update

[Amit Kapadia]

Thanks for tackling this job, Frank.

I guess USGS servers are a limiting factor right now. In having worked with
them over the last few months, I have never seen responses like this.
Earlier today there were responses that indicated maintenance. Hopefully it
clears up soon.

Cheers,
Amit

On Tue, Jan 13, 2015 at 2:17 PM, Frank Warmerdam <warmerdam at pobox.com>
wrote:

> Folks,
>
> I have reworked the ingestion process somewhat so that one process is run
> to download tar files from the usgs, and dump them into a "queue" in S3.
> This process will be run on a single VM at a time, but uses GNU parallel to
> operate up to four downloads at a time.  Any 503's will be missed, but
> picked up on the next run.
>
> Then the process to unpack tar files, build thumbnails, etc (the
> l8_process_scene.py) is run from the S3 queue and so it can be run
> massively in parallel if desired.  I have a Planet Labs specific executive
> that runs a bunch of jobs, collects the summary information and adds that
> to the scene_list.
>
> I'm currentlying seeing this error when I do a search for scenes against
> the USGS API.  I assume they are doing some work and it will hopefully
> working again soon.  Once it is working again I'm going to try to setup
> Jenkins jobs here to run the two stages of processing regularly. In the
> meantime there are roughly 900 scenes in the landsat-pds bucket now and
> ready for review:
>
>   https://s3-us-west-2.amazonaws.com/landsat-pds/scene_list.gz
>
> Best regards,
> Frank
>
> usgs.USGSError: Could not complete search request: 1722 ORA-01722: invalid
> number *WITH search_order AS (
>                     SELECT ROWID, row_number() OVER (ORDER BY search_date
> DESC) AS rownumber
>                         FROM landsat_8
>                         WHERE (search_date BETWEEN TO_DATE('01-Jan-2015',
> 'DD-MON-YYYY') AND TO_DATE('02-Jan-2015','DD-MON-YYYY')) AND
> (SPATIAL_LOCATION IS NOT NULL)
>                     )
>
>                     SELECT A.ENTITY_ID,
> DOWNLOADABLE,ON_DEMAND,to_char(ACQUISITION_DATE,
> 'YYYY-MM-DD')||'T'||to_char(ACQUISITION_DATE,'HH24:MI:SS')||'Z'
> "acquisitionDate",to_char(START_TIME,
> 'YYYY-MM-DD')||'T'||to_char(START_TIME, 'HH24:MI:SS')||'Z'
> "acquisitionStart",to_char(STOP_TIME,
> 'YYYY-MM-DD')||'T'||to_char(STOP_TIME, 'HH24:MI:SS')||'Z'
> "acquisitionEnd",LL_CORNER_LAT "lowerLeftLat",LL_CORNER_LON
> "lowerLeftLng",UL_CORNER_LON "upperLeftLng",UL_CORNER_LAT
> "upperLeftLat",UR_CORNER_LAT "upperRightLat",UR_CORNER_LON
> "upperRightLng",LR_CORNER_LAT "lowerRightLat",LR_CORNER_LON
> "lowerRightLng",to_char(decode(DATE_UPDATED, null, SEARCH_DATE,
> DATE_UPDATED), 'YYYY-MM-DD')||'T'||to_char(decode(DATE_UPDATED, null,
> SEARCH_DATE, DATE_UPDATED),'HH24:MI:SS')||'Z' "modifiedDate",'Entity ID:
> '||ee_display_id||', Acquisition Date: '||acquisition_date||', Path:
> '||WRS_PATH||', Row: '||WRS_ROW
> "summary",SDO_UTIL.TO_WKTGEOMETRY(SDO_GEOM.SDO_MBR(SPATIAL_LOCATION)) as
> SCENE_BOUNDS
>                         FROM landsat_8 A
>                         INNER JOIN search_order SO ON A.ROWID = SO.ROWID
>                         WHERE SO.rownumber <= 500 AND SO.rownumber >= 1
>                         ORDER BY SO.rownumber
>
>
> --
>
> ---------------------------------------+--------------------------------------
> I set the clouds in motion - turn up   | Frank Warmerdam,
> warmerdam at pobox.com
> light and sound - activate the windows | http://pobox.com/~warmerdam
> and watch the world go round - Rush    | Geospatial Software Developer
>
> _______________________________________________
> Landsat-pds mailing list
> Landsat-pds at lists.osgeo.org
> http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/landsat-pds/attachments/20150113/539d1d12/attachment.html>