[Landsat-pds] Status Update

[Frank Warmerdam]

Folks,

I have reworked the ingestion process somewhat so that one process is run
to download tar files from the usgs, and dump them into a "queue" in S3.
This process will be run on a single VM at a time, but uses GNU parallel to
operate up to four downloads at a time.  Any 503's will be missed, but
picked up on the next run.

Then the process to unpack tar files, build thumbnails, etc (the
l8_process_scene.py) is run from the S3 queue and so it can be run
massively in parallel if desired.  I have a Planet Labs specific executive
that runs a bunch of jobs, collects the summary information and adds that
to the scene_list.

I'm currentlying seeing this error when I do a search for scenes against
the USGS API.  I assume they are doing some work and it will hopefully
working again soon.  Once it is working again I'm going to try to setup
Jenkins jobs here to run the two stages of processing regularly. In the
meantime there are roughly 900 scenes in the landsat-pds bucket now and
ready for review:

  https://s3-us-west-2.amazonaws.com/landsat-pds/scene_list.gz

Best regards,
Frank

usgs.USGSError: Could not complete search request: 1722 ORA-01722: invalid
number *WITH search_order AS (
                    SELECT ROWID, row_number() OVER (ORDER BY search_date
DESC) AS rownumber
                        FROM landsat_8
                        WHERE (search_date BETWEEN TO_DATE('01-Jan-2015',
'DD-MON-YYYY') AND TO_DATE('02-Jan-2015','DD-MON-YYYY')) AND
(SPATIAL_LOCATION IS NOT NULL)
                    )

                    SELECT A.ENTITY_ID,
DOWNLOADABLE,ON_DEMAND,to_char(ACQUISITION_DATE,
'YYYY-MM-DD')||'T'||to_char(ACQUISITION_DATE,'HH24:MI:SS')||'Z'
"acquisitionDate",to_char(START_TIME,
'YYYY-MM-DD')||'T'||to_char(START_TIME, 'HH24:MI:SS')||'Z'
"acquisitionStart",to_char(STOP_TIME,
'YYYY-MM-DD')||'T'||to_char(STOP_TIME, 'HH24:MI:SS')||'Z'
"acquisitionEnd",LL_CORNER_LAT "lowerLeftLat",LL_CORNER_LON
"lowerLeftLng",UL_CORNER_LON "upperLeftLng",UL_CORNER_LAT
"upperLeftLat",UR_CORNER_LAT "upperRightLat",UR_CORNER_LON
"upperRightLng",LR_CORNER_LAT "lowerRightLat",LR_CORNER_LON
"lowerRightLng",to_char(decode(DATE_UPDATED, null, SEARCH_DATE,
DATE_UPDATED), 'YYYY-MM-DD')||'T'||to_char(decode(DATE_UPDATED, null,
SEARCH_DATE, DATE_UPDATED),'HH24:MI:SS')||'Z' "modifiedDate",'Entity ID:
'||ee_display_id||', Acquisition Date: '||acquisition_date||', Path:
'||WRS_PATH||', Row: '||WRS_ROW
"summary",SDO_UTIL.TO_WKTGEOMETRY(SDO_GEOM.SDO_MBR(SPATIAL_LOCATION)) as
SCENE_BOUNDS
                        FROM landsat_8 A
                        INNER JOIN search_order SO ON A.ROWID = SO.ROWID
                        WHERE SO.rownumber <= 500 AND SO.rownumber >= 1
                        ORDER BY SO.rownumber


-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam,
warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Software Developer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/landsat-pds/attachments/20150113/e57d4b54/attachment-0001.html>