[Landsat-pds] Converting MTL.txt to JSON

Mon Mar 2 11:49:18 PST 2015

>From glancing at it, this looks good.

What's the easiest way to build this into the ingestor scripts?

On Feb 26, 2015, at 9:52 AM, Layton, Jeff <jefflay at amazon.com<mailto:jefflay at amazon.com>> wrote:

I think I finally have it. I’m attaching the original MTL file, the JSON output (testfile4) and the code I used to create it (test7.py).

Jeff

From: landsat-pds-bounces at lists.osgeo.org<mailto:landsat-pds-bounces at lists.osgeo.org> [mailto:landsat-pds-bounces at lists.osgeo.org] On Behalf Of Layton, Jeff
Sent: Thursday, February 26, 2015 12:44 PM
To: Frank Warmerdam
Cc: landsat-pds at lists.osgeo.org<mailto:landsat-pds at lists.osgeo.org>
Subject: Re: [Landsat-pds] Converting MTL.txt to JSON

Doh! Good catch – silly typo in the code. Give me a minute…

From: fwarmerdam at gmail.com<mailto:fwarmerdam at gmail.com> [mailto:fwarmerdam at gmail.com] On Behalf Of Frank Warmerdam
Sent: Thursday, February 26, 2015 12:35 PM
To: Layton, Jeff
Cc: Sundwall, Jed; landsat-pds at lists.osgeo.org<mailto:landsat-pds at lists.osgeo.org>
Subject: Re: [Landsat-pds] Converting MTL.txt to JSON

Jeff,

I'm not sure what scene that was generated for, but I don't see any values that are strings. For instance, I expect the PRODUCT_METADATA group to have an item called DATA_TYPE with a string value (ie. "L1T").

Also, the whole METADATA_FILE_INFO section appears to be missing.

I am suspecting you lost everything that failed to translate to numbers?

Also, I see now that you have an content-type set *within* the json.  I'm not sure what the utility of that is as opposed to setting it on the S3 object as Jed was discussing.

Best regards,
Frank

On Thu, Feb 26, 2015 at 9:20 AM, Layton, Jeff <jefflay at amazon.com<mailto:jefflay at amazon.com>> wrote:
Everyone,

Try the attached JSON file. It should have everything numeric that can be numerical :)  Let me know if you find any errors (my cursory examination didn’t find any). If it passes muster, I’ll fire off the code.

Thanks!

Jeff

From: Sundwall, Jed
Sent: Thursday, February 26, 2015 11:25 AM
To: Frank Warmerdam
Cc: Layton, Jeff; landsat-pds at lists.osgeo.org<mailto:landsat-pds at lists.osgeo.org>

Subject: Re: [Landsat-pds] Converting MTL.txt to JSON

I’ve uploaded the file to https://s3-us-west-2.amazonaws.com/awsopendata/test/testfile2.json

The mime type is “application/json” which should be appropriate.

I’m not sure where this would fit within the landsat_ingestor scripts, but setting the mime type with boto is as simple as:

 if s3_path.endswith(‘.json'):
         key.content_type = ‘application/json'

Re: converting the numbers, Jeff, could you share the script you’ve written to create the json files? Perhaps someone will be able to see a simple way to make the conversion.

Jed.

On Feb 26, 2015, at 10:55 AM, Frank Warmerdam <warmerdam at pobox.com<mailto:warmerdam at pobox.com>> wrote:

Jeff,

>From an attachment I (don't think) I can know if mime types are getting properly attached in S3.  If you can point me to an object in S3 I can try and check.  I see the AWS S3 browser shows the mimetype under the "Metadata" properties section if you have access to that for our bucket.

I'll second Charlie's suggestion to preserve numeric types for numeric fields.  I don't know this gets obscured by mtlutils.

Best regards,
Frank

On Thu, Feb 26, 2015 at 7:38 AM, Layton, Jeff <jefflay at amazon.com<mailto:jefflay at amazon.com>> wrote:
Frank,

I think I’ve added the mimetype correctly.Can you look at the attached file and let me know if I did?

Thanks!

Jeff

From: fwarmerdam at gmail.com<mailto:fwarmerdam at gmail.com> [mailto:fwarmerdam at gmail.com<mailto:fwarmerdam at gmail.com>] On Behalf Of Frank Warmerdam
Sent: Wednesday, February 25, 2015 5:20 PM
To: Layton, Jeff
Cc: landsat-pds at lists.osgeo.org<mailto:landsat-pds at lists.osgeo.org>
Subject: Re: [Landsat-pds] Converting MTL.txt to JSON

Jeff,

This looks good to me.  I'd suggest we drop these in the same directory as the MTL txt files, and just name them .json.

eg.
LC80150252015005LGN00_B4.TIF
LC80150252015005LGN00_B10.TIF
LC80150252015005LGN00_B5.TIF
LC80150252015005LGN00_B2.TIF
LC80150252015005LGN00_B9.TIF
LC80150252015005LGN00_B1.TIF
LC80150252015005LGN00_B7.TIF
LC80150252015005LGN00_B8.TIF
LC80150252015005LGN00_B11.TIF
LC80150252015005LGN00_B3.TIF
LC80150252015005LGN00_BQA.TIF
LC80150252015005LGN00_B6.TIF
LC80150252015005LGN00_MTL.txt
LC80150252015005LGN00_MTL.json

It might be nice to have the mimetype set properly for json too.

With regard to putting them all in one place, the downside I see to that is they aren't as likely to be noticed by folks who have maneuvered to a scene.  The scene_list.gz file already has the links to each scene directory for folks who need a directory-in-a-file.

Best regards,
Frank

On Wed, Feb 25, 2015 at 1:13 PM, Layton, Jeff <jefflay at amazon.com<mailto:jefflay at amazon.com>> wrote:
Hello everyone,

I just joined the mailing list and I’m working with Jed from AWS (I work for AWS as well). The first project I did was write a simple Python tool to convert the MTL.txt files to JSON. I’m attaching a sample MTL file and the resulting JSON. Let me know what you think!

Thanks!

Jeff

Dr. Jeff Layton
Solutions Architect – HPC
Amazon Web Services
jefflay at amazon.com<mailto:jefflay at amazon.com>
678-310-3685<tel:678-310-3685>

_______________________________________________
Landsat-pds mailing list
Landsat-pds at lists.osgeo.org<mailto:Landsat-pds at lists.osgeo.org>
http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds

--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com<mailto:warmerdam at pobox.com>
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Software Developer

--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com<mailto:warmerdam at pobox.com>
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Software Developer
_______________________________________________
Landsat-pds mailing list
Landsat-pds at lists.osgeo.org<mailto:Landsat-pds at lists.osgeo.org>
http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds

--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com<mailto:warmerdam at pobox.com>
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Software Developer
<LC80361102015008LGN00_MTL.txt>
<test7.py>
<testfile4>
_______________________________________________
Landsat-pds mailing list
Landsat-pds at lists.osgeo.org<mailto:Landsat-pds at lists.osgeo.org>
http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/landsat-pds/attachments/20150302/a71e01dd/attachment-0001.html>