<div dir="ltr">Jeff,<div><br></div><div>I'm not sure what scene that was generated for, but I don't see any values that are strings. For instance, I expect the PRODUCT_METADATA group to have an item called DATA_TYPE with a string value (ie. "L1T"). </div><div><br></div><div>Also, the whole METADATA_FILE_INFO section appears to be missing.</div><div><br></div><div>I am suspecting you lost everything that failed to translate to numbers? </div><div><br></div><div>Also, I see now that you have an content-type set *within* the json. I'm not sure what the utility of that is as opposed to setting it on the S3 object as Jed was discussing.</div><div><br></div><div>Best regards,</div><div>Frank</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 26, 2015 at 9:20 AM, Layton, Jeff <span dir="ltr"><<a href="mailto:jefflay@amazon.com" target="_blank">jefflay@amazon.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Everyone,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Try the attached JSON file. It should have everything numeric that can be numerical
</span><span style="font-size:11.0pt;font-family:Wingdings;color:#1f497d">J</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> Let me know if you find any errors (my cursory examination didn’t find any). If it passes muster,
I’ll fire off the code.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Thanks!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Jeff<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Sundwall, Jed
<br><span class="">
<b>Sent:</b> Thursday, February 26, 2015 11:25 AM<br>
<b>To:</b> Frank Warmerdam<br>
</span><b>Cc:</b> Layton, Jeff; <a href="mailto:landsat-pds@lists.osgeo.org" target="_blank">landsat-pds@lists.osgeo.org</a></span></p><div><div class="h5"><br>
<b>Subject:</b> Re: [Landsat-pds] Converting MTL.txt to JSON<u></u><u></u></div></div><p></p>
</div>
</div><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I’ve uploaded the file to <a href="https://s3-us-west-2.amazonaws.com/awsopendata/test/testfile2.json" target="_blank">https://s3-us-west-2.amazonaws.com/awsopendata/test/testfile2.json</a><u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">The mime type is “application/json” which should be appropriate.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I’m not sure where this would fit within the landsat_ingestor scripts, but setting the mime type with boto is as simple as:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"> if s3_path.endswith(‘.json'):<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> key.content_type = ‘application/json'<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Re: converting the numbers, Jeff, could you share the script you’ve written to create the json files? Perhaps someone will be able to see a simple way to make the conversion.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Jed.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Feb 26, 2015, at 10:55 AM, Frank Warmerdam <<a href="mailto:warmerdam@pobox.com" target="_blank">warmerdam@pobox.com</a>> wrote:<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">Jeff,<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">From an attachment I (don't think) I can know if mime types are getting properly attached in S3. If you can point me to an object in S3 I can try and check. I see the AWS
S3 browser shows the mimetype under the "Metadata" properties section if you have access to that for our bucket. <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">I'll second Charlie's suggestion to preserve numeric types for numeric fields. I don't know this gets obscured by mtlutils. <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">Best regards,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">Frank<u></u><u></u></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">On Thu, Feb 26, 2015 at 7:38 AM, Layton, Jeff<span> </span><<a href="mailto:jefflay@amazon.com" target="_blank">jefflay@amazon.com</a>><span> </span>wrote:<u></u><u></u></span></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Frank,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">I think I’ve added the mimetype correctly.Can you look at the attached file and let me know if I did?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Thanks!</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Jeff</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><a href="mailto:fwarmerdam@gmail.com" target="_blank">fwarmerdam@gmail.com</a><span> </span>[mailto:<a href="mailto:fwarmerdam@gmail.com" target="_blank">fwarmerdam@gmail.com</a>]<span> </span><b>On
Behalf Of<span> </span></b>Frank Warmerdam<br>
<b>Sent:</b><span> </span>Wednesday, February 25, 2015 5:20 PM<br>
<b>To:</b><span> </span>Layton, Jeff<br>
<b>Cc:</b><span> </span><a href="mailto:landsat-pds@lists.osgeo.org" target="_blank">landsat-pds@lists.osgeo.org</a><br>
<b>Subject:</b><span> </span>Re: [Landsat-pds] Converting MTL.txt to JSON</span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">Jeff,<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">This looks good to me. I'd suggest we drop these in the same directory as the MTL txt files, and just name them .json. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">eg.<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">LC80150252015005LGN00_B4.TIF<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">LC80150252015005LGN00_B10.TIF<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">LC80150252015005LGN00_B5.TIF<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">LC80150252015005LGN00_B2.TIF<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">LC80150252015005LGN00_B9.TIF<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">LC80150252015005LGN00_B1.TIF<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">LC80150252015005LGN00_B7.TIF<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">LC80150252015005LGN00_B8.TIF<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">LC80150252015005LGN00_B11.TIF<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">LC80150252015005LGN00_B3.TIF<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">LC80150252015005LGN00_BQA.TIF<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">LC80150252015005LGN00_B6.TIF<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">LC80150252015005LGN00_MTL.txt<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal">LC80150252015005LGN00_MTL.json<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">It might be nice to have the mimetype set properly for json too.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">With regard to putting them all in one place, the downside I see to that is they aren't as likely to be noticed by folks who have maneuvered to a scene. The scene_list.gz file
already has the links to each scene directory for folks who need a directory-in-a-file.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Best regards,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Frank<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On Wed, Feb 25, 2015 at 1:13 PM, Layton, Jeff <<a href="mailto:jefflay@amazon.com" target="_blank">jefflay@amazon.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">Hello everyone,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">I just joined the mailing list and I’m working with Jed from AWS (I work for AWS as well). The first project I did was write a simple Python tool to convert the MTL.txt files to
JSON. I’m attaching a sample MTL file and the resulting JSON. Let me know what you think!<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Thanks!<u></u><u></u></p>
<p class="MsoNormal"><br>
Jeff<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Dr. Jeff Layton<u></u><u></u></p>
<p class="MsoNormal">Solutions Architect – HPC<u></u><u></u></p>
<p class="MsoNormal">Amazon Web Services<u></u><u></u></p>
<p class="MsoNormal"><a href="mailto:jefflay@amazon.com" target="_blank"><span style="color:#0563c1">jefflay@amazon.com</span></a><u></u><u></u></p>
<p class="MsoNormal"><a href="tel:678-310-3685" target="_blank">678-310-3685</a><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Landsat-pds mailing list<br>
<a href="mailto:Landsat-pds@lists.osgeo.org" target="_blank">Landsat-pds@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds" target="_blank">http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal">--<span> </span><u></u><u></u></p>
<div>
<p class="MsoNormal">---------------------------------------+--------------------------------------<br>
I set the clouds in motion - turn up | Frank Warmerdam,<span> </span><a href="mailto:warmerdam@pobox.com" target="_blank">warmerdam@pobox.com</a><br>
light and sound - activate the windows |<span> </span><a href="http://pobox.com/~warmerdam" target="_blank">http://pobox.com/~warmerdam</a><br>
and watch the world go round - Rush | Geospatial Software Developer<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
<br clear="all">
<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><u></u> <u></u></span></p>
</div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">--<span> </span><u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">---------------------------------------+--------------------------------------<br>
I set the clouds in motion - turn up | Frank Warmerdam,<span> </span><a href="mailto:warmerdam@pobox.com" target="_blank">warmerdam@pobox.com</a><br>
light and sound - activate the windows |<span> </span><a href="http://pobox.com/~warmerdam" target="_blank">http://pobox.com/~warmerdam</a><br>
and watch the world go round - Rush | Geospatial Software Developer<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">_______________________________________________<br>
Landsat-pds mailing list<br>
</span><a href="mailto:Landsat-pds@lists.osgeo.org" target="_blank"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">Landsat-pds@lists.osgeo.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
</span><a href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds" target="_blank"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds</span></a><u></u><u></u></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">---------------------------------------+--------------------------------------<br>I set the clouds in motion - turn up | Frank Warmerdam, <a href="mailto:warmerdam@pobox.com" target="_blank">warmerdam@pobox.com</a><br>light and sound - activate the windows | <a href="http://pobox.com/~warmerdam" target="_blank">http://pobox.com/~warmerdam</a><br>and watch the world go round - Rush | Geospatial Software Developer<br></div>
</div>