[Live-demo] OSGeo-Live 7.0 - big data version?

Tue Jul 9 09:30:29 PDT 2013

I like jumbo :)

As you say, Big Data is when download is not an option - so offering a download 
is not fancy.
Why should we use more data? well, a time series diagram with 3 points is not 
exciting - and we want excite, we want to have sexy demos. Isn't it the whole 
purpose of OSGeo Live: to present OS stuff attractively?

As we seem to approach a technical solution, let me ask conversely: if & when we 
get it to work, why miss the opportunity and let 4 GB go empty?

my 2 cents,
Peter

On 07/09/2013 06:18 PM, Alex Mandel wrote:
> Well, extra data can actually be placed directly on the the flash drive 
> separate from the OS, the same way we put the Mac/Win installers. That data is 
> accessible when booted, it's just read-only last time I tried unless you use 
> sudo.
>
> I agree about better representation of types of data, but I don't feel the 
> need, as I've explained previously, to enlarge the normal distro with lots of 
> additional files. Big Data tools should be able to work on subsets. 3 point 
> Time Series over a small are actually plenty useful in a workshop setting (eg. 
> Current , 50 year, 100 year - Avg Temp in a single NetCDF). This demonstrates 
> all the concepts required to know how to handle a 100 time segments later over 
> the globe when an adequate computer is given the task.
>
> If you want to offer such a large dataset I suggest it's a separate download 
> (or user initiated script) to pull the data to the extra space on their stick 
> or an easily re-packed variant of our setup called osgeo-live-jumbo.iso
>
> Thanks,
> Alex
>
> On 07/09/2013 09:09 AM, Peter Baumann wrote:
>> responding inline:
>>
>> On 07/09/2013 06:05 PM, Alex Mandel wrote:
>>> It's not a partition limit, its a per file limit.
>>> Fat32 can create up to 2 TB partitions.
>>>
>>> Since the entirety of the live boot is actually run from 1 file it is
>>> subject to that limit. So, yes if you want to try to split the OS into
>>> multiple files for different mount points (one for /, /home, /var,
>>> etc) that is feasible as long as the first one contains the /boot.
>>>
>>> Or if you don't care if a user can access files on the stick from
>>> their native OS, you could just format the whole thing differently
>>> (not sure what bios booting needs).
>>
>> I don't see that this would be bleedingly necessary for OSGeo Live, so
>> we might simply switch to ext3/4 - Cameron?
>>
>>>
>>> But I'm still missing the incentive to increase the size of the distro.
>>
>> Big Data :)
>> ...which is not only 1 PB onwards (as XLDB defines it), but also V for
>> variety - and here we have 3D image timeseries, 4D climate data, and the
>> like. A 10 MB climate data set is no fun to play with.
>>
>> See http://standards.rasdaman.org for some concrete examples (which
>> still are not "big", BTW).
>>
>> cheers,
>> Peter
>>
>>
>>>
>>> Thanks,
>>> Alex
>>>
>>> On 07/09/2013 08:59 AM, Peter Baumann wrote:
>>>> ah, I see - thanks for pointing us to that, Alex!
>>>> Looks difficult indeed for OSGeo Live.
>>>>
>>>> Another idea, maybe wildly impossible: what about allocating 2x 4 GB
>>>> partitions, and mount them after boot?
>>>>
>>>> -Peter
>>>>
>>>>
>>>> On 07/09/2013 05:55 AM, Alex Mandel wrote:
>>>>> You probably hit the file size limit of fat32
>>>>> https://en.wikipedia.org/wiki/File_Allocation_Table
>>>>> "The maximum possible size for a file on a FAT32 volume is 4 GiB"
>>>>>
>>>>> Sadly there is no universal successor to FAT32 that is easily
>>>>> accessible
>>>>> from all OS, so most external media is still fat32.
>>>>>
>>>>> NTFS is quite good but doesn't ship with Mac by default
>>>>> exFAT an attempt to bridge the gap requires work to be used on Linux
>>>>> or Mac
>>>>> EXT2/3/4 requires drivers to work on windows
>>>>>
>>>>> More info
>>>>> http://arstechnica.com/information-technology/2013/06/review-is-microsofts-new-data-sharing-system-a-cross-platform-savior/ 
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Alex
>>>>>
>>>>> On 07/08/2013 04:32 PM, Baumann, Peter wrote:
>>>>>> FWIW, we also just hit the wall when trying to get 7 GB on a stick,
>>>>>> somehow 4GB seems to be a hard limit. We  have built a rasdaman demo
>>>>>> which is a subset of standards.rasdaman.org - what's loaded runs
>>>>>> in-situ, everything else will redirect to that site (prending
>>>>>> Internet connection).
>>>>>> -Peter
>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> Dr. Peter Baumann- Professor of Computer Science, Jacobs University
>>>>>> Bremen
>>>>>>    http://www.faculty.jacobs-university.de/pbaumann
>>>>>>    mail: p.baumann at jacobs-university.detel: +49-421-200-3178, fax:
>>>>>> +49-421-200-493178
>>>>>> - Executive Director, rasdaman GmbH Bremen (HRB
>>>>>> 26793)http://www.rasdaman.com,
>>>>>>    mail: baumann at rasdaman.com
>>>>>>    tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
>>>>>> "Si forte in alienas manus oberraverit hec peregrina epistola
>>>>>> incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur
>>>>>> cui soli destinata, nec preripiat quisquam non sibi parata." (mail
>>>>>> disclaimer, AD 1083)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ________________________________________
>>>>>> From: live-demo-bounces at lists.osgeo.org
>>>>>> [live-demo-bounces at lists.osgeo.org] on behalf of Cameron Shorter
>>>>>> [cameron.shorter at gmail.com]
>>>>>> Sent: Thursday, July 04, 2013 11:00 PM
>>>>>> To: live-demo at lists.osgeo.org
>>>>>> Subject: [Live-demo] OSGeo-Live 7.0 - big data version?
>>>>>>
>>>>>> On IRC right now we are discussing the possibility of creating a "Big
>>>>>> Data" version of OSGeo-Live 7.0.
>>>>>>
>>>>>> This will likely be the standard OSGeo-Live (which will still need to
>>>>>> work stand along), plus an extra data directory which could include
>>>>>> big
>>>>>> data, such as netCDF datasets. This could be distributed as a VM or on
>>>>>> an 8Gig USB.
>>>>>>
>>>>>> I'm interested to hear thoughts on whether this will work for those
>>>>>> interested in showing big data on OSGeo-Live.
>>>>>>
>>>>>
>>>>
>>>
>>
>

-- 
Dr. Peter Baumann
  - Professor of Computer Science, Jacobs University Bremen
    www.faculty.jacobs-university.de/pbaumann
    mail: p.baumann at jacobs-university.de
    tel: +49-421-200-3178, fax: +49-421-200-493178
  - Executive Director, rasdaman GmbH Bremen (HRB 26793)
    www.rasdaman.com, mail: baumann at rasdaman.com
    tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
"Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail disclaimer, AD 1083)