[Live-demo] OSGeo-Live 7.0 - big data version?

[Peter Baumann]

responding inline:

On 07/09/2013 06:05 PM, Alex Mandel wrote:
> It's not a partition limit, its a per file limit.
> Fat32 can create up to 2 TB partitions.
>
> Since the entirety of the live boot is actually run from 1 file it is subject 
> to that limit. So, yes if you want to try to split the OS into multiple files 
> for different mount points (one for /, /home, /var, etc) that is feasible as 
> long as the first one contains the /boot.
>
> Or if you don't care if a user can access files on the stick from their native 
> OS, you could just format the whole thing differently (not sure what bios 
> booting needs).

I don't see that this would be bleedingly necessary for OSGeo Live, so we might 
simply switch to ext3/4 - Cameron?

>
> But I'm still missing the incentive to increase the size of the distro.

Big Data :)
...which is not only 1 PB onwards (as XLDB defines it), but also V for variety - 
and here we have 3D image timeseries, 4D climate data, and the like. A 10 MB 
climate data set is no fun to play with.

See http://standards.rasdaman.org for some concrete examples (which still are 
not "big", BTW).

cheers,
Peter


>
> Thanks,
> Alex
>
> On 07/09/2013 08:59 AM, Peter Baumann wrote:
>> ah, I see - thanks for pointing us to that, Alex!
>> Looks difficult indeed for OSGeo Live.
>>
>> Another idea, maybe wildly impossible: what about allocating 2x 4 GB
>> partitions, and mount them after boot?
>>
>> -Peter
>>
>>
>> On 07/09/2013 05:55 AM, Alex Mandel wrote:
>>> You probably hit the file size limit of fat32
>>> https://en.wikipedia.org/wiki/File_Allocation_Table
>>> "The maximum possible size for a file on a FAT32 volume is 4 GiB"
>>>
>>> Sadly there is no universal successor to FAT32 that is easily accessible
>>> from all OS, so most external media is still fat32.
>>>
>>> NTFS is quite good but doesn't ship with Mac by default
>>> exFAT an attempt to bridge the gap requires work to be used on Linux
>>> or Mac
>>> EXT2/3/4 requires drivers to work on windows
>>>
>>> More info
>>> http://arstechnica.com/information-technology/2013/06/review-is-microsofts-new-data-sharing-system-a-cross-platform-savior/ 
>>>
>>>
>>>
>>> Thanks,
>>> Alex
>>>
>>> On 07/08/2013 04:32 PM, Baumann, Peter wrote:
>>>> FWIW, we also just hit the wall when trying to get 7 GB on a stick,
>>>> somehow 4GB seems to be a hard limit. We  have built a rasdaman demo
>>>> which is a subset of standards.rasdaman.org - what's loaded runs
>>>> in-situ, everything else will redirect to that site (prending
>>>> Internet connection).
>>>> -Peter
>>>>
>>>>
>>>> -- 
>>>> Dr. Peter Baumann- Professor of Computer Science, Jacobs University
>>>> Bremen
>>>>    http://www.faculty.jacobs-university.de/pbaumann
>>>>    mail: p.baumann at jacobs-university.detel: +49-421-200-3178, fax:
>>>> +49-421-200-493178
>>>> - Executive Director, rasdaman GmbH Bremen (HRB
>>>> 26793)http://www.rasdaman.com,
>>>>    mail: baumann at rasdaman.com
>>>>    tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
>>>> "Si forte in alienas manus oberraverit hec peregrina epistola
>>>> incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur
>>>> cui soli destinata, nec preripiat quisquam non sibi parata." (mail
>>>> disclaimer, AD 1083)
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________________
>>>> From: live-demo-bounces at lists.osgeo.org
>>>> [live-demo-bounces at lists.osgeo.org] on behalf of Cameron Shorter
>>>> [cameron.shorter at gmail.com]
>>>> Sent: Thursday, July 04, 2013 11:00 PM
>>>> To: live-demo at lists.osgeo.org
>>>> Subject: [Live-demo] OSGeo-Live 7.0 - big data version?
>>>>
>>>> On IRC right now we are discussing the possibility of creating a "Big
>>>> Data" version of OSGeo-Live 7.0.
>>>>
>>>> This will likely be the standard OSGeo-Live (which will still need to
>>>> work stand along), plus an extra data directory which could include big
>>>> data, such as netCDF datasets. This could be distributed as a VM or on
>>>> an 8Gig USB.
>>>>
>>>> I'm interested to hear thoughts on whether this will work for those
>>>> interested in showing big data on OSGeo-Live.
>>>>
>>>
>>
>

-- 
Dr. Peter Baumann
  - Professor of Computer Science, Jacobs University Bremen
    www.faculty.jacobs-university.de/pbaumann
    mail: p.baumann at jacobs-university.de
    tel: +49-421-200-3178, fax: +49-421-200-493178
  - Executive Director, rasdaman GmbH Bremen (HRB 26793)
    www.rasdaman.com, mail: baumann at rasdaman.com
    tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
"Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail disclaimer, AD 1083)