[OSGeo-Discuss] Does rasdaman CE solve an open source geospatial problem?

Sun May 22 06:23:07 PDT 2016

Cameron, my comment is in-line:

On 21/05/16 14:06, Cameron Shorter wrote:
> Hi Edzer,
> Thank you for raising this topic questioning the value of radsaman
> community edition. It is pertinent considering recent discussions about
> Rasdaman incubation.
> 
> Peter, your comments about programmers wanting to get paid for their
> work is valid, but does not provide justification for OSGeo promoting a
> proprietary business model. OSGeo is in the business of promoting open
> source software, and helping people who create open source software.
> 
> Rasdaman's business model is in a grey zone. It provides a community
> edition and a proprietary edition. This is often referred to as an "open
> core" business model, or sometimes less favorably called "crippleware".
> I think Rasdaman is the only OSGeo (proposed) project which provides an
> open core model. All prior projects have been pure open source.
> 
> Although a an open core model deviates from OSGeo's original principles,
> one could argue that Rasdaman community edition stands on its own as a
> valuable, quality open source geospatial application by itself, worthy
> of OSGeo promotion.
> 
> Edzer's comments appear to counter this argument. Edzer, I understand
> you suggest Rasdaman community edition is of little value for real world
> problems?

Yes; the example Peter mentions, http://planetserver.eu/ does solve a
problem: providing a download service for large scale imagery, and it is
good if rasdaman CE can do this for 20 Tb, with sufficient performance.
There are however several other open source technologies that can also
do this, potentially much simpler (e.g. thredds data server, maybe even
gdal VRT).

The problems where array data management systems really come into play
is scalable computing: doing something useful with large data sets
without having to download all the data first ("bringing the
computations to the data, instead of data to the computations") - this
is what much of the big data scalability fanfare is about, not about
serving or downloading data. And this is where rasdaman CE does not
scale - only EE seems to do so.

The longer we (the open source community) wait with a good solution to
this problem, the more ground we loose to Google Earth Engine, a very
nice service that is bad for science. See also the last slide ("Google
Earth Engine is Evil") in the excellent presentation by Jordi Inglada,
held last week at the Living Planet Symposium:

http://jordiinglada.net/stok/LivingPlanet/LandCoverSlides.pdf

You might see this as a call to arms, and it is, but a new thread will
be needed to follow up on this.

> Extending from this, OSGeo endorsement of Rasdaman should be questioned
> and potentially withdrawn.
> 
> I'd be interested to hear opinions of others in the field as to whether
> Rasdaman community version is of value for real-world production systems
> by itself.
> 
> A deeper question for the greater OSGeo community is should OSGeo
> endorse Open Core business models?
> 
> Warm regards, Cameron
> 
> On 21/05/2016 6:07 pm, Peter Baumann wrote:
>> oh, just looking at the subject again:
>>
>> several service providers believe indeed rasdaman community does offer
>> a significant advantage:
>> - see the download figures on www.rasdaman.org
>> - concretely, see www.planetserver.eu which is running rasdaman
>> community on - I believe - about 20 TB of Planetary Science data.
>>
>> -Peter
>>
>>
>> On 05/21/2016 09:56 AM, Peter Baumann wrote:
>>> Hm, first of all: this is opening a different thread, talking about
>>> functionality of rasdaman community. Next, it is based on assumptions
>>> - without details (because off topic): conclusions are wrong.
>>>
>>> But to respond to the core message emphasized in the first paragraph:
>>> I respectfully disagree. In particular, such a position does not
>>> benefit the open source community very much as I am trying to explain
>>> below.
>>>
>>> TL;DR:
>>>
>>> You have a strong expertise in Geoinformatics, I know something about
>>> Computer Science. This is where we can talk as professors and
>>> scientists. Your statement is about economics, industry etc. Having
>>> an opinion there (and articulate it) is fair, but in these fields our
>>> opinion weighs not more than anyone else's in the street. We should
>>> not attempt to attain importance through inapplicable roles.
>>>
>>> Let us look at a professor. They have a conveniently high salary
>>> which is paid by society, that is: tax payers. Nobody can influence
>>> what a professor does and how much return s/he generates for society.
>>>
>>> A single open source developer (or a small group, whatever) do not
>>> experience this convenience. They have a dream where they invest,
>>> they try to not make money for getting richer than a professor ;-)
>>> but merely for their economic survival. Some (in particular
>>> scientists) enjoy the money rain coming from publicly funded projects
>>> (again: the tax payer subsidizes), but most in the community have to
>>> struggle hard. They face reluctant customers, competition by the
>>> giants in the market, and many more obstacles.
>>>
>>> From the cosy place of a lifelong position with a secured salary and
>>> decent retirement funds it is easy to say that all software should be
>>> free like free beer (quote from below: "can be reproduced by other
>>> scientists without prohibitive license costs").
>>>
>>> If the open source movement cannibalizes itself it will make it all
>>> so easy for the big players to maintain their dominance, they will
>>> silently applaud. Quoting Jeroen:
>>> > NEVER IGNORE COMPANIES AGAIN IN OSGEO OR FOSS4G! THEY ARE NOT A
>>> THREAT, THEY ARE A NECESSITY.
>>>
>>> That said: It is entirely ok to have the opinion you have. Others,
>>> though, may disagree. I am one of those.
>>>
>>> respectfully,
>>> Peter
>>>
>>>
>>>
>>> On 05/20/2016 09:30 AM, Edzer Pebesma wrote:
>>>> As a scientist, I teach my students that for doing science it is a
>>>> requirement to work with open source software, because only then
>>>> workflows are fully transparent and can be reproduced by other
>>>> scientists without prohibitive license costs. Currently, working
>>>> with large amounts of earth observation (EO) or climate model data
>>>> typically requires to download these data tile by tile, stitch them
>>>> together, and go through all of them. Array databases may simplify
>>>> this substantially: after ingesting the tiles, they can directly
>>>> work on the whole data as a multi-dimensinal array ("data cube").
>>>> Computations on these array are typically embarassingly parallel,
>>>> and scale up with the number of cores in a cluster.
>>>>
>>>> Rasdaman is an array data base that comes in two flavours, the open
>>>> source community edition (CE) and the commercial enterprise edition
>>>> (EE). The differences between the two are clear [1]. When I want to
>>>> use rasdaman CE (open source) for scalable image analysis, I get
>>>> stuck waiting for one core to finish everything [1]. This is not
>>>> going to solve any problems related to computing on large data,
>>>> and is not scalable. The bold claim that rasdaman.org opens with
>>>> ("This worldwide leading array analytics engine distinguishes itself
>>>> by its flexibility, performance, and scalability") is not true for
>>>> the CE advertised. This has been mentioned in the past on mailing
>>>> lists [2,3], but the typical answer from Peter Baumann diverts into
>>>> other arguments. Also the benchmark graph (photo from an AGU poster)
>>>> [4] that Peter sent this week [5] must refer to the enterprise
>>>> edition, since Spark and Hive both scale, but rasdaman CE does not [3].
>>>>
>>>> I assume that on the discussions on this list, ONLY the open
>>>> source community edition is considered, compared, and discussed,
>>>> as a potential future OSGeo project.
>>>>
>>>> OSGeo supports the needs of the open source geospatial community [6].
>>>>
>>>> Given
>>>>
>>>>  * the bold claims and continuing confusion about whether,
>>>>    and which, rasdaman is scalable,
>>>>  * the need for OSGeo to give good advice to prospective users
>>>>    about technologies that do scale EO data analysis,
>>>>  * the current (unfilled!) needs of scientists for good, open source
>>>>    software for such analysis, and
>>>>  * the potential conflict of interest of its creator [7],
>>>>
>>>> I wonder wether OSGeo should recommend rasdaman CE to the open
>>>> source geospatial community.
>>>>
>>>>
>>>> [1] http://rasdaman.org/wiki/Features
>>>> [2] https://lists.osgeo.org/pipermail/incubator/2014-October/002540.html
>>>> [3] https://groups.google.com/forum/#!topic/rasdaman-users/66XL3tmDDQI
>>>> [4]
>>>> https://lists.osgeo.org/pipermail/discuss/attachments/20160515/49200cd4/attachment.jpg
>>>> [5] https://lists.osgeo.org/pipermail/discuss/2016-May/016099.html
>>>> [6] http://www.osgeo.org/content/faq/foundation_faq.html
>>>> [7] https://lists.osgeo.org/pipermail/discuss/2016-May/016045.html
>>>>
>>>>
>>>> _______________________________________________
>>>> Discuss mailing list
>>>> Discuss at lists.osgeo.org
>>>> http://lists.osgeo.org/mailman/listinfo/discuss
>>>
>>> -- 
>>> Dr. Peter Baumann
>>>  - Professor of Computer Science, Jacobs University Bremen
>>>    www.faculty.jacobs-university.de/pbaumann
>>>    mail: p.baumann at jacobs-university.de
>>>    tel: +49-421-200-3178, fax: +49-421-200-493178
>>>  - Executive Director, rasdaman GmbH Bremen (HRB 26793)
>>>    www.rasdaman.com, mail: baumann at rasdaman.com
>>>    tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
>>> "Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail disclaimer, AD 1083)
>>>
>>>
>>
>> -- 
>> Dr. Peter Baumann
>>  - Professor of Computer Science, Jacobs University Bremen
>>    www.faculty.jacobs-university.de/pbaumann
>>    mail: p.baumann at jacobs-university.de
>>    tel: +49-421-200-3178, fax: +49-421-200-493178
>>  - Executive Director, rasdaman GmbH Bremen (HRB 26793)
>>    www.rasdaman.com, mail: baumann at rasdaman.com
>>    tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
>> "Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail disclaimer, AD 1083)
>>
>>
>>
>>
>> _______________________________________________
>> Discuss mailing list
>> Discuss at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/discuss
> 
> -- 
> Cameron Shorter,
> Software and Data Solutions Manager
> LISAsoft
> Suite 112, Jones Bay Wharf,
> 26 - 32 Pirrama Rd, Pyrmont NSW 2009
> 
> P +61 2 9009 5000,  W www.lisasoft.com,  F +61 2 9009 5099
> 

-- 
Edzer Pebesma
Institute for Geoinformatics  (ifgi),  University of Münster
Heisenbergstraße 2, 48149 Münster, Germany; +49 251 83 33081
Journal of Statistical Software:   http://www.jstatsoft.org/
Computers & Geosciences:   http://elsevier.com/locate/cageo/
Spatial Statistics Society http://www.spatialstatistics.info

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: OpenPGP digital signature
URL: <http://lists.osgeo.org/pipermail/discuss/attachments/20160522/cb667aef/attachment.sig>