[SAC] New Hardware, can we purchase now

[Regina Obe]

Just a note about Funtoo.

It's my understanding that we'd be using hardware from the Funtoo projects and the things we had outlined to use on the Funtoo hardware
were initiatives (experiments) that have been stalled because we have no hardware.  So to me this is like a supplement of our cloud server strategy.

Chris comment from IRC  (which I think he had stated before  Alex wrote below note but he had expressed previously on IRC)
"Services to bring up on the funtoo containers are nextcloud, weblate, gitlab, and drone, some or all of which we can migrate to our own hardware once it's running, but probably keeping the more compute intensive loads off our primary server."

So to me the key points are:
1) Funtoo will be providing hardware for this
2) We will be learning how to setup these services, which are things we decided fit our needs, and few of us have much experience if any with anyway.
There is a whole set of intricacies associated with a service that are specific to the service and will be the same regardless what hardware/OS we are running it on.  

3) Yes later once we get our new hardware in place, if we can ever agree on a spec, we can move these services in that we'd decided are useful.
Some we may decide don't quite fit our need.

BTW I've added a couple more items to today's agenda (the Funtoo discussion, and also how we manage changes on servers using git as we seem pretty loosey goosey there)

https://wiki.osgeo.org/wiki/SAC_Meeting_2018-03-15#Agenda

Thanks,
Regina

-----Original Message-----
From: Sac [mailto:sac-bounces at lists.osgeo.org] On Behalf Of Alex M
Sent: Wednesday, March 14, 2018 12:05 PM
To: Chris Giorgi <chrisgiorgi at gmail.com>; System Administration Committee Discussion/OSGeo <sac at lists.osgeo.org>
Subject: Re: [SAC] New Hardware, can we purchase now

My overall response, I'm a little hesitant to implement so many new technologies at the same time with only 1 person who knows them (Chris G).

My opinion
+1 on some use of ZFS, if we have a good guide
-1 on use of Funtoo, We've prefered Debian or Ubuntu for many years and have more people comfortable with them.
+1 on trying LXD
+1 on Optane
?0 on the SSD caching

1. What tool are we using to configure write-caching on the SSDs? I'd rather not be making an overly complicated database configuration.

2. That seems a reasonable answer to me, though do we still need the SSDs if we use the Optane for caching? It sounds to me like Optane or SSD would suffice.

3. Disks -  Yes if we plan to archive OSGeo Live that would benefit from larger disks. I'm a -1 on storing data for the geodata committee, unless they can find large data that is not publicly hosted elsewhere. At which point I would recommend we find partners to host the data like GeoForAll members or companies like Amazon/Google etc... Keep in mind we also need to plan for backup space. Note, I don't see the total usable disk size of backup in the wiki, can someone figure that out and add it. We need to update https://wiki.osgeo.org/wiki/SAC:Backups

New question, which disk are we installing the OS on, and therefore the ZFS packages?

Thanks,
Alex

On 03/13/2018 12:57 PM, Chris Giorgi wrote:
>  Hi Alex,
> Answers inline below:
> Take care,
>    ~~~Chris~~~
> 
> On Mon, Mar 12, 2018 at 10:41 AM, Alex M <tech_dev at wildintellect.com> wrote:
>> On 03/02/2018 12:25 PM, Regina Obe wrote:
>>> I'm in IRC meeting with Chris and he recalls the only outstanding 
>>> thing before hardware purchase was the disk size
>>>
>>> [15:17] <TemptorSent> From my reply to the mailing list a while 
>>> back, the pricing for larger drives: (+$212 for 4x10he or +$540 for 
>>> 4x12he)  [15:19] <TemptorSent> That gives us practical 
>>> double-redundant storage of 12-16TB and 16-20TB respectively, depending how we use it.
>>>
>>>
>>> If that is all, can we just get the bigger disk and move forward 
>>> with the hardware purchase.  Unless of course the purchase has already been made.
>>>
>>>
>>> Thanks,
>>> Regina
>>>
>>
>> Apologies, I dropped the ball on many things while traveling for work...
>>
>> My take on this, I was unclear on if we really understood how we 
>> would utilize the hardware for the needs, since there are a few new 
>> technologies in discussion we haven't used before. Was also in favor 
>> of small savings as we're over the line item, and that money could be 
>> used for things like people hours or 3rd party hosting, spare parts, etc...
>>
>> So a few questions:
>> 1. If we get the optane card, do we really need the SSDs? What would 
>> we put on the SSDs that would benefit from it, considering the Optane card?
> 
> The Optane is intended for caching frequently read data on very fast storage.
> As a single unmirrored device, it is not recommended for write-caching 
> of important data, but will serve quite well for temporary scratch space.
> 
> Mirrored SSDs are required for write caching to prevent failure of a 
> single device causing data loss. The size of the write cache is very 
> small by comparison to the read cache, but the write-to-read ratio is 
> much higher, necessitating the larger total DWPD*size rating. The SSDs 
> can also provide the fast tablespace for databases as needed, which 
> also have high write- amplification. The total allocated space should 
> probably be 40-60% of the device size to ensure long-term endurance. 
> The data stored on the SSDs can be automatically backed up to the 
> spinning rust on a regular basis for improved redundancy.
> 
>> 2. What caching tool will we use with the Optane? Something like 
>> fscache/CacheFS that just does everything accessed, or something 
>> configured per site like varnish/memcache etc?
> 
> We can do both if desirable, allocating large cache for the fs (L2ARC 
> in ZFS), as well as providing an explicit cache where desirable. This 
> configuration can be modified at any time, as the system's operation 
> is not dependent on the caching device being active.
> 
>> 3. Our storage growth is modest, not that I don't consider the quoted 
>> 8 or 10 TB to be reliable, but the 2 and 4 TB models have a lot more 
>> reliability data, and take significantly less time to rebuild in a 
>> Raid configuration. So how much storage do we really need for 
>> Downloads and Foss4g archives?
> 
> OSGeo-Live alone has a growth rate and retention policy that indicates 
> needs for on the order of 100GB-1TB over the next 5 years from my 
> quick calculations, not including any additional large datasets. 
> Supporting the geodata project would likely consume every bit of 
> storage we throw at it and still be thirsty for more in short order, 
> so I would propose serving only the warm data on the new server and 
> re-purposing one of the older machines for bulk cold storage and 
> backups once services have been migrated successfully.
> 
> Remember, the usable capacity will approximately equal the total 
> capacity of a single drive in a doubly redundant configuration with 4 
> drives  at proper filesystem fill ratios. We'll gain some due to 
> compression, but also want to provision for snapshots and backup of 
> the SSD based storage, so 1x single drive size is a good SWAG. 
> Resliver times for ZFS are based on actual stored data, not disk size, 
> and can be done online with minimal degradation of service, so that's 
> a moot point I believe.
> 
>> 4. Do we know what we plan to put on the SSD drives vs the Spinning Disks?
> 
> See (1).
> 
>> I think with the answers to these we'll be able to vote this week and order.
>>
>> Thanks,
>> Alex
>> _______________________________________________
>> Sac mailing list
>> Sac at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/sac

_______________________________________________
Sac mailing list
Sac at lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/sac