[SAC] Next meeting discussion - people I'd like to hear from

[Chris Giorgi]

On Fri, Feb 16, 2018 at 4:24 PM, Alex M <tech_dev at wildintellect.com> wrote:
> On 02/16/2018 03:19 PM, Martin Spott wrote:
==8<---
>> WRT suggestions concerning "The New Hardware": Quite a few ideas have been
>> added to the agenda adressing I/O performance on the new machine.
>> Even though more I/O performance is always nice to have, I really wonder if
>> hardware limitations in this domain have been identified as current or
>> upcoming issue in OSGeo's own infrastructure.
==8<---
>>
>> Cheers,
>>       Martin.
>>
>
> There have been some performance issues with Trac/SVN, though that might
> be solved with the updates and tweaks to apache. Download does need more
> disk space along with the Foss4g archives.
>
> I'm not sure if you should spend time testing the Debian8/9 upgrades as
> all those VMs will be migrated to the new hardware in some fashion be it
> new VMs, LXD, or other method. In which case it might be saner to start
> with the newest and migrate the data and configuration over.
>
> The migration of all things off OSGeo4 and then OSGeo3(with the new
> hardware) will be a big place you can help, along with the SSL related
> tickets, etc.
>
> I am agnostic on the Optane card, the other options all seem within the
> norm of what we do. I do understand the value of it to add caching
> performance, which could remove all sorts of load from ever causing any
> issues. Some of the choices are clearly based on mitigating risk on the
> SSD drives. If the Optane prevents lots of wear on the disks that seems
> worth it.
>
> There does seems to still be plenty of debate on the precise
> implementation, etc. But other than lots of ram I'm not sure ZFS
> requires anything special. I also don't think this part of the debate
> should hold up the purchase. Since it's all done during the install and
> setup.
>
> Thanks,
> Alex
>

The rationale behind optimizing the i/o path is based on both the
mnuin data and observed service responsiveness.
Currently, latency appears to be moderate, but peaky, with disk
response times on the order of hundred millisecond being fairly
common. Write latency is generally higher than read (normal), but
occasionally appears to reach unacceptable levels for interactive use.
As load increases, the processors will be I/O bound very quickly if
the storage subsystem fails to provide sufficient read and write
throughput in the short-duration bursts typical of the types of
services provided. The machine will have plenty of ram, and few of the
services provided will have any significant computational overhead,
leaving the storage and network as the only remaining potential
bottlenecks -- and the built in networking hardware can easily handle
the entire available link provided I believe.

The reasoning behind Optane card specifically was twofold:
1 - physical considerations - The selected chassis has a very limited
number of locations to install storage; with 4HDDs in the hotswap bays
and 2SSDs in the remaining storage bays, the only location remaining
is the PCIe slot.
2 - performance and lifetime - Optane memory is both significantly
faster for random IO and more durable in terms of write cycles than
any existing SSD, which are ideal characteristics for a primary read
cache; With proper provisioning, the less expensive workstation grade
900p specified (vs DC P4800X) should have a service life greater than
7 years even with significantly higher loading than at current.
-- An alternative to utilize the available PCIe slot might be
something like this with a mirrored pair of m.2 NVMe drives:
https://www.supermicro.com/products/accessories/addon/AOC-SLG3-2M2.cfm

The hardware decisions are not contingent on the software details so
much as the intended application, so as long as the hardware appears
to be a good match to the services we expect to provide, there is no
reason to delay purchasing until a final software configuration has
been decided.

Take care,
   ~~~Chris~~~