[all] etcd3 as base service - update

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[all] etcd3 as base service - update

Davanum Srinivas
Team,

Here's the update to the base services resolution from the TC:
https://governance.openstack.org/tc/reference/base-services.html

First request is to Distros, Packagers, Deployers, anyone who
installs/configures OpenStack:
Please make sure you have latest etcd 3.x available in your
environment for Services to use, Fedora already does, we need help in
making sure all distros and architectures are covered.

Any project who want to use etcd v3 API via grpc, please use:
https://pypi.python.org/pypi/etcd3 (works only for non-eventlet services)

Those that depend on eventlet, please use the etcd3 v3alpha HTTP API using:
https://pypi.python.org/pypi/etcd3gw

If you use tooz, there are 2 driver choices for you:
https://github.com/openstack/tooz/blob/master/setup.cfg#L29
https://github.com/openstack/tooz/blob/master/setup.cfg#L30

If you use oslo.cache, there is a driver for you:
https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33

Devstack installs etcd3 by default and points cinder to it:
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356

Review in progress for keystone to use etcd3 for caching:
https://review.openstack.org/#/c/469621/

Doug is working on proposal(s) for oslo.config to store some
configuration in etcd3:
https://review.openstack.org/#/c/454897/

So, feel free to turn on / test with etcd3 and report issues.

Thanks,
Dims

--
Davanum Srinivas :: https://twitter.com/dims

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

Davanum Srinivas
One clarification: Since https://pypi.python.org/pypi/etcd3gw just
uses the HTTP API (/v3alpha) it will work under both eventlet and
non-eventlet environments.

Thanks,
Dims


On Wed, Jun 7, 2017 at 6:47 AM, Davanum Srinivas <[hidden email]> wrote:

> Team,
>
> Here's the update to the base services resolution from the TC:
> https://governance.openstack.org/tc/reference/base-services.html
>
> First request is to Distros, Packagers, Deployers, anyone who
> installs/configures OpenStack:
> Please make sure you have latest etcd 3.x available in your
> environment for Services to use, Fedora already does, we need help in
> making sure all distros and architectures are covered.
>
> Any project who want to use etcd v3 API via grpc, please use:
> https://pypi.python.org/pypi/etcd3 (works only for non-eventlet services)
>
> Those that depend on eventlet, please use the etcd3 v3alpha HTTP API using:
> https://pypi.python.org/pypi/etcd3gw
>
> If you use tooz, there are 2 driver choices for you:
> https://github.com/openstack/tooz/blob/master/setup.cfg#L29
> https://github.com/openstack/tooz/blob/master/setup.cfg#L30
>
> If you use oslo.cache, there is a driver for you:
> https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33
>
> Devstack installs etcd3 by default and points cinder to it:
> http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3
> http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356
>
> Review in progress for keystone to use etcd3 for caching:
> https://review.openstack.org/#/c/469621/
>
> Doug is working on proposal(s) for oslo.config to store some
> configuration in etcd3:
> https://review.openstack.org/#/c/454897/
>
> So, feel free to turn on / test with etcd3 and report issues.
>
> Thanks,
> Dims
>
> --
> Davanum Srinivas :: https://twitter.com/dims



--
Davanum Srinivas :: https://twitter.com/dims

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

Joshua Harlow-3
So just out of curiosity, but do people really even know what etcd is
good for? I am thinking that there should be some guidance from folks in
the community as to where etcd should be used and where it shouldn't
(otherwise we just all end up in a mess).

Perhaps a good idea to actually give examples of how it should be used,
how it shouldn't be used, what it offers, what it doesn't... Or at least
provide links for people to read up on this.

Thoughts?

Davanum Srinivas wrote:

> One clarification: Since https://pypi.python.org/pypi/etcd3gw just
> uses the HTTP API (/v3alpha) it will work under both eventlet and
> non-eventlet environments.
>
> Thanks,
> Dims
>
>
> On Wed, Jun 7, 2017 at 6:47 AM, Davanum Srinivas<[hidden email]>  wrote:
>> Team,
>>
>> Here's the update to the base services resolution from the TC:
>> https://governance.openstack.org/tc/reference/base-services.html
>>
>> First request is to Distros, Packagers, Deployers, anyone who
>> installs/configures OpenStack:
>> Please make sure you have latest etcd 3.x available in your
>> environment for Services to use, Fedora already does, we need help in
>> making sure all distros and architectures are covered.
>>
>> Any project who want to use etcd v3 API via grpc, please use:
>> https://pypi.python.org/pypi/etcd3 (works only for non-eventlet services)
>>
>> Those that depend on eventlet, please use the etcd3 v3alpha HTTP API using:
>> https://pypi.python.org/pypi/etcd3gw
>>
>> If you use tooz, there are 2 driver choices for you:
>> https://github.com/openstack/tooz/blob/master/setup.cfg#L29
>> https://github.com/openstack/tooz/blob/master/setup.cfg#L30
>>
>> If you use oslo.cache, there is a driver for you:
>> https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33
>>
>> Devstack installs etcd3 by default and points cinder to it:
>> http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3
>> http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356
>>
>> Review in progress for keystone to use etcd3 for caching:
>> https://review.openstack.org/#/c/469621/
>>
>> Doug is working on proposal(s) for oslo.config to store some
>> configuration in etcd3:
>> https://review.openstack.org/#/c/454897/
>>
>> So, feel free to turn on / test with etcd3 and report issues.
>>
>> Thanks,
>> Dims
>>
>> --
>> Davanum Srinivas :: https://twitter.com/dims
>
>
>

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

Thierry Carrez
Joshua Harlow wrote:

> So just out of curiosity, but do people really even know what etcd is
> good for? I am thinking that there should be some guidance from folks in
> the community as to where etcd should be used and where it shouldn't
> (otherwise we just all end up in a mess).
>
> Perhaps a good idea to actually give examples of how it should be used,
> how it shouldn't be used, what it offers, what it doesn't... Or at least
> provide links for people to read up on this.
>
> Thoughts?

I think that's a great idea. More generally, we should document base
services, the benefits of each tech, and what data should be in
database, what data should be in a message queue, and what data should
be in etcd. It feels like the Project Team Guide could be a place where
such information could live...

--
Thierry Carrez (ttx)

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

Davanum Srinivas
In reply to this post by Joshua Harlow-3
Josh,

Of the initial targets, we have covered the use of tooz and oslo.cache
with etcd already. One thing that remains is Jay's os-lively PoC [1].

So +1 to more knowledgeable folks chiming in with stuff that can be
done and other scenarios. I'll let nature take its course here :)

Thanks,
Dims

[1] https://github.com/jaypipes/os-lively/

On Thu, Jun 8, 2017 at 12:47 AM, Joshua Harlow <[hidden email]> wrote:

> So just out of curiosity, but do people really even know what etcd is good
> for? I am thinking that there should be some guidance from folks in the
> community as to where etcd should be used and where it shouldn't (otherwise
> we just all end up in a mess).
>
> Perhaps a good idea to actually give examples of how it should be used, how
> it shouldn't be used, what it offers, what it doesn't... Or at least provide
> links for people to read up on this.
>
> Thoughts?
>
>
> Davanum Srinivas wrote:
>>
>> One clarification: Since https://pypi.python.org/pypi/etcd3gw just
>> uses the HTTP API (/v3alpha) it will work under both eventlet and
>> non-eventlet environments.
>>
>> Thanks,
>> Dims
>>
>>
>> On Wed, Jun 7, 2017 at 6:47 AM, Davanum Srinivas<[hidden email]>
>> wrote:
>>>
>>> Team,
>>>
>>> Here's the update to the base services resolution from the TC:
>>> https://governance.openstack.org/tc/reference/base-services.html
>>>
>>> First request is to Distros, Packagers, Deployers, anyone who
>>> installs/configures OpenStack:
>>> Please make sure you have latest etcd 3.x available in your
>>> environment for Services to use, Fedora already does, we need help in
>>> making sure all distros and architectures are covered.
>>>
>>> Any project who want to use etcd v3 API via grpc, please use:
>>> https://pypi.python.org/pypi/etcd3 (works only for non-eventlet services)
>>>
>>> Those that depend on eventlet, please use the etcd3 v3alpha HTTP API
>>> using:
>>> https://pypi.python.org/pypi/etcd3gw
>>>
>>> If you use tooz, there are 2 driver choices for you:
>>> https://github.com/openstack/tooz/blob/master/setup.cfg#L29
>>> https://github.com/openstack/tooz/blob/master/setup.cfg#L30
>>>
>>> If you use oslo.cache, there is a driver for you:
>>> https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33
>>>
>>> Devstack installs etcd3 by default and points cinder to it:
>>> http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3
>>> http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356
>>>
>>> Review in progress for keystone to use etcd3 for caching:
>>> https://review.openstack.org/#/c/469621/
>>>
>>> Doug is working on proposal(s) for oslo.config to store some
>>> configuration in etcd3:
>>> https://review.openstack.org/#/c/454897/
>>>
>>> So, feel free to turn on / test with etcd3 and report issues.
>>>
>>> Thanks,
>>> Dims
>>>
>>> --
>>> Davanum Srinivas :: https://twitter.com/dims
>>
>>
>>
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Davanum Srinivas :: https://twitter.com/dims

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

Michael Bayer
In reply to this post by Joshua Harlow-3


On 06/08/2017 12:47 AM, Joshua Harlow wrote:
> So just out of curiosity, but do people really even know what etcd is
> good for? I am thinking that there should be some guidance from folks in
> the community as to where etcd should be used and where it shouldn't
> (otherwise we just all end up in a mess).

So far I've seen a proposal of etcd3 as a replacement for memcached in
keystone, and a new dogpile connector was added to oslo.cache to handle
referring to etcd3 as a cache backend.  This is a really simplistic /
minimal kind of use case for a key-store.

But, keeping in mind I don't know anything about etcd3 other than "it's
another key-store", it's the only database used by Kubernetes as a
whole, which suggests it's doing a better job than Redis in terms of
"durable".   So I wouldn't be surprised if new / existing openstack
applications express some gravitational pull towards using it as their
own datastore as well.    I'll be trying to hang onto the etcd3 track as
much as possible so that if/when that happens I still have a job :).



>
> Perhaps a good idea to actually give examples of how it should be used,
> how it shouldn't be used, what it offers, what it doesn't... Or at least
> provide links for people to read up on this.
>
> Thoughts?
>
> Davanum Srinivas wrote:
>> One clarification: Since https://pypi.python.org/pypi/etcd3gw just
>> uses the HTTP API (/v3alpha) it will work under both eventlet and
>> non-eventlet environments.
>>
>> Thanks,
>> Dims
>>
>>
>> On Wed, Jun 7, 2017 at 6:47 AM, Davanum Srinivas<[hidden email]>  
>> wrote:
>>> Team,
>>>
>>> Here's the update to the base services resolution from the TC:
>>> https://governance.openstack.org/tc/reference/base-services.html
>>>
>>> First request is to Distros, Packagers, Deployers, anyone who
>>> installs/configures OpenStack:
>>> Please make sure you have latest etcd 3.x available in your
>>> environment for Services to use, Fedora already does, we need help in
>>> making sure all distros and architectures are covered.
>>>
>>> Any project who want to use etcd v3 API via grpc, please use:
>>> https://pypi.python.org/pypi/etcd3 (works only for non-eventlet
>>> services)
>>>
>>> Those that depend on eventlet, please use the etcd3 v3alpha HTTP API
>>> using:
>>> https://pypi.python.org/pypi/etcd3gw
>>>
>>> If you use tooz, there are 2 driver choices for you:
>>> https://github.com/openstack/tooz/blob/master/setup.cfg#L29
>>> https://github.com/openstack/tooz/blob/master/setup.cfg#L30
>>>
>>> If you use oslo.cache, there is a driver for you:
>>> https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33
>>>
>>> Devstack installs etcd3 by default and points cinder to it:
>>> http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3
>>> http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356 
>>>
>>>
>>> Review in progress for keystone to use etcd3 for caching:
>>> https://review.openstack.org/#/c/469621/
>>>
>>> Doug is working on proposal(s) for oslo.config to store some
>>> configuration in etcd3:
>>> https://review.openstack.org/#/c/454897/
>>>
>>> So, feel free to turn on / test with etcd3 and report issues.
>>>
>>> Thanks,
>>> Dims
>>>
>>> --
>>> Davanum Srinivas :: https://twitter.com/dims
>>
>>
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

Lance Bragstad-2
After digging into etcd a bit, one place this might be help deployer experience would be the handling of fernet keys for token encryption in keystone. Currently, all keys used to encrypt and decrypt tokens are kept on disk for each keystone node in the deployment. While simple, it requires operators to perform rotation on a single node and then push, or sync, the new key set to the rest of the nodes. This must be done in lock step in order to prevent early token invalidation and inconsistent token responses.

An alternative would be to keep the keys in etcd and make the fernet bits pluggable so that it's possible to read keys from disk or etcd (pending configuration). The advantage would be that operators could initiate key rotations from any keystone node in the deployment (or using etcd directly) and not have to worry about distributing the new key set. Since etcd associates metadata to the key-value pairs, we might be able to simplify the rotation strategy as well.

On Thu, Jun 8, 2017 at 11:37 AM, Mike Bayer <[hidden email]> wrote:


On 06/08/2017 12:47 AM, Joshua Harlow wrote:
So just out of curiosity, but do people really even know what etcd is good for? I am thinking that there should be some guidance from folks in the community as to where etcd should be used and where it shouldn't (otherwise we just all end up in a mess).

So far I've seen a proposal of etcd3 as a replacement for memcached in keystone, and a new dogpile connector was added to oslo.cache to handle referring to etcd3 as a cache backend.  This is a really simplistic / minimal kind of use case for a key-store.

But, keeping in mind I don't know anything about etcd3 other than "it's another key-store", it's the only database used by Kubernetes as a whole, which suggests it's doing a better job than Redis in terms of "durable".   So I wouldn't be surprised if new / existing openstack applications express some gravitational pull towards using it as their own datastore as well.    I'll be trying to hang onto the etcd3 track as much as possible so that if/when that happens I still have a job :).





Perhaps a good idea to actually give examples of how it should be used, how it shouldn't be used, what it offers, what it doesn't... Or at least provide links for people to read up on this.

Thoughts?

Davanum Srinivas wrote:
One clarification: Since https://pypi.python.org/pypi/etcd3gw just
uses the HTTP API (/v3alpha) it will work under both eventlet and
non-eventlet environments.

Thanks,
Dims


On Wed, Jun 7, 2017 at 6:47 AM, Davanum Srinivas<[hidden email]>  wrote:
Team,

Here's the update to the base services resolution from the TC:
https://governance.openstack.org/tc/reference/base-services.html

First request is to Distros, Packagers, Deployers, anyone who
installs/configures OpenStack:
Please make sure you have latest etcd 3.x available in your
environment for Services to use, Fedora already does, we need help in
making sure all distros and architectures are covered.

Any project who want to use etcd v3 API via grpc, please use:
https://pypi.python.org/pypi/etcd3 (works only for non-eventlet services)

Those that depend on eventlet, please use the etcd3 v3alpha HTTP API using:
https://pypi.python.org/pypi/etcd3gw

If you use tooz, there are 2 driver choices for you:
https://github.com/openstack/tooz/blob/master/setup.cfg#L29
https://github.com/openstack/tooz/blob/master/setup.cfg#L30

If you use oslo.cache, there is a driver for you:
https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33

Devstack installs etcd3 by default and points cinder to it:
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356

Review in progress for keystone to use etcd3 for caching:
https://review.openstack.org/#/c/469621/

Doug is working on proposal(s) for oslo.config to store some
configuration in etcd3:
https://review.openstack.org/#/c/454897/

So, feel free to turn on / test with etcd3 and report issues.

Thanks,
Dims

--
Davanum Srinivas :: https://twitter.com/dims




__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@...enstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@...enstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

Jay Pipes
In reply to this post by Michael Bayer
On 06/08/2017 12:37 PM, Mike Bayer wrote:

> On 06/08/2017 12:47 AM, Joshua Harlow wrote:
>> So just out of curiosity, but do people really even know what etcd is
>> good for? I am thinking that there should be some guidance from folks
>> in the community as to where etcd should be used and where it
>> shouldn't (otherwise we just all end up in a mess).
>
> So far I've seen a proposal of etcd3 as a replacement for memcached in
> keystone, and a new dogpile connector was added to oslo.cache to handle
> referring to etcd3 as a cache backend.  This is a really simplistic /
> minimal kind of use case for a key-store.
>
> But, keeping in mind I don't know anything about etcd3 other than "it's
> another key-store", it's the only database used by Kubernetes as a
> whole, which suggests it's doing a better job than Redis in terms of
> "durable".   So I wouldn't be surprised if new / existing openstack
> applications express some gravitational pull towards using it as their
> own datastore as well.    I'll be trying to hang onto the etcd3 track as
> much as possible so that if/when that happens I still have a job :).

Please see
http://lists.openstack.org/pipermail/openstack-dev/2017-May/117316.html 
:) I think we'll still have jobs working with those silly old RDBMS in a
couple years.

Now, in three years... who knows?

Best,
-jay

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

Fox, Kevin M
In reply to this post by Lance Bragstad-2
So, one thing to remember, I don't think etcd has an authz mechanism yet.

You usually want your fernet keys to be accessible by just the keystone nodes and no others.

This might require a etcd cluster just for keystone fernet tokens, which might work great. but is an operator overhead to install/maintain.


From: Lance Bragstad [[hidden email]]
Sent: Thursday, June 08, 2017 10:34 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [all] etcd3 as base service - update

After digging into etcd a bit, one place this might be help deployer experience would be the handling of fernet keys for token encryption in keystone. Currently, all keys used to encrypt and decrypt tokens are kept on disk for each keystone node in the deployment. While simple, it requires operators to perform rotation on a single node and then push, or sync, the new key set to the rest of the nodes. This must be done in lock step in order to prevent early token invalidation and inconsistent token responses.

An alternative would be to keep the keys in etcd and make the fernet bits pluggable so that it's possible to read keys from disk or etcd (pending configuration). The advantage would be that operators could initiate key rotations from any keystone node in the deployment (or using etcd directly) and not have to worry about distributing the new key set. Since etcd associates metadata to the key-value pairs, we might be able to simplify the rotation strategy as well.

On Thu, Jun 8, 2017 at 11:37 AM, Mike Bayer <[hidden email]> wrote:


On 06/08/2017 12:47 AM, Joshua Harlow wrote:
So just out of curiosity, but do people really even know what etcd is good for? I am thinking that there should be some guidance from folks in the community as to where etcd should be used and where it shouldn't (otherwise we just all end up in a mess).

So far I've seen a proposal of etcd3 as a replacement for memcached in keystone, and a new dogpile connector was added to oslo.cache to handle referring to etcd3 as a cache backend.  This is a really simplistic / minimal kind of use case for a key-store.

But, keeping in mind I don't know anything about etcd3 other than "it's another key-store", it's the only database used by Kubernetes as a whole, which suggests it's doing a better job than Redis in terms of "durable".   So I wouldn't be surprised if new / existing openstack applications express some gravitational pull towards using it as their own datastore as well.    I'll be trying to hang onto the etcd3 track as much as possible so that if/when that happens I still have a job :).





Perhaps a good idea to actually give examples of how it should be used, how it shouldn't be used, what it offers, what it doesn't... Or at least provide links for people to read up on this.

Thoughts?

Davanum Srinivas wrote:
One clarification: Since https://pypi.python.org/pypi/etcd3gw just
uses the HTTP API (/v3alpha) it will work under both eventlet and
non-eventlet environments.

Thanks,
Dims


On Wed, Jun 7, 2017 at 6:47 AM, Davanum Srinivas<[hidden email]>  wrote:
Team,

Here's the update to the base services resolution from the TC:
https://governance.openstack.org/tc/reference/base-services.html

First request is to Distros, Packagers, Deployers, anyone who
installs/configures OpenStack:
Please make sure you have latest etcd 3.x available in your
environment for Services to use, Fedora already does, we need help in
making sure all distros and architectures are covered.

Any project who want to use etcd v3 API via grpc, please use:
https://pypi.python.org/pypi/etcd3 (works only for non-eventlet services)

Those that depend on eventlet, please use the etcd3 v3alpha HTTP API using:
https://pypi.python.org/pypi/etcd3gw

If you use tooz, there are 2 driver choices for you:
https://github.com/openstack/tooz/blob/master/setup.cfg#L29
https://github.com/openstack/tooz/blob/master/setup.cfg#L30

If you use oslo.cache, there is a driver for you:
https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33

Devstack installs etcd3 by default and points cinder to it:
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356

Review in progress for keystone to use etcd3 for caching:
https://review.openstack.org/#/c/469621/

Doug is working on proposal(s) for oslo.config to store some
configuration in etcd3:
https://review.openstack.org/#/c/454897/

So, feel free to turn on / test with etcd3 and report issues.

Thanks,
Dims

--
Davanum Srinivas :: https://twitter.com/dims




__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@...enstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@...enstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

Drew Fisher
In reply to this post by Davanum Srinivas


On 6/7/17 4:47 AM, Davanum Srinivas wrote:

> Team,
>
> Here's the update to the base services resolution from the TC:
> https://governance.openstack.org/tc/reference/base-services.html
>
> First request is to Distros, Packagers, Deployers, anyone who
> installs/configures OpenStack:
> Please make sure you have latest etcd 3.x available in your
> environment for Services to use, Fedora already does, we need help in
> making sure all distros and architectures are covered.

As a Solaris OpenStack dev, I have a questions about this change.

If Solaris were to *only* run the nova-compute service, and leave the
rest of the OpenStack services to Linux, is etcd 3.x required on the
compute node for Pike+ ?

Thanks!

-Drew



__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

jimi olugboyega
In reply to this post by Davanum Srinivas
Hello all,

I am wondering how I can unsubscribe from this mailing list.

Regards,
Olujimi Olugboyega.

On Wed, Jun 7, 2017 at 3:47 AM, Davanum Srinivas <[hidden email]> wrote:
Team,

Here's the update to the base services resolution from the TC:
https://governance.openstack.org/tc/reference/base-services.html

First request is to Distros, Packagers, Deployers, anyone who
installs/configures OpenStack:
Please make sure you have latest etcd 3.x available in your
environment for Services to use, Fedora already does, we need help in
making sure all distros and architectures are covered.

Any project who want to use etcd v3 API via grpc, please use:
https://pypi.python.org/pypi/etcd3 (works only for non-eventlet services)

Those that depend on eventlet, please use the etcd3 v3alpha HTTP API using:
https://pypi.python.org/pypi/etcd3gw

If you use tooz, there are 2 driver choices for you:
https://github.com/openstack/tooz/blob/master/setup.cfg#L29
https://github.com/openstack/tooz/blob/master/setup.cfg#L30

If you use oslo.cache, there is a driver for you:
https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33

Devstack installs etcd3 by default and points cinder to it:
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356

Review in progress for keystone to use etcd3 for caching:
https://review.openstack.org/#/c/469621/

Doug is working on proposal(s) for oslo.config to store some
configuration in etcd3:
https://review.openstack.org/#/c/454897/

So, feel free to turn on / test with etcd3 and report issues.

Thanks,
Dims

--
Davanum Srinivas :: https://twitter.com/dims

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

Emilien Macchi-4
In reply to this post by Lance Bragstad-2
On Thu, Jun 8, 2017 at 7:34 PM, Lance Bragstad <[hidden email]> wrote:
> After digging into etcd a bit, one place this might be help deployer
> experience would be the handling of fernet keys for token encryption in
> keystone. Currently, all keys used to encrypt and decrypt tokens are kept on
> disk for each keystone node in the deployment. While simple, it requires
> operators to perform rotation on a single node and then push, or sync, the
> new key set to the rest of the nodes. This must be done in lock step in
> order to prevent early token invalidation and inconsistent token responses.

This is what we discussed a few months ago :-)

http://lists.openstack.org/pipermail/openstack-dev/2017-March/113943.html

I'm glad it's coming back ;-)

> An alternative would be to keep the keys in etcd and make the fernet bits
> pluggable so that it's possible to read keys from disk or etcd (pending
> configuration). The advantage would be that operators could initiate key
> rotations from any keystone node in the deployment (or using etcd directly)
> and not have to worry about distributing the new key set. Since etcd
> associates metadata to the key-value pairs, we might be able to simplify the
> rotation strategy as well.
>
> On Thu, Jun 8, 2017 at 11:37 AM, Mike Bayer <[hidden email]> wrote:
>>
>>
>>
>> On 06/08/2017 12:47 AM, Joshua Harlow wrote:
>>>
>>> So just out of curiosity, but do people really even know what etcd is
>>> good for? I am thinking that there should be some guidance from folks in the
>>> community as to where etcd should be used and where it shouldn't (otherwise
>>> we just all end up in a mess).
>>
>>
>> So far I've seen a proposal of etcd3 as a replacement for memcached in
>> keystone, and a new dogpile connector was added to oslo.cache to handle
>> referring to etcd3 as a cache backend.  This is a really simplistic /
>> minimal kind of use case for a key-store.
>>
>> But, keeping in mind I don't know anything about etcd3 other than "it's
>> another key-store", it's the only database used by Kubernetes as a whole,
>> which suggests it's doing a better job than Redis in terms of "durable".
>> So I wouldn't be surprised if new / existing openstack applications express
>> some gravitational pull towards using it as their own datastore as well.
>> I'll be trying to hang onto the etcd3 track as much as possible so that
>> if/when that happens I still have a job :).
>>
>>
>>
>>
>>>
>>> Perhaps a good idea to actually give examples of how it should be used,
>>> how it shouldn't be used, what it offers, what it doesn't... Or at least
>>> provide links for people to read up on this.
>>>
>>> Thoughts?
>>>
>>> Davanum Srinivas wrote:
>>>>
>>>> One clarification: Since https://pypi.python.org/pypi/etcd3gw just
>>>> uses the HTTP API (/v3alpha) it will work under both eventlet and
>>>> non-eventlet environments.
>>>>
>>>> Thanks,
>>>> Dims
>>>>
>>>>
>>>> On Wed, Jun 7, 2017 at 6:47 AM, Davanum Srinivas<[hidden email]>
>>>> wrote:
>>>>>
>>>>> Team,
>>>>>
>>>>> Here's the update to the base services resolution from the TC:
>>>>> https://governance.openstack.org/tc/reference/base-services.html
>>>>>
>>>>> First request is to Distros, Packagers, Deployers, anyone who
>>>>> installs/configures OpenStack:
>>>>> Please make sure you have latest etcd 3.x available in your
>>>>> environment for Services to use, Fedora already does, we need help in
>>>>> making sure all distros and architectures are covered.
>>>>>
>>>>> Any project who want to use etcd v3 API via grpc, please use:
>>>>> https://pypi.python.org/pypi/etcd3 (works only for non-eventlet
>>>>> services)
>>>>>
>>>>> Those that depend on eventlet, please use the etcd3 v3alpha HTTP API
>>>>> using:
>>>>> https://pypi.python.org/pypi/etcd3gw
>>>>>
>>>>> If you use tooz, there are 2 driver choices for you:
>>>>> https://github.com/openstack/tooz/blob/master/setup.cfg#L29
>>>>> https://github.com/openstack/tooz/blob/master/setup.cfg#L30
>>>>>
>>>>> If you use oslo.cache, there is a driver for you:
>>>>> https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33
>>>>>
>>>>> Devstack installs etcd3 by default and points cinder to it:
>>>>> http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3
>>>>>
>>>>> http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356
>>>>>
>>>>> Review in progress for keystone to use etcd3 for caching:
>>>>> https://review.openstack.org/#/c/469621/
>>>>>
>>>>> Doug is working on proposal(s) for oslo.config to store some
>>>>> configuration in etcd3:
>>>>> https://review.openstack.org/#/c/454897/
>>>>>
>>>>> So, feel free to turn on / test with etcd3 and report issues.
>>>>>
>>>>> Thanks,
>>>>> Dims
>>>>>
>>>>> --
>>>>> Davanum Srinivas :: https://twitter.com/dims
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> [hidden email]?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: [hidden email]?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



--
Emilien Macchi

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

Julien Danjou
In reply to this post by Michael Bayer
On Thu, Jun 08 2017, Mike Bayer wrote:

> So far I've seen a proposal of etcd3 as a replacement for memcached in
> keystone, and a new dogpile connector was added to oslo.cache to handle
> referring to etcd3 as a cache backend.  This is a really simplistic / minimal
> kind of use case for a key-store.

etcd3 is not meant to be a cache. Synchronizing caching value using the
Raft protocol sounds a bit overkill. A cluster of memcached would be
probably of a better use.

> But, keeping in mind I don't know anything about etcd3 other than "it's another
> key-store", it's the only database used by Kubernetes as a whole, which
> suggests it's doing a better job than Redis in terms of "durable".

Not sure about that. And Redis has much more data structure than etcd,
that is can be faster/more efficient than etcd. But it does not have
Raft and a synchronisation protocol. Its clustering is rather poor in
comparison of etcd.

> So I wouldn't be surprised if new / existing openstack applications
> express some gravitational pull towards using it as their own
> datastore as well. I'll be trying to hang onto the etcd3 track as much
> as possible so that if/when that happens I still have a job :).

Sounds like a recipe for disaster. :)

--
Julien Danjou
/* Free Software hacker
   https://julien.danjou.info */

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

signature.asc (815 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

Fox, Kevin M
In reply to this post by Drew Fisher
hmm... a very interesting question....

I would think control plane only.

Thanks,
Kevin
________________________________________
From: Drew Fisher [[hidden email]]
Sent: Thursday, June 08, 2017 1:07 PM
To: [hidden email]
Subject: Re: [openstack-dev] [all] etcd3 as base service - update

On 6/7/17 4:47 AM, Davanum Srinivas wrote:

> Team,
>
> Here's the update to the base services resolution from the TC:
> https://governance.openstack.org/tc/reference/base-services.html
>
> First request is to Distros, Packagers, Deployers, anyone who
> installs/configures OpenStack:
> Please make sure you have latest etcd 3.x available in your
> environment for Services to use, Fedora already does, we need help in
> making sure all distros and architectures are covered.

As a Solaris OpenStack dev, I have a questions about this change.

If Solaris were to *only* run the nova-compute service, and leave the
rest of the OpenStack services to Linux, is etcd 3.x required on the
compute node for Pike+ ?

Thanks!

-Drew



__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

Fox, Kevin M
In reply to this post by jimi olugboyega
See the footer at the bottom of this email.

From: jimi olugboyega [[hidden email]]
Sent: Thursday, June 08, 2017 1:19 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [all] etcd3 as base service - update

Hello all,

I am wondering how I can unsubscribe from this mailing list.

Regards,
Olujimi Olugboyega.

On Wed, Jun 7, 2017 at 3:47 AM, Davanum Srinivas <[hidden email]> wrote:
Team,

Here's the update to the base services resolution from the TC:
https://governance.openstack.org/tc/reference/base-services.html

First request is to Distros, Packagers, Deployers, anyone who
installs/configures OpenStack:
Please make sure you have latest etcd 3.x available in your
environment for Services to use, Fedora already does, we need help in
making sure all distros and architectures are covered.

Any project who want to use etcd v3 API via grpc, please use:
https://pypi.python.org/pypi/etcd3 (works only for non-eventlet services)

Those that depend on eventlet, please use the etcd3 v3alpha HTTP API using:
https://pypi.python.org/pypi/etcd3gw

If you use tooz, there are 2 driver choices for you:
https://github.com/openstack/tooz/blob/master/setup.cfg#L29
https://github.com/openstack/tooz/blob/master/setup.cfg#L30

If you use oslo.cache, there is a driver for you:
https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33

Devstack installs etcd3 by default and points cinder to it:
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356

Review in progress for keystone to use etcd3 for caching:
https://review.openstack.org/#/c/469621/

Doug is working on proposal(s) for oslo.config to store some
configuration in etcd3:
https://review.openstack.org/#/c/454897/

So, feel free to turn on / test with etcd3 and report issues.

Thanks,
Dims

--
Davanum Srinivas :: https://twitter.com/dims

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

Davanum Srinivas
In reply to this post by Drew Fisher
On Thu, Jun 8, 2017 at 4:07 PM, Drew Fisher <[hidden email]> wrote:

>
>
> On 6/7/17 4:47 AM, Davanum Srinivas wrote:
>> Team,
>>
>> Here's the update to the base services resolution from the TC:
>> https://governance.openstack.org/tc/reference/base-services.html
>>
>> First request is to Distros, Packagers, Deployers, anyone who
>> installs/configures OpenStack:
>> Please make sure you have latest etcd 3.x available in your
>> environment for Services to use, Fedora already does, we need help in
>> making sure all distros and architectures are covered.
>
> As a Solaris OpenStack dev, I have a questions about this change.
>
> If Solaris were to *only* run the nova-compute service, and leave the
> rest of the OpenStack services to Linux, is etcd 3.x required on the
> compute node for Pike+ ?

Yes, this should be fine.

> Thanks!
>
> -Drew
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Davanum Srinivas :: https://twitter.com/dims

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

Lance Bragstad-2
In reply to this post by Emilien Macchi-4


On Thu, Jun 8, 2017 at 3:21 PM, Emilien Macchi <[hidden email]> wrote:
On Thu, Jun 8, 2017 at 7:34 PM, Lance Bragstad <[hidden email]> wrote:
> After digging into etcd a bit, one place this might be help deployer
> experience would be the handling of fernet keys for token encryption in
> keystone. Currently, all keys used to encrypt and decrypt tokens are kept on
> disk for each keystone node in the deployment. While simple, it requires
> operators to perform rotation on a single node and then push, or sync, the
> new key set to the rest of the nodes. This must be done in lock step in
> order to prevent early token invalidation and inconsistent token responses.

This is what we discussed a few months ago :-)

http://lists.openstack.org/pipermail/openstack-dev/2017-March/113943.html

I'm glad it's coming back ;-)

Yep! I've proposed a pretty basic spec to backlog [0] in an effort to capture the discussion. I've also noted the point Kevin brought up about authorization in etcd (thanks, Kevin!)

If someone feels compelled to take that and run with it, they are more than welcome to.



> An alternative would be to keep the keys in etcd and make the fernet bits
> pluggable so that it's possible to read keys from disk or etcd (pending
> configuration). The advantage would be that operators could initiate key
> rotations from any keystone node in the deployment (or using etcd directly)
> and not have to worry about distributing the new key set. Since etcd
> associates metadata to the key-value pairs, we might be able to simplify the
> rotation strategy as well.
>
> On Thu, Jun 8, 2017 at 11:37 AM, Mike Bayer <[hidden email]> wrote:
>>
>>
>>
>> On 06/08/2017 12:47 AM, Joshua Harlow wrote:
>>>
>>> So just out of curiosity, but do people really even know what etcd is
>>> good for? I am thinking that there should be some guidance from folks in the
>>> community as to where etcd should be used and where it shouldn't (otherwise
>>> we just all end up in a mess).
>>
>>
>> So far I've seen a proposal of etcd3 as a replacement for memcached in
>> keystone, and a new dogpile connector was added to oslo.cache to handle
>> referring to etcd3 as a cache backend.  This is a really simplistic /
>> minimal kind of use case for a key-store.
>>
>> But, keeping in mind I don't know anything about etcd3 other than "it's
>> another key-store", it's the only database used by Kubernetes as a whole,
>> which suggests it's doing a better job than Redis in terms of "durable".
>> So I wouldn't be surprised if new / existing openstack applications express
>> some gravitational pull towards using it as their own datastore as well.
>> I'll be trying to hang onto the etcd3 track as much as possible so that
>> if/when that happens I still have a job :).
>>
>>
>>
>>
>>>
>>> Perhaps a good idea to actually give examples of how it should be used,
>>> how it shouldn't be used, what it offers, what it doesn't... Or at least
>>> provide links for people to read up on this.
>>>
>>> Thoughts?
>>>
>>> Davanum Srinivas wrote:
>>>>
>>>> One clarification: Since https://pypi.python.org/pypi/etcd3gw just
>>>> uses the HTTP API (/v3alpha) it will work under both eventlet and
>>>> non-eventlet environments.
>>>>
>>>> Thanks,
>>>> Dims
>>>>
>>>>
>>>> On Wed, Jun 7, 2017 at 6:47 AM, Davanum Srinivas<[hidden email]>
>>>> wrote:
>>>>>
>>>>> Team,
>>>>>
>>>>> Here's the update to the base services resolution from the TC:
>>>>> https://governance.openstack.org/tc/reference/base-services.html
>>>>>
>>>>> First request is to Distros, Packagers, Deployers, anyone who
>>>>> installs/configures OpenStack:
>>>>> Please make sure you have latest etcd 3.x available in your
>>>>> environment for Services to use, Fedora already does, we need help in
>>>>> making sure all distros and architectures are covered.
>>>>>
>>>>> Any project who want to use etcd v3 API via grpc, please use:
>>>>> https://pypi.python.org/pypi/etcd3 (works only for non-eventlet
>>>>> services)
>>>>>
>>>>> Those that depend on eventlet, please use the etcd3 v3alpha HTTP API
>>>>> using:
>>>>> https://pypi.python.org/pypi/etcd3gw
>>>>>
>>>>> If you use tooz, there are 2 driver choices for you:
>>>>> https://github.com/openstack/tooz/blob/master/setup.cfg#L29
>>>>> https://github.com/openstack/tooz/blob/master/setup.cfg#L30
>>>>>
>>>>> If you use oslo.cache, there is a driver for you:
>>>>> https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33
>>>>>
>>>>> Devstack installs etcd3 by default and points cinder to it:
>>>>> http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3
>>>>>
>>>>> http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356
>>>>>
>>>>> Review in progress for keystone to use etcd3 for caching:
>>>>> https://review.openstack.org/#/c/469621/
>>>>>
>>>>> Doug is working on proposal(s) for oslo.config to store some
>>>>> configuration in etcd3:
>>>>> https://review.openstack.org/#/c/454897/
>>>>>
>>>>> So, feel free to turn on / test with etcd3 and report issues.
>>>>>
>>>>> Thanks,
>>>>> Dims
>>>>>
>>>>> --
>>>>> Davanum Srinivas :: https://twitter.com/dims
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



--
Emilien Macchi

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

Joshua Harlow-3
In reply to this post by Julien Danjou
Julien Danjou wrote:

> On Thu, Jun 08 2017, Mike Bayer wrote:
>
>> So far I've seen a proposal of etcd3 as a replacement for memcached in
>> keystone, and a new dogpile connector was added to oslo.cache to handle
>> referring to etcd3 as a cache backend.  This is a really simplistic / minimal
>> kind of use case for a key-store.
>
> etcd3 is not meant to be a cache. Synchronizing caching value using the
> Raft protocol sounds a bit overkill. A cluster of memcached would be
> probably of a better use.

Agreed from me,

My thinking is that people should look over https://raft.github.io/ or
http://thesecretlivesofdata.com/raft/ (or both or others...)

At least read how it sort of works, before trying to put it everywhere
(the same can and should be said for any new service), because its not a
solution for all the things.

The other big thing to know is how writes happen in this kind of system,
they all go through a single node (the leader, who sends the same data
to followers and waits for a certain number of followers to respond
before committing)....

Anyways, with great power comes great responsibility...

IMHO just be careful and understand the technology before using it for
things it may not really be good for. Oh ya and perhaps someone will
want to finally take more advantage of
https://docs.openstack.org/developer/taskflow/jobs.html#overview (which
uses the same concepts etcd exposes to make highly available workflows
that can survive node failure).

>
>> But, keeping in mind I don't know anything about etcd3 other than "it's another
>> key-store", it's the only database used by Kubernetes as a whole, which
>> suggests it's doing a better job than Redis in terms of "durable".
>
> Not sure about that. And Redis has much more data structure than etcd,
> that is can be faster/more efficient than etcd. But it does not have
> Raft and a synchronisation protocol. Its clustering is rather poor in
> comparison of etcd.
>
>> So I wouldn't be surprised if new / existing openstack applications
>> express some gravitational pull towards using it as their own
>> datastore as well. I'll be trying to hang onto the etcd3 track as much
>> as possible so that if/when that happens I still have a job :).
>
> Sounds like a recipe for disaster. :)
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

gordon chung-2


On 09/06/17 12:37 AM, Joshua Harlow wrote:
> My thinking is that people should look over https://raft.github.io/ or
> http://thesecretlivesofdata.com/raft/ (or both or others...)
>

this was really useful. thanks for this! love how they described it so
simply with visuals. spend a few minutes and look this ^

cheers,

--
gord
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [all] etcd3 as base service - update

Michael Bayer
In reply to this post by Lance Bragstad-2


On 06/08/2017 01:34 PM, Lance Bragstad wrote:

> After digging into etcd a bit, one place this might be help deployer
> experience would be the handling of fernet keys for token encryption in
> keystone. Currently, all keys used to encrypt and decrypt tokens are
> kept on disk for each keystone node in the deployment. While simple, it
> requires operators to perform rotation on a single node and then push,
> or sync, the new key set to the rest of the nodes. This must be done in
> lock step in order to prevent early token invalidation and inconsistent
> token responses.
>
> An alternative would be to keep the keys in etcd and make the fernet
> bits pluggable so that it's possible to read keys from disk or etcd
> (pending configuration). The advantage would be that operators could
> initiate key rotations from any keystone node in the deployment (or
> using etcd directly) and not have to worry about distributing the new
> key set. Since etcd associates metadata to the key-value pairs, we might
> be able to simplify the rotation strategy as well.

Interesting, I had the mis-conception that "fernet" keys no longer
required any server-side storage (how is "kept-on-disk" now
implemented?) .  We've had continuous issues with the pre-fernet
Keystone tokens filling up databases, even when operators were correctly
expunging old tokens; some environments just did so many requests that
the keystone-token table still blew up to where MySQL can no longer
delete from it without producing a too-large transaction for Galera.

So after all the "finally fernet solves this problem" we propose, hey
lets put them *back* in the database :).  That's great.  But, lets
please not leave "cleaning out old tokens" as some kind of
cron/worry-about-it-later thing.  that was a terrible architectural
decision, with apologies to whoever made it.    if you're putting some
kind of "we create an infinite, rapidly growing,
turns-to-garbage-in-30-seconds" kind of data in a database, removing
that data robustly and ASAP needs to be part of the process.





>
> On Thu, Jun 8, 2017 at 11:37 AM, Mike Bayer <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
>
>     On 06/08/2017 12:47 AM, Joshua Harlow wrote:
>
>         So just out of curiosity, but do people really even know what
>         etcd is good for? I am thinking that there should be some
>         guidance from folks in the community as to where etcd should be
>         used and where it shouldn't (otherwise we just all end up in a
>         mess).
>
>
>     So far I've seen a proposal of etcd3 as a replacement for memcached
>     in keystone, and a new dogpile connector was added to oslo.cache to
>     handle referring to etcd3 as a cache backend.  This is a really
>     simplistic / minimal kind of use case for a key-store.
>
>     But, keeping in mind I don't know anything about etcd3 other than
>     "it's another key-store", it's the only database used by Kubernetes
>     as a whole, which suggests it's doing a better job than Redis in
>     terms of "durable".   So I wouldn't be surprised if new / existing
>     openstack applications express some gravitational pull towards using
>     it as their own datastore as well.    I'll be trying to hang onto
>     the etcd3 track as much as possible so that if/when that happens I
>     still have a job :).
>
>
>
>
>
>         Perhaps a good idea to actually give examples of how it should
>         be used, how it shouldn't be used, what it offers, what it
>         doesn't... Or at least provide links for people to read up on this.
>
>         Thoughts?
>
>         Davanum Srinivas wrote:
>
>             One clarification: Since
>             https://pypi.python.org/pypi/etcd3gw
>             <https://pypi.python.org/pypi/etcd3gw> just
>             uses the HTTP API (/v3alpha) it will work under both
>             eventlet and
>             non-eventlet environments.
>
>             Thanks,
>             Dims
>
>
>             On Wed, Jun 7, 2017 at 6:47 AM, Davanum
>             Srinivas<[hidden email] <mailto:[hidden email]>>  wrote:
>
>                 Team,
>
>                 Here's the update to the base services resolution from
>                 the TC:
>                 https://governance.openstack.org/tc/reference/base-services.html
>                 <https://governance.openstack.org/tc/reference/base-services.html>
>
>                 First request is to Distros, Packagers, Deployers,
>                 anyone who
>                 installs/configures OpenStack:
>                 Please make sure you have latest etcd 3.x available in your
>                 environment for Services to use, Fedora already does, we
>                 need help in
>                 making sure all distros and architectures are covered.
>
>                 Any project who want to use etcd v3 API via grpc, please
>                 use:
>                 https://pypi.python.org/pypi/etcd3
>                 <https://pypi.python.org/pypi/etcd3> (works only for
>                 non-eventlet services)
>
>                 Those that depend on eventlet, please use the etcd3
>                 v3alpha HTTP API using:
>                 https://pypi.python.org/pypi/etcd3gw
>                 <https://pypi.python.org/pypi/etcd3gw>
>
>                 If you use tooz, there are 2 driver choices for you:
>                 https://github.com/openstack/tooz/blob/master/setup.cfg#L29
>                 <https://github.com/openstack/tooz/blob/master/setup.cfg#L29>
>                 https://github.com/openstack/tooz/blob/master/setup.cfg#L30
>                 <https://github.com/openstack/tooz/blob/master/setup.cfg#L30>
>
>                 If you use oslo.cache, there is a driver for you:
>                 https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33
>                 <https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33>
>
>                 Devstack installs etcd3 by default and points cinder to it:
>                 http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3
>                 <http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3>
>                 http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356
>                 <http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356>
>
>
>                 Review in progress for keystone to use etcd3 for caching:
>                 https://review.openstack.org/#/c/469621/
>                 <https://review.openstack.org/#/c/469621/>
>
>                 Doug is working on proposal(s) for oslo.config to store some
>                 configuration in etcd3:
>                 https://review.openstack.org/#/c/454897/
>                 <https://review.openstack.org/#/c/454897/>
>
>                 So, feel free to turn on / test with etcd3 and report
>                 issues.
>
>                 Thanks,
>                 Dims
>
>                 --
>                 Davanum Srinivas :: https://twitter.com/dims
>
>
>
>
>
>         __________________________________________________________________________
>         OpenStack Development Mailing List (not for usage questions)
>         Unsubscribe:
>         [hidden email]?subject:unsubscribe
>         <http://OpenStack-dev-request@...?subject:unsubscribe>
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
>
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     [hidden email]?subject:unsubscribe
>     <http://OpenStack-dev-request@...?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
12