[Keystone] Cockroachdb for Keystone Multi-master

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Keystone] Cockroachdb for Keystone Multi-master

Adrian Turjak
Hello fellow OpenStackers,

For the last while I've been looking at options for multi-region
multi-master Keystone, as well as multi-master for other services I've
been developing and one thing that always came up was there aren't many
truly good options for a true multi-master backend. Recently I've been
looking at Cockroachdb and while I haven't had the chance to do any
testing I'm curious if anyone else has looked into it. It sounds like
the perfect solution, and if it can be proved to be stable enough it
could solve a lot of problems.

So, specifically in the realm of Keystone, since we are using sqlalchemy
we already have Postgresql support, and since Cockroachdb does talk
Postgres it shouldn't be too hard to back Keystone with it. At that
stage you have a Keystone DB that could be multi-region, multi-master,
consistent, and mostly impervious to disaster. Is that not the holy
grail for a service like Keystone? Combine that with fernet tokens and
suddenly Keystone becomes a service you can't really kill, and can
mostly forget about.

I'm welcome to being called mad, but I am curious if anyone has looked
at this. I'm likely to do some tests at some stage regarding this,
because I'm hoping this is the solution I've been hoping to find for
quite a long time.

Further reading:
https://www.cockroachlabs.com/
https://github.com/cockroachdb/cockroach
https://www.cockroachlabs.com/docs/build-a-python-app-with-cockroachdb-sqlalchemy.html

Cheers,
- Adrian Turjak


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

Curtis
On Thu, May 18, 2017 at 4:13 PM, Adrian Turjak <[hidden email]> wrote:

> Hello fellow OpenStackers,
>
> For the last while I've been looking at options for multi-region
> multi-master Keystone, as well as multi-master for other services I've
> been developing and one thing that always came up was there aren't many
> truly good options for a true multi-master backend. Recently I've been
> looking at Cockroachdb and while I haven't had the chance to do any
> testing I'm curious if anyone else has looked into it. It sounds like
> the perfect solution, and if it can be proved to be stable enough it
> could solve a lot of problems.
>
> So, specifically in the realm of Keystone, since we are using sqlalchemy
> we already have Postgresql support, and since Cockroachdb does talk
> Postgres it shouldn't be too hard to back Keystone with it. At that
> stage you have a Keystone DB that could be multi-region, multi-master,
> consistent, and mostly impervious to disaster. Is that not the holy
> grail for a service like Keystone? Combine that with fernet tokens and
> suddenly Keystone becomes a service you can't really kill, and can
> mostly forget about.
>
> I'm welcome to being called mad, but I am curious if anyone has looked
> at this. I'm likely to do some tests at some stage regarding this,
> because I'm hoping this is the solution I've been hoping to find for
> quite a long time.

I was going to take a look at this a bit myself, just try it out. I
can't completely speak for the Fog/Edge/Massively Distributed working
group in OpenStack, but I feel like this might be something they look
into.

For standard multi-site I don't know how much it would help, say if
you only had a couple or three clouds, but more than that maybe this
starts to make sense. Also running Galera has gotten easier but still
not that easy.

I had thought that the OpenStack community was deprecating Postgres
support though, so that could make things a bit harder here (I might
be wrong about this).

Thanks,
Curtis.

>
> Further reading:
> https://www.cockroachlabs.com/
> https://github.com/cockroachdb/cockroach
> https://www.cockroachlabs.com/docs/build-a-python-app-with-cockroachdb-sqlalchemy.html
>
> Cheers,
> - Adrian Turjak
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Blog: serverascode.com

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

Adrian Turjak
In reply to this post by Adrian Turjak

On 19 May 2017 11:43 am, Curtis <[hidden email]> wrote:

On Thu, May 18, 2017 at 4:13 PM, Adrian Turjak <[hidden email]> wrote:
> Hello fellow OpenStackers,
>
> For the last while I've been looking at options for multi-region
> multi-master Keystone, as well as multi-master for other services I've
> been developing and one thing that always came up was there aren't many
> truly good options for a true multi-master backend. Recently I've been
> looking at Cockroachdb and while I haven't had the chance to do any
> testing I'm curious if anyone else has looked into it. It sounds like
> the perfect solution, and if it can be proved to be stable enough it
> could solve a lot of problems.
>
> So, specifically in the realm of Keystone, since we are using sqlalchemy
> we already have Postgresql support, and since Cockroachdb does talk
> Postgres it shouldn't be too hard to back Keystone with it. At that
> stage you have a Keystone DB that could be multi-region, multi-master,
> consistent, and mostly impervious to disaster. Is that not the holy
> grail for a service like Keystone? Combine that with fernet tokens and
> suddenly Keystone becomes a service you can't really kill, and can
> mostly forget about.
>
> I'm welcome to being called mad, but I am curious if anyone has looked
> at this. I'm likely to do some tests at some stage regarding this,
> because I'm hoping this is the solution I've been hoping to find for
> quite a long time.

I was going to take a look at this a bit myself, just try it out. I
can't completely speak for the Fog/Edge/Massively Distributed working
group in OpenStack, but I feel like this might be something they look
into.

For standard multi-site I don't know how much it would help, say if
you only had a couple or three clouds, but more than that maybe this
starts to make sense. Also running Galera has gotten easier but still
not that easy.


Multi-site with a shared Keystone was my goal because auth has to be shared in all regions for us. Fernet solves a part of it, but user data, roles, etc also needs to be replicated if we want a Keystone running in each region. That's where Cockroachdb could prove useful.

I had thought that the OpenStack community was deprecating Postgres
support though, so that could make things a bit harder here (I might
be wrong about this).


I really hope not, because that will take Cockroachdb off the table entirely (unless they add MySQL support) and it may prove to be a great option overall once it is known to be stable and has been tested in larger scale setups.

I remember reading about the possibility of deprecating Postgres but there are people using it in production so I assumed we didn't go down that path. Would be good to have someone confirm.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

Chris Dent-2
On Fri, 19 May 2017, Adrian Turjak wrote:

> On 19 May 2017 11:43 am, Curtis <[hidden email]> wrote:
>       I had thought that the OpenStack community was deprecating Postgres
>       support though, so that could make things a bit harder here (I might
>       be wrong about this).
>
> I really hope not, because that will take Cockroachdb off the table entirely (unless they add MySQL support) and it may prove to be a
> great option overall once it is known to be stable and has been tested in larger scale setups.
>
> I remember reading about the possibility of deprecating Postgres but there are people using it in production so I assumed we didn't go
> down that path. Would be good to have someone confirm.
Deprecating postgreSQL is not a done deal, it's up for review at
[1] and [2]. And at this point it is more about documenting reality
that postgreSQL is not a focus of upstream development.

Deprecation is likely to happen, however, if there isn't an increase
in the number people willing to:

* actively share pg knowledge in the OpenStack community
* help with ensuring there is gate testing and responsiveness to
   failures
* address some of the mysql-oriented issue listed in [1]

I'd rather not see it happen, especially if it allows an easy step
to using cockroachdb. So I'd encourage you (and anyone else) to
participate in those reviews, especially if they are able to make
some commitments about future involvement.

[1] https://review.openstack.org/#/c/427880/
[2] https://review.openstack.org/#/c/465589/

--
Chris Dent                  ┬──┬◡ノ(° -°ノ)       https://anticdent.org/
freenode: cdent                                         tw: @anticdent
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

Lance Bragstad-2
In reply to this post by Curtis


On Thu, May 18, 2017 at 6:43 PM, Curtis <[hidden email]> wrote:
On Thu, May 18, 2017 at 4:13 PM, Adrian Turjak <[hidden email]> wrote:
> Hello fellow OpenStackers,
>
> For the last while I've been looking at options for multi-region
> multi-master Keystone, as well as multi-master for other services I've
> been developing and one thing that always came up was there aren't many
> truly good options for a true multi-master backend. Recently I've been
> looking at Cockroachdb and while I haven't had the chance to do any
> testing I'm curious if anyone else has looked into it. It sounds like
> the perfect solution, and if it can be proved to be stable enough it
> could solve a lot of problems.
>
> So, specifically in the realm of Keystone, since we are using sqlalchemy
> we already have Postgresql support, and since Cockroachdb does talk
> Postgres it shouldn't be too hard to back Keystone with it. At that
> stage you have a Keystone DB that could be multi-region, multi-master,
> consistent, and mostly impervious to disaster. Is that not the holy
> grail for a service like Keystone? Combine that with fernet tokens and
> suddenly Keystone becomes a service you can't really kill, and can
> mostly forget about.

++
 
>
> I'm welcome to being called mad, but I am curious if anyone has looked
> at this. I'm likely to do some tests at some stage regarding this,
> because I'm hoping this is the solution I've been hoping to find for
> quite a long time.

I was going to take a look at this a bit myself, just try it out. I
can't completely speak for the Fog/Edge/Massively Distributed working
group in OpenStack, but I feel like this might be something they look
into.

For standard multi-site I don't know how much it would help, say if
you only had a couple or three clouds, but more than that maybe this
starts to make sense. Also running Galera has gotten easier but still
not that easy.

When we originally tested a PoC fernet implementation, we did it globally distributed across five data centers. We didn't generate enough non-token load to notice significant service degradation due to replication lag or issues. I have heard replication across regions in the double digits is where you start getting into some real interesting problems (gyee was one of the folks in keystone who knew more about that). Dusting off those cases with something like cockroachdb would be an interesting exercise!
 

I had thought that the OpenStack community was deprecating Postgres
support though, so that could make things a bit harder here (I might
be wrong about this).

Thanks,
Curtis.

>
> Further reading:
> https://www.cockroachlabs.com/
> https://github.com/cockroachdb/cockroach
> https://www.cockroachlabs.com/docs/build-a-python-app-with-cockroachdb-sqlalchemy.html
>
> Cheers,
> - Adrian Turjak
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Blog: serverascode.com

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

Mike Bayer
In reply to this post by Adrian Turjak


On 05/18/2017 06:13 PM, Adrian Turjak wrote:
>
> So, specifically in the realm of Keystone, since we are using sqlalchemy
> we already have Postgresql support, and since Cockroachdb does talk
> Postgres it shouldn't be too hard to back Keystone with it. At that
> stage you have a Keystone DB that could be multi-region, multi-master,
> consistent, and mostly impervious to disaster. Is that not the holy
> grail for a service like Keystone? Combine that with fernet tokens and
> suddenly Keystone becomes a service you can't really kill, and can
> mostly forget about.

So this is exhibit A for why I think keeping some level of "this might
need to work on other databases" within a codebase is always a great
idea even if you are not actively supporting other DBs at the moment.
Even if Openstack dumped Postgresql completely, I'd not take the
rudimental PG-related utilities out of oslo.db nor would I rename all
the "mysql_XYZ" facilities to be "XYZ".

Cockroachdb advertises SQLAlchemy compatibility very prominently.  While
their tutorial at
https://www.cockroachlabs.com/docs/build-a-python-app-with-cockroachdb-sqlalchemy.html 
says it uses psycopg2 as the database driver, they have implemented
their own "cockroachdb://" dialect on top of it, which likely smooths
out the SQL dialect and connectivity quirks between real Postgresql and
CockroachDB.

This is not the first "distributed database" to build on the Postgresql
protocol, I did a bunch of work for a database that started out called
"Akiban", then got merged to "FoundationDB", and then sadly was sucked
into a black hole shaped like a huge Apple and the entire product and
staff were gone forever.  CockroachDB seems to be filling in that same
hole that I was hoping FoundationDB was going to do (until they fell
into said hole).

>
> I'm welcome to being called mad, but I am curious if anyone has looked
> at this. I'm likely to do some tests at some stage regarding this,
> because I'm hoping this is the solution I've been hoping to find for
> quite a long time.

I'd have a blast if Keystone wanted to get into this.   Distributed /
NewSQL is something I have a lot of optimism about.   Please keep me
looped in.



>
> Further reading:
> https://www.cockroachlabs.com/
> https://github.com/cockroachdb/cockroach
> https://www.cockroachlabs.com/docs/build-a-python-app-with-cockroachdb-sqlalchemy.html
>
> Cheers,
> - Adrian Turjak
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

lebre.adrien
In reply to this post by Curtis


----- Mail original -----

> De: "Curtis" <[hidden email]>
> À: "OpenStack Development Mailing List (not for usage questions)" <[hidden email]>
> Cc: [hidden email]
> Envoyé: Vendredi 19 Mai 2017 01:43:39
> Objet: Re: [openstack-dev] [Keystone] Cockroachdb for Keystone Multi-master
>
> On Thu, May 18, 2017 at 4:13 PM, Adrian Turjak
> <[hidden email]> wrote:
> > Hello fellow OpenStackers,
> >
> > For the last while I've been looking at options for multi-region
> > multi-master Keystone, as well as multi-master for other services
> > I've
> > been developing and one thing that always came up was there aren't
> > many
> > truly good options for a true multi-master backend. Recently I've
> > been
> > looking at Cockroachdb and while I haven't had the chance to do any
> > testing I'm curious if anyone else has looked into it. It sounds
> > like
> > the perfect solution, and if it can be proved to be stable enough
> > it
> > could solve a lot of problems.
> >
> > So, specifically in the realm of Keystone, since we are using
> > sqlalchemy
> > we already have Postgresql support, and since Cockroachdb does talk
> > Postgres it shouldn't be too hard to back Keystone with it. At that
> > stage you have a Keystone DB that could be multi-region,
> > multi-master,
> > consistent, and mostly impervious to disaster. Is that not the holy
> > grail for a service like Keystone? Combine that with fernet tokens
> > and
> > suddenly Keystone becomes a service you can't really kill, and can
> > mostly forget about.
> >
> > I'm welcome to being called mad, but I am curious if anyone has
> > looked
> > at this. I'm likely to do some tests at some stage regarding this,
> > because I'm hoping this is the solution I've been hoping to find
> > for
> > quite a long time.
>
> I was going to take a look at this a bit myself, just try it out. I
> can't completely speak for the Fog/Edge/Massively Distributed working
> group in OpenStack, but I feel like this might be something they look
> into.
>

Thanks Curtis for highlighting this.

Indeed, among the actions we defined during our F2F meeting in Boston, one is to investigate the feasibility of using cockroachDB as the backend. We gave really preliminary investigations and identify several aspects that we should consider (dependence w-r-t postgres, maturity of cockroach code,...)

For your information, we did a PoC one year ago to show that it is possible to use a Non SQL backend (i.e Redis in our case) instead of the historical MySQL backends. Our goal was to investigate a solution to distribute the storage backends accros several nodes/sites. Although it was ``just'' a PoC (i.e., the quality of the code was definitely improvable), it demonstrated that using a non-SQL backend can make sense (for more information see [1] [2]). Among the remarks we got, the fact of loosing the SQL API seemed to be an issue.

Using new SQL systems such as CockRoach can solve this issue and that's why we want to study how this can be done (1./ identify the effort in terms of development, 2./ find a way to evaluate performance pros/cons 3./ do the job)

Among the different systems (cockroach, vitess, ...), the important point from the FEMDC working group [3] is the fact that we do not want to have centralized service. From what I learnt by browsing a few pages of the Vitess website, the architecture does not satisfy this aspect.

In any case, we need to spend more times on these questions before having enough materials to get valuable answers.
In others words, if you are interested to deal with such questions, do not hesitate to take part in our IRC meetings and follow the mailing list.

Best,
Ad_rien_


[1] https://hal.inria.fr/hal-01273427/
[2] https://www.openstack.org/summit/austin-2016/summit-schedule/events/7342
[3] https://wiki.openstack.org/wiki/Fog_Edge_Massively_Distributed_Clouds


> For standard multi-site I don't know how much it would help, say if
> you only had a couple or three clouds, but more than that maybe this
> starts to make sense. Also running Galera has gotten easier but still
> not that easy.
>
> I had thought that the OpenStack community was deprecating Postgres
> support though, so that could make things a bit harder here (I might
> be wrong about this).
>
> Thanks,
> Curtis.
>
> >
> > Further reading:
> > https://www.cockroachlabs.com/
> > https://github.com/cockroachdb/cockroach
> > https://www.cockroachlabs.com/docs/build-a-python-app-with-cockroachdb-sqlalchemy.html
> >
> > Cheers,
> > - Adrian Turjak
> >
> >
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > [hidden email]?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Blog: serverascode.com
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

Adrian Turjak
In reply to this post by Mike Bayer

On 20/05/17 09:31, Mike Bayer wrote:

>
> On 05/18/2017 06:13 PM, Adrian Turjak wrote:
>>
>> So, specifically in the realm of Keystone, since we are using sqlalchemy
>> we already have Postgresql support, and since Cockroachdb does talk
>> Postgres it shouldn't be too hard to back Keystone with it. At that
>> stage you have a Keystone DB that could be multi-region, multi-master,
>> consistent, and mostly impervious to disaster. Is that not the holy
>> grail for a service like Keystone? Combine that with fernet tokens and
>> suddenly Keystone becomes a service you can't really kill, and can
>> mostly forget about.
>
> So this is exhibit A for why I think keeping some level of "this might
> need to work on other databases" within a codebase is always a great
> idea even if you are not actively supporting other DBs at the moment.
> Even if Openstack dumped Postgresql completely, I'd not take the
> rudimental PG-related utilities out of oslo.db nor would I rename all
> the "mysql_XYZ" facilities to be "XYZ".
I have posted on the reviews, but my hope is that if we do drop
PostgreSQL support we at least also state that we will not support any
database specific features that preclude someone from using/supporting a
different database should they wish to, and that we will not deny
patches that add/fix support for other databases provided they do not
interfere with support for the ones we do 'support' officially.

>
> Cockroachdb advertises SQLAlchemy compatibility very prominently.
> While their tutorial at
> https://www.cockroachlabs.com/docs/build-a-python-app-with-cockroachdb-sqlalchemy.html
> says it uses psycopg2 as the database driver, they have implemented
> their own "cockroachdb://" dialect on top of it, which likely smooths
> out the SQL dialect and connectivity quirks between real Postgresql
> and CockroachDB.
>
> This is not the first "distributed database" to build on the
> Postgresql protocol, I did a bunch of work for a database that started
> out called "Akiban", then got merged to "FoundationDB", and then sadly
> was sucked into a black hole shaped like a huge Apple and the entire
> product and staff were gone forever.  CockroachDB seems to be filling
> in that same hole that I was hoping FoundationDB was going to do
> (until they fell into said hole).
>
>>
>> I'm welcome to being called mad, but I am curious if anyone has looked
>> at this. I'm likely to do some tests at some stage regarding this,
>> because I'm hoping this is the solution I've been hoping to find for
>> quite a long time.
>
> I'd have a blast if Keystone wanted to get into this.   Distributed /
> NewSQL is something I have a lot of optimism about.   Please keep me
> looped in.
>

I can't speak for the Keystone team itself, this was mainly my own
speculation and general ideas about how to better handle
multi-master/DR. I know that standard MySQL async mult-master would work
for a lot of cases, but async multi-master has a bunch of problems that
are often very weird, rare, or painful and hard to debug so people tend
to avoid it unless there is no other option.  That's why I have a lot of
interest in CockroachDB because it was built pretty much entirely to
solve just that case in the Google Spanner mold.

The scenario I'm interested with this is Keystone setup for Fernet
tokens, multi-site, multi-master, and geo-loadbalancing so you always
talk to your nearest datacenter. With non-admin users being able to self
manage their own projects, users, and roles within some scope. Thus data
needs to be consistent across regions because of DR and
geo-loadbalancing, but also you may have multiple people editing the
same users at the same time in different places. I'm curious how much
better Cockroachdb may handle those cases, but from the looks of things
a lot of those problems aren't there or only crop up in very large or
complex transactions which Keystone doesn't seem to have too many of (I
could be wrong though). My use case likely isn't going to hit many
problems with multi-master since we are currently only running 3 sites
that share auth, but I prefer to plan for worst case scenario! Plus the
more I can trust the DB layer, the more I can focus elsewhere.


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

Thierry Carrez
In reply to this post by Mike Bayer
Mike Bayer wrote:

> On 05/18/2017 06:13 PM, Adrian Turjak wrote:
>>
>> So, specifically in the realm of Keystone, since we are using sqlalchemy
>> we already have Postgresql support, and since Cockroachdb does talk
>> Postgres it shouldn't be too hard to back Keystone with it. At that
>> stage you have a Keystone DB that could be multi-region, multi-master,
>> consistent, and mostly impervious to disaster. Is that not the holy
>> grail for a service like Keystone? Combine that with fernet tokens and
>> suddenly Keystone becomes a service you can't really kill, and can
>> mostly forget about.
>
> So this is exhibit A for why I think keeping some level of "this might
> need to work on other databases" within a codebase is always a great
> idea even if you are not actively supporting other DBs at the moment.
> Even if Openstack dumped Postgresql completely, I'd not take the
> rudimental PG-related utilities out of oslo.db nor would I rename all
> the "mysql_XYZ" facilities to be "XYZ".
> [...]
Yes, that sounds like another reason why we'd not want to aggressively
contract to the MySQL family of databases. At the very least, before we
do that, we should experiment with CockroachDB and see how reasonable it
would be to use in an OpenStack context. It might (might) hit a sweet
spot between performance, durability, database decentralization and
keeping SQL advanced features -- I'd hate it if we discovered that too late.

--
Thierry Carrez (ttx)

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

Mike Bayer


On 05/22/2017 05:02 AM, Thierry Carrez wrote:

> Mike Bayer wrote:
>> On 05/18/2017 06:13 PM, Adrian Turjak wrote:
>>>
>>> So, specifically in the realm of Keystone, since we are using sqlalchemy
>>> we already have Postgresql support, and since Cockroachdb does talk
>>> Postgres it shouldn't be too hard to back Keystone with it. At that
>>> stage you have a Keystone DB that could be multi-region, multi-master,
>>> consistent, and mostly impervious to disaster. Is that not the holy
>>> grail for a service like Keystone? Combine that with fernet tokens and
>>> suddenly Keystone becomes a service you can't really kill, and can
>>> mostly forget about.
>>
>> So this is exhibit A for why I think keeping some level of "this might
>> need to work on other databases" within a codebase is always a great
>> idea even if you are not actively supporting other DBs at the moment.
>> Even if Openstack dumped Postgresql completely, I'd not take the
>> rudimental PG-related utilities out of oslo.db nor would I rename all
>> the "mysql_XYZ" facilities to be "XYZ".
>> [...]
> Yes, that sounds like another reason why we'd not want to aggressively
> contract to the MySQL family of databases. At the very least, before we
> do that, we should experiment with CockroachDB and see how reasonable it
> would be to use in an OpenStack context. It might (might) hit a sweet
> spot between performance, durability, database decentralization and
> keeping SQL advanced features -- I'd hate it if we discovered that too late.

there's a difference between "architecting for pluggability" and
"supporting database X, Y, and Z".   I only maintain we should keep the
notion of pluggability around.  This doesn't mean you can't use MySQL
specific features, it only means, anytime you're using a MySQL feature,
it's in the context of a unit of code that would be swapped out when a
different database backend were to be implemented.   The vast majority
of our database code is like this already, mostly implicitly due to
SQLAlchemy and in other cases explicitly as we see in a lot of the
migration scripts.

I think the existence of the PG backend, combined with the immediate
interest in getting NDB to work, and now Cockroach DB, not to mention
that there are two major MySQL variants (MySQL, MariaDB) which do have
significant differences (the JSON type one of the biggest examples)
should suggest that any modern database-enabled application can't really
afford to completely hardcode to a single database backend without at
least basic layers of abstraction being present.

>

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

Jay Pipes
In reply to this post by Adrian Turjak
Sorry for the delay in getting back on this... comments inline.

On 05/18/2017 06:13 PM, Adrian Turjak wrote:
> Hello fellow OpenStackers,
>
> For the last while I've been looking at options for multi-region
> multi-master Keystone, as well as multi-master for other services I've
> been developing and one thing that always came up was there aren't many
> truly good options for a true multi-master backend.

Not sure whether you've looked into Galera? We had a geo-distributed
12-site Galera cluster servicing our Keystone assignment/identity
information WAN-replicated. Worked a charm for us at AT&T. Much easier
to administer than master-slave replication topologies and the
performance (yes, even over WAN links) of the ws-rep replication was
excellent. And yes, I'm aware Galera doesn't have complete snapshot
isolation support, but for Keystone's workloads (heavy, heavy read, very
little write) it is indeed ideal.

(BTW, the cluster setup and node-join operations for CockroachDB and
Galera are virtually identical...)

 > Recently I've been
> looking at Cockroachdb and while I haven't had the chance to do any
> testing I'm curious if anyone else has looked into it. It sounds like
> the perfect solution, and if it can be proved to be stable enough it
> could solve a lot of problems.
>
> So, specifically in the realm of Keystone, since we are using sqlalchemy
> we already have Postgresql support, and since Cockroachdb does talk
> Postgres it shouldn't be too hard to back Keystone with it.

OK, now I understand why you didn't consider Galera :) Sounds like
you're pinned to PostgreSQL for your RDBMS needs...

For the record, CockroachDB doesn't "support PostgreSQL". It supports
the binary pgsql client/server protocol and, oddly, a view of the
internal system information in PostgreSQL's pg_catalog schema (though
also publishes the standard information_schema schema).

The actual *SQL* that CockroachDB uses is definitely not PostgreSQL's
variant of SQL. CockroachDB's version of SQL is actually pretty close to
MySQL's version of SQL in a number of ways:

  * EXPLAIN
  * SHOW (TABLES, COLUMNS, CREATE TABLE, DATABASES, etc)
  * RENAME (TABLE, DATABASE, COLUMN, etc)

In other ways, CockroachDB's version of SQL is more like PostgreSQL's
including:

  * UPSERT (MySQL uses the INSERT ... ON DUPLICATE KEY UPDATE construct)

 > At that

> stage you have a Keystone DB that could be multi-region, multi-master,
> consistent, and mostly impervious to disaster. Is that not the holy
> grail for a service like Keystone? Combine that with fernet tokens and
> suddenly Keystone becomes a service you can't really kill, and can
> mostly forget about.
>
> I'm welcome to being called mad, but I am curious if anyone has looked
> at this. I'm likely to do some tests at some stage regarding this,
> because I'm hoping this is the solution I've been hoping to find for
> quite a long time.
>
> Further reading:
> https://www.cockroachlabs.com/
> https://github.com/cockroachdb/cockroach
> https://www.cockroachlabs.com/docs/build-a-python-app-with-cockroachdb-sqlalchemy.html

Another link for folks to read:

https://jepsen.io/analyses/cockroachdb-beta-20160829

I think it's worth investigating and thoroughly testing CockroachDB.
But, it's pretty new, frankly, and I wouldn't bet a production system on
it. Also, please do follow up on the performance of CockroachDB, which
as aphyr notes in the jepsen link above, was far, far below other RDBMS
that have been tested.

Best,
-jay

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

Clint Byrum
Excerpts from Jay Pipes's message of 2017-05-30 14:52:01 -0400:

> Sorry for the delay in getting back on this... comments inline.
>
> On 05/18/2017 06:13 PM, Adrian Turjak wrote:
> > Hello fellow OpenStackers,
> >
> > For the last while I've been looking at options for multi-region
> > multi-master Keystone, as well as multi-master for other services I've
> > been developing and one thing that always came up was there aren't many
> > truly good options for a true multi-master backend.
>
> Not sure whether you've looked into Galera? We had a geo-distributed
> 12-site Galera cluster servicing our Keystone assignment/identity
> information WAN-replicated. Worked a charm for us at AT&T. Much easier
> to administer than master-slave replication topologies and the
> performance (yes, even over WAN links) of the ws-rep replication was
> excellent. And yes, I'm aware Galera doesn't have complete snapshot
> isolation support, but for Keystone's workloads (heavy, heavy read, very
> little write) it is indeed ideal.
>

This has not been my experience.

We had a 3 site, 9 node global cluster and it was _extremely_ sensitive
to latency. We'd lose even read ability whenever we had a latency storm
due to quorum problems.

Our sites were London, Dallas, and Sydney, so it was pretty common for
there to be latency between any of them.

I lost track of it after some reorgs, but I believe the solution was
to just have a single site 3-node galera for writes, and then use async
replication for reads. We even helped land patches in Keystone to allow
split read/write host configuration.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

Jay Pipes
On 05/30/2017 05:07 PM, Clint Byrum wrote:

> Excerpts from Jay Pipes's message of 2017-05-30 14:52:01 -0400:
>> Sorry for the delay in getting back on this... comments inline.
>>
>> On 05/18/2017 06:13 PM, Adrian Turjak wrote:
>>> Hello fellow OpenStackers,
>>>
>>> For the last while I've been looking at options for multi-region
>>> multi-master Keystone, as well as multi-master for other services I've
>>> been developing and one thing that always came up was there aren't many
>>> truly good options for a true multi-master backend.
>>
>> Not sure whether you've looked into Galera? We had a geo-distributed
>> 12-site Galera cluster servicing our Keystone assignment/identity
>> information WAN-replicated. Worked a charm for us at AT&T. Much easier
>> to administer than master-slave replication topologies and the
>> performance (yes, even over WAN links) of the ws-rep replication was
>> excellent. And yes, I'm aware Galera doesn't have complete snapshot
>> isolation support, but for Keystone's workloads (heavy, heavy read, very
>> little write) it is indeed ideal.
>>
>
> This has not been my experience.
>
> We had a 3 site, 9 node global cluster and it was _extremely_ sensitive
> to latency. We'd lose even read ability whenever we had a latency storm
> due to quorum problems.
>
> Our sites were London, Dallas, and Sydney, so it was pretty common for
> there to be latency between any of them.
>
> I lost track of it after some reorgs, but I believe the solution was
> to just have a single site 3-node galera for writes, and then use async
> replication for reads. We even helped land patches in Keystone to allow
> split read/write host configuration.

Interesting, thanks for the info. Can I ask, were you using the Galera
cluster for read-heavy data like Keystone identity/assignment storage?
Or did you have write-heavy data mixed in (like Keystone's old UUID
token storage...)

It should be noted that CockroachDB's documentation specifically calls
out that it is extremely sensitive to latency due to the way it measures
clock skew... so might not be suitable for WAN-separated clusters?

Best,
-jay

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

Clint Byrum
Excerpts from Jay Pipes's message of 2017-05-30 21:06:59 -0400:

> On 05/30/2017 05:07 PM, Clint Byrum wrote:
> > Excerpts from Jay Pipes's message of 2017-05-30 14:52:01 -0400:
> >> Sorry for the delay in getting back on this... comments inline.
> >>
> >> On 05/18/2017 06:13 PM, Adrian Turjak wrote:
> >>> Hello fellow OpenStackers,
> >>>
> >>> For the last while I've been looking at options for multi-region
> >>> multi-master Keystone, as well as multi-master for other services I've
> >>> been developing and one thing that always came up was there aren't many
> >>> truly good options for a true multi-master backend.
> >>
> >> Not sure whether you've looked into Galera? We had a geo-distributed
> >> 12-site Galera cluster servicing our Keystone assignment/identity
> >> information WAN-replicated. Worked a charm for us at AT&T. Much easier
> >> to administer than master-slave replication topologies and the
> >> performance (yes, even over WAN links) of the ws-rep replication was
> >> excellent. And yes, I'm aware Galera doesn't have complete snapshot
> >> isolation support, but for Keystone's workloads (heavy, heavy read, very
> >> little write) it is indeed ideal.
> >>
> >
> > This has not been my experience.
> >
> > We had a 3 site, 9 node global cluster and it was _extremely_ sensitive
> > to latency. We'd lose even read ability whenever we had a latency storm
> > due to quorum problems.
> >
> > Our sites were London, Dallas, and Sydney, so it was pretty common for
> > there to be latency between any of them.
> >
> > I lost track of it after some reorgs, but I believe the solution was
> > to just have a single site 3-node galera for writes, and then use async
> > replication for reads. We even helped land patches in Keystone to allow
> > split read/write host configuration.
>
> Interesting, thanks for the info. Can I ask, were you using the Galera
> cluster for read-heavy data like Keystone identity/assignment storage?
> Or did you have write-heavy data mixed in (like Keystone's old UUID
> token storage...)
>
> It should be noted that CockroachDB's documentation specifically calls
> out that it is extremely sensitive to latency due to the way it measures
> clock skew... so might not be suitable for WAN-separated clusters?
>

That particular Galera was for Keystone only, and we were using Fernet
tokens.

Revocation events were a constant but manageable source of writes. I
believe some optimizations were made to reduce the frequency of the events
but that was after we had worked around the problems they created. Using
async replication simply meant that we were accepting the replication lag
window as a period of time where a revocation event might not apply. I
don't know that we ever got hard numbers, but with the write data we had
we speculated the worst result would be that you'd revoke a token in
Dallas and Sydney or London might keep working with that token for
however long a latency storm lasted + the recovery time for applying the
backlog. At worst, a few minutes.

Either way, it should be much simpler to manage slave lag than to deal
with a Galera cluster that won't accept any writes at all because it
can't get quorum.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

Jay Pipes
On 05/31/2017 02:14 AM, Clint Byrum wrote:
> Either way, it should be much simpler to manage slave lag than to deal
> with a Galera cluster that won't accept any writes at all because it
> can't get quorum.

Would CockroachDB be any better at achieving quorum?

Genuinely curious. :)

Best,
-jay



__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

Amrith Kumar-2

> -----Original Message-----
> From: Jay Pipes [mailto:[hidden email]]
> Sent: Wednesday, May 31, 2017 12:00 PM
> To: [hidden email]
> Subject: Re: [openstack-dev] [Keystone] Cockroachdb for Keystone Multi-master
>
> On 05/31/2017 02:14 AM, Clint Byrum wrote:
> > Either way, it should be much simpler to manage slave lag than to deal
> > with a Galera cluster that won't accept any writes at all because it
> > can't get quorum.
>
> Would CockroachDB be any better at achieving quorum?
>
> Genuinely curious. :)

[Amrith Kumar] Last I read about this was in [1] and it sounded like it would be no better, or worse.

[1] https://www.cockroachlabs.com/blog/consensus-made-thrive/

>
> Best,
> -jay
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

Mike Bayer
In reply to this post by Jay Pipes


On 05/30/2017 09:06 PM, Jay Pipes wrote:

> On 05/30/2017 05:07 PM, Clint Byrum wrote:
>> Excerpts from Jay Pipes's message of 2017-05-30 14:52:01 -0400:
>>> Sorry for the delay in getting back on this... comments inline.
>>>
>>> On 05/18/2017 06:13 PM, Adrian Turjak wrote:
>>>> Hello fellow OpenStackers,
>>>>
>>>> For the last while I've been looking at options for multi-region
>>>> multi-master Keystone, as well as multi-master for other services I've
>>>> been developing and one thing that always came up was there aren't many
>>>> truly good options for a true multi-master backend.
>>>
>>> Not sure whether you've looked into Galera? We had a geo-distributed
>>> 12-site Galera cluster servicing our Keystone assignment/identity
>>> information WAN-replicated. Worked a charm for us at AT&T. Much easier
>>> to administer than master-slave replication topologies and the
>>> performance (yes, even over WAN links) of the ws-rep replication was
>>> excellent. And yes, I'm aware Galera doesn't have complete snapshot
>>> isolation support, but for Keystone's workloads (heavy, heavy read, very
>>> little write) it is indeed ideal.
>>>
>>
>> This has not been my experience.
>>
>> We had a 3 site, 9 node global cluster and it was _extremely_ sensitive
>> to latency. We'd lose even read ability whenever we had a latency storm
>> due to quorum problems.
>>
>> Our sites were London, Dallas, and Sydney, so it was pretty common for
>> there to be latency between any of them.
>>
>> I lost track of it after some reorgs, but I believe the solution was
>> to just have a single site 3-node galera for writes, and then use async
>> replication for reads. We even helped land patches in Keystone to allow
>> split read/write host configuration.
>
> Interesting, thanks for the info. Can I ask, were you using the Galera
> cluster for read-heavy data like Keystone identity/assignment storage?
> Or did you have write-heavy data mixed in (like Keystone's old UUID
> token storage...)

I'd also throw in, there's lots of versions of Galera with different
bugfixes / improvements as we go along, not to mention configuration
settings.... if Jay observes it working great on a distributed cluster
and Clint observes it working terribly, it could be that these were not
the same Galera versions being used.



>
> It should be noted that CockroachDB's documentation specifically calls
> out that it is extremely sensitive to latency due to the way it measures
> clock skew... so might not be suitable for WAN-separated clusters?
>
> Best,
> -jay
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

Jay Pipes
On 05/31/2017 11:06 PM, Mike Bayer wrote:
> I'd also throw in, there's lots of versions of Galera with different
> bugfixes / improvements as we go along, not to mention configuration
> settings.... if Jay observes it working great on a distributed cluster
> and Clint observes it working terribly, it could be that these were not
> the same Galera versions being used.

Agreed. The version of Galera we were using IIRC was Percona XtraDB
Cluster 5.6. And, remember that the wsrep_provider_options do make a big
difference, especially in WAN-replicated setups.

We also increased the tolerance settings for network disruption so that
the cluster operated without hiccups over the WAN. I think the
wsrep_provider_options setting was evs.inactive_timeout=PT30Sm
evs.suspect_timeout=PT15S, and evs.join_retrans_period=PT1S.

Also, regardless of settings, if your network sucks, none of these
distributed databases are going to be fun to operate :)

At AT&T, we jumped through a lot of hoops to ensure multiple levels of
redundancy and high performance for the network links inside and between
datacenters. It really makes a huge difference when your network rocks.

Best,
-jay

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

Andrey Grebennikov
We had a very similar conversation multiple times with Keystone cores (multi-site Keystone).
Geo-rep Galera was suggested first and it was immediately declined (one of the reasons was the case of complete corruption of Keystone DB everywhere in case of accidental table corrupt in one site) by me as well as current customer.
Right after that I was told many times that federation is the only right way to go nowadays.

Is this statement still valid?

On Thu, Jun 1, 2017 at 12:51 PM, Jay Pipes <[hidden email]> wrote:
On 05/31/2017 11:06 PM, Mike Bayer wrote:
I'd also throw in, there's lots of versions of Galera with different bugfixes / improvements as we go along, not to mention configuration settings.... if Jay observes it working great on a distributed cluster and Clint observes it working terribly, it could be that these were not the same Galera versions being used.

Agreed. The version of Galera we were using IIRC was Percona XtraDB Cluster 5.6. And, remember that the wsrep_provider_options do make a big difference, especially in WAN-replicated setups.

We also increased the tolerance settings for network disruption so that the cluster operated without hiccups over the WAN. I think the wsrep_provider_options setting was evs.inactive_timeout=PT30Sm evs.suspect_timeout=PT15S, and evs.join_retrans_period=PT1S.

Also, regardless of settings, if your network sucks, none of these distributed databases are going to be fun to operate :)

At AT&T, we jumped through a lot of hoops to ensure multiple levels of redundancy and high performance for the network links inside and between datacenters. It really makes a huge difference when your network rocks.


Best,
-jay

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@...enstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Andrey Grebennikov
Principal Deployment Engineer
Mirantis Inc, Austin TX

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Keystone] Cockroachdb for Keystone Multi-master

Lance Bragstad-2


On Thu, Jun 1, 2017 at 3:46 PM, Andrey Grebennikov <[hidden email]> wrote:
We had a very similar conversation multiple times with Keystone cores (multi-site Keystone). 
Geo-rep Galera was suggested first and it was immediately declined (one of the reasons was the case of complete corruption of Keystone DB everywhere in case of accidental table corrupt in one site) by me as well as current customer.
Right after that I was told many times that federation is the only right way to go nowadays.

After doing some digging, I found the original specification [0] and the meeting agenda [1] where we talked about the alternative.

If I recall correctly, I thought I remember the proposal (being able to specify project IDs at creation time) being driven by not wanting to replicate all of keystone's backends in multi-region deployments,but still wanting to validate tokens across regions. Today, if you have a region in Seattle and region in Sydney, a token obtained from a keystone in Seattle and validated in Sydney would require both regions to share identity, resource, and assignment backends (among others depending on what kind of token it is). The request in the specification would allow only the identity and role backends to be replicated but the project backend in each region wouldn't need to be synced or replicated. Instead, operators could create projects with matching IDs in each region in order for tokens generated from one to be validated in the other. Most folks involved in the meeting considered this behavior for project IDs to be a slippery-slope.

Federation was brought up because sharing identity information globally, but not project or role information globally sounded like federation (e.g. having all your user information in an IdP somewhere and setting up each region's keystone to federate to the IdP). The group seemed eager to expose gaps in the federation implementation that prevented that case and address those.

Hopefully that helps capture some of the context (feel free to fill in gaps if I missed any).


 

Is this statement still valid?

On Thu, Jun 1, 2017 at 12:51 PM, Jay Pipes <[hidden email]> wrote:
On 05/31/2017 11:06 PM, Mike Bayer wrote:
I'd also throw in, there's lots of versions of Galera with different bugfixes / improvements as we go along, not to mention configuration settings.... if Jay observes it working great on a distributed cluster and Clint observes it working terribly, it could be that these were not the same Galera versions being used.

Agreed. The version of Galera we were using IIRC was Percona XtraDB Cluster 5.6. And, remember that the wsrep_provider_options do make a big difference, especially in WAN-replicated setups.

We also increased the tolerance settings for network disruption so that the cluster operated without hiccups over the WAN. I think the wsrep_provider_options setting was evs.inactive_timeout=PT30Sm evs.suspect_timeout=PT15S, and evs.join_retrans_period=PT1S.

Also, regardless of settings, if your network sucks, none of these distributed databases are going to be fun to operate :)

At AT&T, we jumped through a lot of hoops to ensure multiple levels of redundancy and high performance for the network links inside and between datacenters. It really makes a huge difference when your network rocks.


Best,
-jay

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@...enstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Andrey Grebennikov
Principal Deployment Engineer
Mirantis Inc, Austin TX

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
12
Loading...