[qa] [keystone] Random Patrole failures related to Identity v3 Extensions API

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[qa] [keystone] Random Patrole failures related to Identity v3 Extensions API

Felipe Monteiro
Patrole tests occasionally fail while executing tests that test the
Identity v3 Extensions API [0]. Previously, this was not the case when
we used Fernet tokens and used a time.sleep(1) to allow for
role-switching to work correctly. However, we recently changed over to
UUID tokens in the Patrole gates to avoid doing a time.sleep(1), as a
time efficiency change. Ordinarily -- for well over 500 or so tests --
this approach works successfully, with the exception of what appears
to be *random* v3 API extension tests [1][2] (random means different
tests pass or fail randomly across separate test runs).

While there are a few solutions that come to mind on how to solve this
Patrole-side (like re-introducing a time.sleep() for specific APIs or
even avoiding role-switching altogether which is not as
straightforward as it sounds), we would still not understand *why* the
issue is happening in the first place. Is it a data-race condition?
Something specific to the identity v3 extensions API? A potential bug
or intended behavior somewhere?

[0] https://developer.openstack.org/api-ref/identity/v3-ext/
[1] http://status.openstack.org/openstack-health/#/test/patrole_tempest_plugin.tests.api.identity.v3.test_ep_filter_groups_rbac.EndpointFilterGroupsV3RbacTest.test_create_endpoint_group
[2] http://logs.openstack.org/41/490641/3/gate/gate-tempest-dsvm-patrole-py35-member-ubuntu-xenial/be95da4/console.html#_2017-08-11_14_47_57_515906

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [qa] [keystone] Random Patrole failures related to Identity v3 Extensions API

Morgan Fainberg-2
On Fri, Aug 11, 2017 at 8:44 AM, Felipe Monteiro
<[hidden email]> wrote:

> Patrole tests occasionally fail while executing tests that test the
> Identity v3 Extensions API [0]. Previously, this was not the case when
> we used Fernet tokens and used a time.sleep(1) to allow for
> role-switching to work correctly. However, we recently changed over to
> UUID tokens in the Patrole gates to avoid doing a time.sleep(1), as a
> time efficiency change. Ordinarily -- for well over 500 or so tests --
> this approach works successfully, with the exception of what appears
> to be *random* v3 API extension tests [1][2] (random means different
> tests pass or fail randomly across separate test runs).
>
> While there are a few solutions that come to mind on how to solve this
> Patrole-side (like re-introducing a time.sleep() for specific APIs or
> even avoiding role-switching altogether which is not as
> straightforward as it sounds), we would still not understand *why* the
> issue is happening in the first place. Is it a data-race condition?
> Something specific to the identity v3 extensions API? A potential bug
> or intended behavior somewhere?
>
> [0] https://developer.openstack.org/api-ref/identity/v3-ext/
> [1] http://status.openstack.org/openstack-health/#/test/patrole_tempest_plugin.tests.api.identity.v3.test_ep_filter_groups_rbac.EndpointFilterGroupsV3RbacTest.test_create_endpoint_group
> [2] http://logs.openstack.org/41/490641/3/gate/gate-tempest-dsvm-patrole-py35-member-ubuntu-xenial/be95da4/console.html#_2017-08-11_14_47_57_515906
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Are your tests causing token revocations? if so, there is a case where
a revocation event is issued in the same second as a token (we've seen
similar cases even in fernet) meaning the token is invalid when it is
issued according to keystone. It's a long running bug.

For the record, UUID tokens are deprecated and slated for removal in
the R release. I recommend reverting to using Fernet tokens sooner
rather than later.

Last of all, the endpoint-filtering is generally not a great tool to
use. I highly recommend not using it (or encouraging the use of it),
it makes the catalog different depending on scope and provides zero
added security benefit (anyone who knows the endpoint can use it
anyway).

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [qa] [keystone] Random Patrole failures related to Identity v3 Extensions API

Morgan Fainberg-2
On Fri, Aug 11, 2017 at 9:25 AM, Morgan Fainberg
<[hidden email]> wrote:

> On Fri, Aug 11, 2017 at 8:44 AM, Felipe Monteiro
> <[hidden email]> wrote:
>> Patrole tests occasionally fail while executing tests that test the
>> Identity v3 Extensions API [0]. Previously, this was not the case when
>> we used Fernet tokens and used a time.sleep(1) to allow for
>> role-switching to work correctly. However, we recently changed over to
>> UUID tokens in the Patrole gates to avoid doing a time.sleep(1), as a
>> time efficiency change. Ordinarily -- for well over 500 or so tests --
>> this approach works successfully, with the exception of what appears
>> to be *random* v3 API extension tests [1][2] (random means different
>> tests pass or fail randomly across separate test runs).
>>
>> While there are a few solutions that come to mind on how to solve this
>> Patrole-side (like re-introducing a time.sleep() for specific APIs or
>> even avoiding role-switching altogether which is not as
>> straightforward as it sounds), we would still not understand *why* the
>> issue is happening in the first place. Is it a data-race condition?
>> Something specific to the identity v3 extensions API? A potential bug
>> or intended behavior somewhere?
>>
>> [0] https://developer.openstack.org/api-ref/identity/v3-ext/
>> [1] http://status.openstack.org/openstack-health/#/test/patrole_tempest_plugin.tests.api.identity.v3.test_ep_filter_groups_rbac.EndpointFilterGroupsV3RbacTest.test_create_endpoint_group
>> [2] http://logs.openstack.org/41/490641/3/gate/gate-tempest-dsvm-patrole-py35-member-ubuntu-xenial/be95da4/console.html#_2017-08-11_14_47_57_515906
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: [hidden email]?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> Are your tests causing token revocations? if so, there is a case where
> a revocation event is issued in the same second as a token (we've seen
> similar cases even in fernet) meaning the token is invalid when it is
> issued according to keystone. It's a long running bug.
>
> For the record, UUID tokens are deprecated and slated for removal in
> the R release. I recommend reverting to using Fernet tokens sooner
> rather than later.
>
> Last of all, the endpoint-filtering is generally not a great tool to
> use. I highly recommend not using it (or encouraging the use of it),
> it makes the catalog different depending on scope and provides zero
> added security benefit (anyone who knows the endpoint can use it
> anyway).

I am spinning up a change for Pike RC (right now) to address the
revocations occurring in the same second as the issuance of the token
(similar to a bug with password updates, which will also be fixed).

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [qa] [keystone] Random Patrole failures related to Identity v3 Extensions API

Lance Bragstad-2
More context on the patch Morgan is working on can be found in the bug
report [0].
[0]

On 08/11/2017 11:26 AM, Morgan Fainberg wrote:

> On Fri, Aug 11, 2017 at 9:25 AM, Morgan Fainberg
> <[hidden email]> wrote:
>> On Fri, Aug 11, 2017 at 8:44 AM, Felipe Monteiro
>> <[hidden email]> wrote:
>>> Patrole tests occasionally fail while executing tests that test the
>>> Identity v3 Extensions API [0]. Previously, this was not the case when
>>> we used Fernet tokens and used a time.sleep(1) to allow for
>>> role-switching to work correctly. However, we recently changed over to
>>> UUID tokens in the Patrole gates to avoid doing a time.sleep(1), as a
>>> time efficiency change. Ordinarily -- for well over 500 or so tests --
>>> this approach works successfully, with the exception of what appears
>>> to be *random* v3 API extension tests [1][2] (random means different
>>> tests pass or fail randomly across separate test runs).
>>>
>>> While there are a few solutions that come to mind on how to solve this
>>> Patrole-side (like re-introducing a time.sleep() for specific APIs or
>>> even avoiding role-switching altogether which is not as
>>> straightforward as it sounds), we would still not understand *why* the
>>> issue is happening in the first place. Is it a data-race condition?
>>> Something specific to the identity v3 extensions API? A potential bug
>>> or intended behavior somewhere?
>>>
>>> [0] https://developer.openstack.org/api-ref/identity/v3-ext/
>>> [1] http://status.openstack.org/openstack-health/#/test/patrole_tempest_plugin.tests.api.identity.v3.test_ep_filter_groups_rbac.EndpointFilterGroupsV3RbacTest.test_create_endpoint_group
>>> [2] http://logs.openstack.org/41/490641/3/gate/gate-tempest-dsvm-patrole-py35-member-ubuntu-xenial/be95da4/console.html#_2017-08-11_14_47_57_515906
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: [hidden email]?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> Are your tests causing token revocations? if so, there is a case where
>> a revocation event is issued in the same second as a token (we've seen
>> similar cases even in fernet) meaning the token is invalid when it is
>> issued according to keystone. It's a long running bug.
>>
>> For the record, UUID tokens are deprecated and slated for removal in
>> the R release. I recommend reverting to using Fernet tokens sooner
>> rather than later.
>>
>> Last of all, the endpoint-filtering is generally not a great tool to
>> use. I highly recommend not using it (or encouraging the use of it),
>> it makes the catalog different depending on scope and provides zero
>> added security benefit (anyone who knows the endpoint can use it
>> anyway).
> I am spinning up a change for Pike RC (right now) to address the
> revocations occurring in the same second as the issuance of the token
> (similar to a bug with password updates, which will also be fixed).
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [qa] [keystone] Random Patrole failures related to Identity v3 Extensions API

Lance Bragstad-2
In reply to this post by Morgan Fainberg-2
Help if you actually attach the link you want to send [0].

[0] https://bugs.launchpad.net/keystone/+bug/1702211

On 08/11/2017 11:26 AM, Morgan Fainberg wrote:

> On Fri, Aug 11, 2017 at 9:25 AM, Morgan Fainberg
> <[hidden email]> wrote:
>> On Fri, Aug 11, 2017 at 8:44 AM, Felipe Monteiro
>> <[hidden email]> wrote:
>>> Patrole tests occasionally fail while executing tests that test the
>>> Identity v3 Extensions API [0]. Previously, this was not the case when
>>> we used Fernet tokens and used a time.sleep(1) to allow for
>>> role-switching to work correctly. However, we recently changed over to
>>> UUID tokens in the Patrole gates to avoid doing a time.sleep(1), as a
>>> time efficiency change. Ordinarily -- for well over 500 or so tests --
>>> this approach works successfully, with the exception of what appears
>>> to be *random* v3 API extension tests [1][2] (random means different
>>> tests pass or fail randomly across separate test runs).
>>>
>>> While there are a few solutions that come to mind on how to solve this
>>> Patrole-side (like re-introducing a time.sleep() for specific APIs or
>>> even avoiding role-switching altogether which is not as
>>> straightforward as it sounds), we would still not understand *why* the
>>> issue is happening in the first place. Is it a data-race condition?
>>> Something specific to the identity v3 extensions API? A potential bug
>>> or intended behavior somewhere?
>>>
>>> [0] https://developer.openstack.org/api-ref/identity/v3-ext/
>>> [1] http://status.openstack.org/openstack-health/#/test/patrole_tempest_plugin.tests.api.identity.v3.test_ep_filter_groups_rbac.EndpointFilterGroupsV3RbacTest.test_create_endpoint_group
>>> [2] http://logs.openstack.org/41/490641/3/gate/gate-tempest-dsvm-patrole-py35-member-ubuntu-xenial/be95da4/console.html#_2017-08-11_14_47_57_515906
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: [hidden email]?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> Are your tests causing token revocations? if so, there is a case where
>> a revocation event is issued in the same second as a token (we've seen
>> similar cases even in fernet) meaning the token is invalid when it is
>> issued according to keystone. It's a long running bug.
>>
>> For the record, UUID tokens are deprecated and slated for removal in
>> the R release. I recommend reverting to using Fernet tokens sooner
>> rather than later.
>>
>> Last of all, the endpoint-filtering is generally not a great tool to
>> use. I highly recommend not using it (or encouraging the use of it),
>> it makes the catalog different depending on scope and provides zero
>> added security benefit (anyone who knows the endpoint can use it
>> anyway).
> I am spinning up a change for Pike RC (right now) to address the
> revocations occurring in the same second as the issuance of the token
> (similar to a bug with password updates, which will also be fixed).
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

signature.asc (849 bytes) Download Attachment
Loading...