[Nova] too many tokens

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[Nova] too many tokens

Ala Rezmerita
Hi everyone,

I have a question regarding too many token generation in nova when using quantumclient (also related to bug reports https://bugs.launchpad.net/nova/+bug/1192383 + https://bugs.launchpad.net/nova-project/+bug/1191159)

For instance during the periodic task  heal_instance_info_cache  (every 60s) nova calls quantum API method  get_instance_nw_info that calls _build_network_info_model (backtrace at the end of the mail). 

During the execution of this method,  4 quantum clients intances are created (all of them use the same context object) and for each of them a new token is generated.  

Is it possible to change this behavior by updating the context.auth_token property the first time a quantumclient for a given context is created (so that the same token will be reused among the 4 client instances) ?  Is there some security issue that can appear?

Thanks

Ala Rezmerita
Cloudwatt

The backtrace :

  /usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py(194)main()
-> result = function(*args, **kwargs)
  /opt/stack/nova/nova/openstack/common/loopingcall.py(125)_inner()
-> idle = self.f(*self.args, **self.kw)
  /opt/stack/nova/nova/service.py(283)periodic_tasks()
-> return self.manager.periodic_tasks(ctxt, raise_on_error=raise_on_error)
  /opt/stack/nova/nova/manager.py(100)periodic_tasks()
-> return self.run_periodic_tasks(context, raise_on_error=raise_on_error)
  /opt/stack/nova/nova/openstack/common/periodic_task.py(179)run_periodic_tasks()
-> task(self, context)
  /opt/stack/nova/nova/compute/manager.py(3654)_heal_instance_info_cache()
-> self._get_instance_nw_info(context, instance)
  /opt/stack/nova/nova/compute/manager.py(767)_get_instance_nw_info()
-> instance, conductor_api=self.conductor_api)
  /opt/stack/nova/nova/network/quantumv2/api.py(367)get_instance_nw_info()
-> result = self._get_instance_nw_info(context, instance, networks)
  /opt/stack/nova/nova/network/quantumv2/api.py(375)_get_instance_nw_info()
-> nw_info = self._build_network_info_model(context, instance, networks)
  /opt/stack/nova/nova/network/quantumv2/api.py(840)_build_network_info_model()
-> client = quantumv2.get_client(context, admin=True)
> /opt/stack/nova/nova/network/quantumv2/__init__.py(67)get_client()


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Nova] too many tokens

Matt Riedemann
For some history, there was an attempt at consolidating some of this here:

https://github.com/openstack/nova/commit/dd9c27f999221001bae9faa03571645824d2a681

But that caused some issues and was reverted here:

https://github.com/openstack/nova/commit/ee5d9ae8d376e41e852b06488e922400cf69b4ac



Thanks,

MATT RIEDEMANN
Advisory Software Engineer
Cloud Solutions and OpenStack Development

Phone: 1-507-253-7622 | Mobile: 1-507-990-1889
E-mail:
mriedem@...
IBM

3605 Hwy 52 N
Rochester, MN 55901-1407
United States





From:        Ala Rezmerita <[hidden email]>
To:        OpenStack Development Mailing List <[hidden email]>,
Cc:        [hidden email], [hidden email]
Date:        07/03/2013 11:26 AM
Subject:        [openstack-dev] [Nova] too many tokens




Hi everyone,

I have a question regarding too many token generation in nova when using quantumclient (also related to bug reports https://bugs.launchpad.net/nova/+bug/1192383 + https://bugs.launchpad.net/nova-project/+bug/1191159)

For instance during the periodic task  heal_instance_info_cache  (every 60s) nova calls quantum API method  get_instance_nw_info that calls _build_network_info_model (backtrace at the end of the mail). 

During the execution of this method,  4 quantum clients intances are created (all of them use the same context object) and for each of them a new token is generated.  

Is it possible to change this behavior by updating the context.auth_token property the first time a quantumclient for a given context is created (so that the same token will be reused among the 4 client instances) ?  Is there some security issue that can appear?

Thanks

Ala Rezmerita
Cloudwatt

The backtrace :

  /usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py(194)main()
-> result = function(*args, **kwargs)
  /opt/stack/nova/nova/openstack/common/loopingcall.py(125)_inner()
-> idle = self.f(*self.args, **
self.kw)
  /opt/stack/nova/nova/service.py(283)periodic_tasks()
-> return self.manager.periodic_tasks(ctxt, raise_on_error=raise_on_error)
  /opt/stack/nova/nova/manager.py(100)periodic_tasks()
-> return self.run_periodic_tasks(context, raise_on_error=raise_on_error)
  /opt/stack/nova/nova/openstack/common/periodic_task.py(179)run_periodic_tasks()
-> task(self, context)
  /opt/stack/nova/nova/compute/manager.py(3654)_heal_instance_info_cache()
-> self._get_instance_nw_info(context, instance)
  /opt/stack/nova/nova/compute/manager.py(767)_get_instance_nw_info()
-> instance, conductor_api=self.conductor_api)
  /opt/stack/nova/nova/network/quantumv2/api.py(367)get_instance_nw_info()
-> result = self._get_instance_nw_info(context, instance, networks)
  /opt/stack/nova/nova/network/quantumv2/api.py(375)_get_instance_nw_info()
-> nw_info = self._build_network_info_model(context, instance, networks)
  /opt/stack/nova/nova/network/quantumv2/api.py(840)_build_network_info_model()
-> client = quantumv2.get_client(context, admin=True)
> /opt/stack/nova/nova/network/quantumv2/__init__.py(67)get_client()
_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply | Threaded
Open this post in threaded view
|

Re: [Nova] too many tokens

Ala Rezmerita

Thanks Matt for the pointers. I was already aware of this cache mechanism.

However my approach is different: I do not want to cache clients, but simply

update the context with the token that was obtained. And if the same context

is reused, no new token will be generated. 

Ala

 



2013/7/3 Matt Riedemann <[hidden email]>
For some history, there was an attempt at consolidating some of this here:

https://github.com/openstack/nova/commit/dd9c27f999221001bae9faa03571645824d2a681

But that caused some issues and was reverted here:

https://github.com/openstack/nova/commit/ee5d9ae8d376e41e852b06488e922400cf69b4ac



Thanks,

MATT RIEDEMANN
Advisory Software Engineer
Cloud Solutions and OpenStack Development

Phone: <a href="tel:1-507-253-7622" value="+15072537622" target="_blank">1-507-253-7622 | Mobile: <a href="tel:1-507-990-1889" value="+15079901889" target="_blank">1-507-990-1889
E-mail:
[hidden email]
IBM

3605 Hwy 52 N
Rochester, MN 55901-1407
United States





From:        Ala Rezmerita <[hidden email]>
To:        OpenStack Development Mailing List <[hidden email]>,
Cc:        [hidden email], [hidden email]
Date:        07/03/2013 11:26 AM
Subject:        [openstack-dev] [Nova] too many tokens




Hi everyone,

I have a question regarding too many token generation in nova when using quantumclient (also related to bug reports https://bugs.launchpad.net/nova/+bug/1192383 + https://bugs.launchpad.net/nova-project/+bug/1191159)

For instance during the periodic task  heal_instance_info_cache  (every 60s) nova calls quantum API method  get_instance_nw_info that calls _build_network_info_model (backtrace at the end of the mail). 

During the execution of this method,  4 quantum clients intances are created (all of them use the same context object) and for each of them a new token is generated.  

Is it possible to change this behavior by updating the context.auth_token property the first time a quantumclient for a given context is created (so that the same token will be reused among the 4 client instances) ?  Is there some security issue that can appear?

Thanks

Ala Rezmerita
Cloudwatt

The backtrace :

  /usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py(194)main()
-> result = function(*args, **kwargs)
  /opt/stack/nova/nova/openstack/common/loopingcall.py(125)_inner()
-> idle = self.f(*self.args, **
self.kw
)

  /opt/stack/nova/nova/service.py(283)periodic_tasks()
-> return self.manager.periodic_tasks(ctxt, raise_on_error=raise_on_error)
  /opt/stack/nova/nova/manager.py(100)periodic_tasks()
-> return self.run_periodic_tasks(context, raise_on_error=raise_on_error)
  /opt/stack/nova/nova/openstack/common/periodic_task.py(179)run_periodic_tasks()
-> task(self, context)
  /opt/stack/nova/nova/compute/manager.py(3654)_heal_instance_info_cache()
-> self._get_instance_nw_info(context, instance)
  /opt/stack/nova/nova/compute/manager.py(767)_get_instance_nw_info()
-> instance, conductor_api=self.conductor_api)
  /opt/stack/nova/nova/network/quantumv2/api.py(367)get_instance_nw_info()
-> result = self._get_instance_nw_info(context, instance, networks)
  /opt/stack/nova/nova/network/quantumv2/api.py(375)_get_instance_nw_info()
-> nw_info = self._build_network_info_model(context, instance, networks)
  /opt/stack/nova/nova/network/quantumv2/api.py(840)_build_network_info_model()
-> client = quantumv2.get_client(context, admin=True)
> /opt/stack/nova/nova/network/quantumv2/__init__.py(67)get_client()
_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Ala Rezmerita
Software Engineer
CloudWatt
Tel : (+33) 06 77 43 23 91

_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Nova] too many tokens

Joe Gordon-2
In reply to this post by Ala Rezmerita


On Jul 3, 2013 5:22 PM, "Ala Rezmerita" <[hidden email]> wrote:
>
> Hi everyone,
>
> I have a question regarding too many token generation in nova when using quantumclient (also related to bug reports https://bugs.launchpad.net/nova/+bug/1192383 + https://bugs.launchpad.net/nova-project/+bug/1191159)
>
> For instance during the periodic task  heal_instance_info_cache  (every 60s) nova calls quantum API method  get_instance_nw_info that calls _build_network_info_model (backtrace at the end of the mail). 
>
> During the execution of this method,  4 quantum clients intances are created (all of them use the same context object) and for each of them a new token is generated.  
>
> Is it possible to change this behavior by updating the context.auth_token property the first time a quantumclient for a given context is created (so that the same token will be reused among the 4 client instances) ?  Is there some security issue that can appear?

We have this issue elsewhere, too many tokens being generated because the clients aren't caching the tokens across client instances.  So why not try to fix that instead, as solving that once and for all will fix many other use cases as well?

I know some attempts have been made at this but clearly they are not working for most people.

>
> Thanks
>
> Ala Rezmerita
> Cloudwatt
>
> The backtrace :
>
>   /usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py(194)main()
> -> result = function(*args, **kwargs)
>   /opt/stack/nova/nova/openstack/common/loopingcall.py(125)_inner()
> -> idle = self.f(*self.args, **self.kw)
>   /opt/stack/nova/nova/service.py(283)periodic_tasks()
> -> return self.manager.periodic_tasks(ctxt, raise_on_error=raise_on_error)
>   /opt/stack/nova/nova/manager.py(100)periodic_tasks()
> -> return self.run_periodic_tasks(context, raise_on_error=raise_on_error)
>   /opt/stack/nova/nova/openstack/common/periodic_task.py(179)run_periodic_tasks()
> -> task(self, context)
>   /opt/stack/nova/nova/compute/manager.py(3654)_heal_instance_info_cache()
> -> self._get_instance_nw_info(context, instance)
>   /opt/stack/nova/nova/compute/manager.py(767)_get_instance_nw_info()
> -> instance, conductor_api=self.conductor_api)
>   /opt/stack/nova/nova/network/quantumv2/api.py(367)get_instance_nw_info()
> -> result = self._get_instance_nw_info(context, instance, networks)
>   /opt/stack/nova/nova/network/quantumv2/api.py(375)_get_instance_nw_info()
> -> nw_info = self._build_network_info_model(context, instance, networks)
>   /opt/stack/nova/nova/network/quantumv2/api.py(840)_build_network_info_model()
> -> client = quantumv2.get_client(context, admin=True)
> > /opt/stack/nova/nova/network/quantumv2/__init__.py(67)get_client()
>
> _______________________________________________
> OpenStack-dev mailing list
> [hidden email]
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev