blueprint for 'spare-hosts'

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

blueprint for 'spare-hosts'

Matthew Sherborne
In summary, when one sets up a nova install, they can set some of the machines to spare.

Spare means disabled plus the guarantee that there are no customer vms running on there.

It differs from maintenance mode in that setting a spare that has VMs on it, will just fail, rather than cause a migration.

Spare machines can be used for quick chassis swaps or can be enabled to receive an emergency migration, or when an end user needs more space than was initially planned.

Having 'spare' machines in your install a useful safety buffer to let you know when to start adding more.


----

Please comment on implementation ideas, additional features, and perceived usefulness.

Many Thanks,
Matthew Sherborne

_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: blueprint for 'spare-hosts'

Sam Morrison
Hi Matthew,

I think you should be able to do all this with host aggregates?

Note: availability zones are no longer tied to aggregates in Grizzly. The docs need to be updated.


Cheers,
Sam


On 23/03/2013, at 3:27 AM, Matthew Sherborne <[hidden email]> wrote:

In summary, when one sets up a nova install, they can set some of the machines to spare.

Spare means disabled plus the guarantee that there are no customer vms running on there.

It differs from maintenance mode in that setting a spare that has VMs on it, will just fail, rather than cause a migration.

Spare machines can be used for quick chassis swaps or can be enabled to receive an emergency migration, or when an end user needs more space than was initially planned.

Having 'spare' machines in your install a useful safety buffer to let you know when to start adding more.


----

Please comment on implementation ideas, additional features, and perceived usefulness.

Many Thanks,
Matthew Sherborne
_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: blueprint for 'spare-hosts'

Anne Bertucio
Hi Sam,
Can you add more detail about what needs to be updated to this doc bug?
Thanks,
Anne


On Sun, Mar 24, 2013 at 5:50 PM, Sam Morrison <[hidden email]> wrote:
Hi Matthew,

I think you should be able to do all this with host aggregates?

Note: availability zones are no longer tied to aggregates in Grizzly. The docs need to be updated.


Cheers,
Sam


On 23/03/2013, at 3:27 AM, Matthew Sherborne <[hidden email]> wrote:

In summary, when one sets up a nova install, they can set some of the machines to spare.

Spare means disabled plus the guarantee that there are no customer vms running on there.

It differs from maintenance mode in that setting a spare that has VMs on it, will just fail, rather than cause a migration.

Spare machines can be used for quick chassis swaps or can be enabled to receive an emergency migration, or when an end user needs more space than was initially planned.

Having 'spare' machines in your install a useful safety buffer to let you know when to start adding more.


----

Please comment on implementation ideas, additional features, and perceived usefulness.

Many Thanks,
Matthew Sherborne
_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: blueprint for 'spare-hosts'

Anne Bertucio
Ah, actually we're already tracking that change with https://bugs.launchpad.net/openstack-manuals/+bug/1107419


On Sun, Mar 24, 2013 at 7:01 PM, Anne Gentle <[hidden email]> wrote:
Hi Sam,
Can you add more detail about what needs to be updated to this doc bug?
Thanks,
Anne


On Sun, Mar 24, 2013 at 5:50 PM, Sam Morrison <[hidden email]> wrote:
Hi Matthew,

I think you should be able to do all this with host aggregates?

Note: availability zones are no longer tied to aggregates in Grizzly. The docs need to be updated.


Cheers,
Sam


On 23/03/2013, at 3:27 AM, Matthew Sherborne <[hidden email]> wrote:

In summary, when one sets up a nova install, they can set some of the machines to spare.

Spare means disabled plus the guarantee that there are no customer vms running on there.

It differs from maintenance mode in that setting a spare that has VMs on it, will just fail, rather than cause a migration.

Spare machines can be used for quick chassis swaps or can be enabled to receive an emergency migration, or when an end user needs more space than was initially planned.

Having 'spare' machines in your install a useful safety buffer to let you know when to start adding more.


----

Please comment on implementation ideas, additional features, and perceived usefulness.

Many Thanks,
Matthew Sherborne
_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: blueprint for 'spare-hosts'

Day, Phil
In reply to this post by Matthew Sherborne

Hi Matthew,

 

Doesn’t just setting “enable_new_services=False” give you what you want ?   

 

We run this on all of our servers so that any newly installed systems can be checked before they are allowed to start hosting Instances.    Couple that with a change we’ve been trying to get through for a while now to add a short text field to the services to capture what they are disabled (https://review.openstack.org/#/c/10877/  ) and you could just set the reason field to “Spare” on the ones you decide not to enable.

 

Cheers,

Phil

 

From: Matthew Sherborne [mailto:[hidden email]]
Sent: 22 March 2013 16:28
To: OpenStack Development Mailing List
Subject: [openstack-dev] blueprint for 'spare-hosts'

 

In summary, when one sets up a nova install, they can set some of the machines to spare.

Spare means disabled plus the guarantee that there are no customer vms running on there.

It differs from maintenance mode in that setting a spare that has VMs on it, will just fail, rather than cause a migration.

 

Spare machines can be used for quick chassis swaps or can be enabled to receive an emergency migration, or when an end user needs more space than was initially planned.

Having 'spare' machines in your install a useful safety buffer to let you know when to start adding more.


----

Please comment on implementation ideas, additional features, and perceived usefulness.

Many Thanks,

Matthew Sherborne


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: blueprint for 'spare-hosts'

Belmiro Moreira
Hi Matthew,
for "spare" compute nodes we are using what Phil said.
The difference that I see in your blueprint is that a "spare host will be guaranteed not to have any VMs".
An this is not true if we disable the nova-compute service.

Instead, couldn't we have a flag in nova that guarantees (or not) this behaviour in disabled compute nodes?

cheers,
Belmiro

On Mar 25, 2013, at 11:19 AM, "Day, Phil" <[hidden email]> wrote:

> Hi Matthew,
>  
> Doesn’t just setting “enable_new_services=False” give you what you want ?  
>  
> We run this on all of our servers so that any newly installed systems can be checked before they are allowed to start hosting Instances.    Couple that with a change we’ve been trying to get through for a while now to add a short text field to the services to capture what they are disabled (https://review.openstack.org/#/c/10877/  ) and you could just set the reason field to “Spare” on the ones you decide not to enable.
>  
> Cheers,
> Phil
>  
> From: Matthew Sherborne [mailto:[hidden email]]
> Sent: 22 March 2013 16:28
> To: OpenStack Development Mailing List
> Subject: [openstack-dev] blueprint for 'spare-hosts'
>  
> All comments appreciated.
>
> https://blueprints.launchpad.net/nova/+spec/spare-hosts
>
> In summary, when one sets up a nova install, they can set some of the machines to spare.
>
> Spare means disabled plus the guarantee that there are no customer vms running on there.
>
> It differs from maintenance mode in that setting a spare that has VMs on it, will just fail, rather than cause a migration.
>  
> Spare machines can be used for quick chassis swaps or can be enabled to receive an emergency migration, or when an end user needs more space than was initially planned.
>
> Having 'spare' machines in your install a useful safety buffer to let you know when to start adding more.
>
> https://wiki.openstack.org/wiki/Blueprint-spare-hosts#Possible_later_additions
>
> ----
>
> Please comment on implementation ideas, additional features, and perceived usefulness.
>
> Many Thanks,
> Matthew Sherborne
> _______________________________________________
> OpenStack-dev mailing list
> [hidden email]
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: blueprint for 'spare-hosts'

Matthew Sherborne
Hi Phil et al,

Thanks for the great feedback.

I can see the 'reason for disabled' patch as a usable work around. I'm sad that it's a hassle to get the patch in, but I'm glad to see I'm not the only one that happens to. I'm still getting used to the openstack commit process. It's a lot more political than I've ever seen before, but it is producing high quality code for sure.

The availability_zone scheduler filter also looks useful as a replacement.

The only concern I have if we were to use those, would be the race condition, where a migration starts towards the host, then you set it to disabled. I'm not sure if that's possible, but I can imagine a sysadmin setting it a host to spare and walking away, then later using it for a chassis swap, then some customer saying, 'dude! mybigsite.com is down'.

I'll investigate the possibility of the race condition happening, but if anyone already knows the answer, you could save me some work ;)

Many Thanks,
Matthew Sherborne


On Mon, Mar 25, 2013 at 8:54 PM, Belmiro Moreira <[hidden email]> wrote:
Hi Matthew,
for "spare" compute nodes we are using what Phil said.
The difference that I see in your blueprint is that a "spare host will be guaranteed not to have any VMs".
An this is not true if we disable the nova-compute service.

Instead, couldn't we have a flag in nova that guarantees (or not) this behaviour in disabled compute nodes?

cheers,
Belmiro

On Mar 25, 2013, at 11:19 AM, "Day, Phil" <[hidden email]> wrote:

> Hi Matthew,
>
> Doesn’t just setting “enable_new_services=False” give you what you want ?
>
> We run this on all of our servers so that any newly installed systems can be checked before they are allowed to start hosting Instances.    Couple that with a change we’ve been trying to get through for a while now to add a short text field to the services to capture what they are disabled (https://review.openstack.org/#/c/10877/  ) and you could just set the reason field to “Spare” on the ones you decide not to enable.
>
> Cheers,
> Phil
>
> From: Matthew Sherborne [mailto:[hidden email]]
> Sent: 22 March 2013 16:28
> To: OpenStack Development Mailing List
> Subject: [openstack-dev] blueprint for 'spare-hosts'
>
> All comments appreciated.
>
> https://blueprints.launchpad.net/nova/+spec/spare-hosts
>
> In summary, when one sets up a nova install, they can set some of the machines to spare.
>
> Spare means disabled plus the guarantee that there are no customer vms running on there.
>
> It differs from maintenance mode in that setting a spare that has VMs on it, will just fail, rather than cause a migration.
>
> Spare machines can be used for quick chassis swaps or can be enabled to receive an emergency migration, or when an end user needs more space than was initially planned.
>
> Having 'spare' machines in your install a useful safety buffer to let you know when to start adding more.
>
> https://wiki.openstack.org/wiki/Blueprint-spare-hosts#Possible_later_additions
>
> ----
>
> Please comment on implementation ideas, additional features, and perceived usefulness.
>
> Many Thanks,
> Matthew Sherborne
> _______________________________________________
> OpenStack-dev mailing list
> [hidden email]
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: blueprint for 'spare-hosts'

Rosa, Andrea (HP Cloud Services)
Hi Matthew,

>I can see the 'reason for disabled' patch as a usable work around. I'm sad that it's a hassle to get the patch in, but I'm glad to see I'm not the only one that happens to.

You can find here[1] the new blueprint for adding the "reason" field for disabling services. This new version is based on the API extension os-services and not on nova-manage script.
It's taking time to get finished for several reasons one of them is there are other tasks/bugs for that extensions to get merged before going ahead with the new feature.

Hope this helps.
Regards
--
Andrea Rosa

[1] https://blueprints.launchpad.net/nova/+spec/record-reason-for-disabling-service
_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev