[nova][cinder][neutron] Cross-cell cold migration

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

[nova][cinder][neutron] Cross-cell cold migration

Matt Riedemann-3
Hi everyone,

I have started an etherpad for cells topics at the Stein PTG [1]. The
main issue in there right now is dealing with cross-cell cold migration
in nova.

At a high level, I am going off these requirements:

* Cells can shard across flavors (and hardware type) so operators would
like to move users off the old flavors/hardware (old cell) to new
flavors in a new cell.

* There is network isolation between compute hosts in different cells,
so no ssh'ing the disk around like we do today. But the image service is
global to all cells.

Based on this, for the initial support for cross-cell cold migration, I
am proposing that we leverage something like shelve offload/unshelve
masquerading as resize. We shelve offload from the source cell and
unshelve in the target cell. This should work for both volume-backed and
non-volume-backed servers (we use snapshots for shelved offloaded
non-volume-backed servers).

There are, of course, some complications. The main ones that I need help
with right now are what happens with volumes and ports attached to the
server. Today we detach from the source and attach at the target, but
that's assuming the storage backend and network are available to both
hosts involved in the move of the server. Will that be the case across
cells? I am assuming that depends on the network topology (are routed
networks being used?) and storage backend (routed storage?). If the
network and/or storage backend are not available across cells, how do we
migrate volumes and ports? Cinder has a volume migrate API for admins
but I do not know how nova would know the proper affinity per-cell to
migrate the volume to the proper host (cinder does not have a routed
storage concept like routed provider networks in neutron, correct?). And
as far as I know, there is no such thing as port migration in Neutron.

Could Placement help with the volume/port migration stuff? Neutron
routed provider networks rely on placement aggregates to schedule the VM
to a compute host in the same network segment as the port used to create
the VM, however, if that segment does not span cells we are kind of
stuck, correct?

To summarize the issues as I see them (today):

* How to deal with the targeted cell during scheduling? This is so we
can even get out of the source cell in nova.

* How does the API deal with the same instance being in two DBs at the
same time during the move?

* How to handle revert resize?

* How are volumes and ports handled?

I can get feedback from my company's operators based on what their
deployment will look like for this, but that does not mean it will work
for others, so I need as much feedback from operators, especially those
running with multiple cells today, as possible. Thanks in advance.

[1] https://etherpad.openstack.org/p/nova-ptg-stein-cells

--

Thanks,

Matt

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [nova][cinder][neutron] Cross-cell cold migration

Sam Morrison
I think in our case we’d only migrate between cells if we know the network and storage is accessible and would never do it if not.
Thinking moving from old to new hardware at a cell level.

If storage and network isn’t available ideally it would fail at the api request.

There is also ceph backed instances and so this is also something to take into account which nova would be responsible for.

I’ll be in Denver so we can discuss more there too.

Cheers,
Sam





> On 23 Aug 2018, at 11:23 am, Matt Riedemann <[hidden email]> wrote:
>
> Hi everyone,
>
> I have started an etherpad for cells topics at the Stein PTG [1]. The main issue in there right now is dealing with cross-cell cold migration in nova.
>
> At a high level, I am going off these requirements:
>
> * Cells can shard across flavors (and hardware type) so operators would like to move users off the old flavors/hardware (old cell) to new flavors in a new cell.
>
> * There is network isolation between compute hosts in different cells, so no ssh'ing the disk around like we do today. But the image service is global to all cells.
>
> Based on this, for the initial support for cross-cell cold migration, I am proposing that we leverage something like shelve offload/unshelve masquerading as resize. We shelve offload from the source cell and unshelve in the target cell. This should work for both volume-backed and non-volume-backed servers (we use snapshots for shelved offloaded non-volume-backed servers).
>
> There are, of course, some complications. The main ones that I need help with right now are what happens with volumes and ports attached to the server. Today we detach from the source and attach at the target, but that's assuming the storage backend and network are available to both hosts involved in the move of the server. Will that be the case across cells? I am assuming that depends on the network topology (are routed networks being used?) and storage backend (routed storage?). If the network and/or storage backend are not available across cells, how do we migrate volumes and ports? Cinder has a volume migrate API for admins but I do not know how nova would know the proper affinity per-cell to migrate the volume to the proper host (cinder does not have a routed storage concept like routed provider networks in neutron, correct?). And as far as I know, there is no such thing as port migration in Neutron.
>
> Could Placement help with the volume/port migration stuff? Neutron routed provider networks rely on placement aggregates to schedule the VM to a compute host in the same network segment as the port used to create the VM, however, if that segment does not span cells we are kind of stuck, correct?
>
> To summarize the issues as I see them (today):
>
> * How to deal with the targeted cell during scheduling? This is so we can even get out of the source cell in nova.
>
> * How does the API deal with the same instance being in two DBs at the same time during the move?
>
> * How to handle revert resize?
>
> * How are volumes and ports handled?
>
> I can get feedback from my company's operators based on what their deployment will look like for this, but that does not mean it will work for others, so I need as much feedback from operators, especially those running with multiple cells today, as possible. Thanks in advance.
>
> [1] https://etherpad.openstack.org/p/nova-ptg-stein-cells
>
> --
>
> Thanks,
>
> Matt


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [nova][cinder][neutron] Cross-cell cold migration

Gorka Eguileor
In reply to this post by Matt Riedemann-3
On 22/08, Matt Riedemann wrote:

> Hi everyone,
>
> I have started an etherpad for cells topics at the Stein PTG [1]. The main
> issue in there right now is dealing with cross-cell cold migration in nova.
>
> At a high level, I am going off these requirements:
>
> * Cells can shard across flavors (and hardware type) so operators would like
> to move users off the old flavors/hardware (old cell) to new flavors in a
> new cell.
>
> * There is network isolation between compute hosts in different cells, so no
> ssh'ing the disk around like we do today. But the image service is global to
> all cells.
>
> Based on this, for the initial support for cross-cell cold migration, I am
> proposing that we leverage something like shelve offload/unshelve
> masquerading as resize. We shelve offload from the source cell and unshelve
> in the target cell. This should work for both volume-backed and
> non-volume-backed servers (we use snapshots for shelved offloaded
> non-volume-backed servers).
>
> There are, of course, some complications. The main ones that I need help
> with right now are what happens with volumes and ports attached to the
> server. Today we detach from the source and attach at the target, but that's
> assuming the storage backend and network are available to both hosts
> involved in the move of the server. Will that be the case across cells? I am
> assuming that depends on the network topology (are routed networks being
> used?) and storage backend (routed storage?). If the network and/or storage
> backend are not available across cells, how do we migrate volumes and ports?
> Cinder has a volume migrate API for admins but I do not know how nova would
> know the proper affinity per-cell to migrate the volume to the proper host
> (cinder does not have a routed storage concept like routed provider networks
> in neutron, correct?). And as far as I know, there is no such thing as port
> migration in Neutron.

Hi Matt,

I think Nova should never have to rely on Cinder's hosts/backends
information to do migrations or any other operation.

In this case even if Nova had that info, it wouldn't be the solution.
Cinder would reject migrations if there's an incompatibility on the
Volume Type (AZ, Referenced backend, capabilities...)

I don't know anything about Nova cells, so I don't know the specifics of
how we could do the mapping between them and Cinder backends, but
considering the limited range of possibilities in Cinder I would say we
only have Volume Types and AZs to work a solution.

>
> Could Placement help with the volume/port migration stuff? Neutron routed
> provider networks rely on placement aggregates to schedule the VM to a
> compute host in the same network segment as the port used to create the VM,
> however, if that segment does not span cells we are kind of stuck, correct?
>

I don't know how the Nova Placement works, but it could hold an
equivalency mapping of volume types to cells as in:

 Cell#1        Cell#2

VolTypeA <--> VolTypeD
VolTypeB <--> VolTypeE
VolTypeC <--> VolTypeF

Then it could do volume retypes (allowing migration) and that would
properly move the volumes from one backend to another.

Cheers,
Gorka.


> To summarize the issues as I see them (today):
>
> * How to deal with the targeted cell during scheduling? This is so we can
> even get out of the source cell in nova.
>
> * How does the API deal with the same instance being in two DBs at the same
> time during the move?
>
> * How to handle revert resize?
>
> * How are volumes and ports handled?
>
> I can get feedback from my company's operators based on what their
> deployment will look like for this, but that does not mean it will work for
> others, so I need as much feedback from operators, especially those running
> with multiple cells today, as possible. Thanks in advance.
>
> [1] https://etherpad.openstack.org/p/nova-ptg-stein-cells
>
> --
>
> Thanks,
>
> Matt
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [nova][cinder][neutron] Cross-cell cold migration

Dan Smith-2
> I think Nova should never have to rely on Cinder's hosts/backends
> information to do migrations or any other operation.
>
> In this case even if Nova had that info, it wouldn't be the solution.
> Cinder would reject migrations if there's an incompatibility on the
> Volume Type (AZ, Referenced backend, capabilities...)

I think I'm missing a bunch of cinder knowledge required to fully grok
this situation and probably need to do some reading. Is there some
reason that a volume type can't exist in multiple backends or something?
I guess I think of volume type as flavor, and the same definition in two
places would be interchangeable -- is that not the case?

> I don't know anything about Nova cells, so I don't know the specifics of
> how we could do the mapping between them and Cinder backends, but
> considering the limited range of possibilities in Cinder I would say we
> only have Volume Types and AZs to work a solution.

I think the only mapping we need is affinity or distance. The point of
needing to migrate the volume would purely be because moving cells
likely means you moved physically farther away from where you were,
potentially with different storage connections and networking. It
doesn't *have* to mean that, but I think in reality it would. So the
question I think Matt is looking to answer here is "how do we move an
instance from a DC in building A to building C and make sure the
volume gets moved to some storage local in the new building so we're
not just transiting back to the original home for no reason?"

Does that explanation help or are you saying that's fundamentally hard
to do/orchestrate?

Fundamentally, the cells thing doesn't even need to be part of the
discussion, as the same rules would apply if we're just doing a normal
migration but need to make sure that storage remains affined to compute.

> I don't know how the Nova Placement works, but it could hold an
> equivalency mapping of volume types to cells as in:
>
>  Cell#1        Cell#2
>
> VolTypeA <--> VolTypeD
> VolTypeB <--> VolTypeE
> VolTypeC <--> VolTypeF
>
> Then it could do volume retypes (allowing migration) and that would
> properly move the volumes from one backend to another.

The only way I can think that we could do this in placement would be if
volume types were resource providers and we assigned them traits that
had special meaning to nova indicating equivalence. Several of the words
in that sentence are likely to freak out placement people, myself
included :)

So is the concern just that we need to know what volume types in one
backend map to those in another so that when we do the migration we know
what to ask for? Is "they are the same name" not enough? Going back to
the flavor analogy, you could kinda compare two flavor definitions and
have a good idea if they're equivalent or not...

--Dan

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [nova][cinder][neutron] Cross-cell cold migration

Sean McGinnis
In reply to this post by Matt Riedemann-3
On Wed, Aug 22, 2018 at 08:23:41PM -0500, Matt Riedemann wrote:

> Hi everyone,
>
> I have started an etherpad for cells topics at the Stein PTG [1]. The main
> issue in there right now is dealing with cross-cell cold migration in nova.
>
> At a high level, I am going off these requirements:
>
> * Cells can shard across flavors (and hardware type) so operators would like
> to move users off the old flavors/hardware (old cell) to new flavors in a
> new cell.
>
> * There is network isolation between compute hosts in different cells, so no
> ssh'ing the disk around like we do today. But the image service is global to
> all cells.
>
> Based on this, for the initial support for cross-cell cold migration, I am
> proposing that we leverage something like shelve offload/unshelve
> masquerading as resize. We shelve offload from the source cell and unshelve
> in the target cell. This should work for both volume-backed and
> non-volume-backed servers (we use snapshots for shelved offloaded
> non-volume-backed servers).
>
> There are, of course, some complications. The main ones that I need help
> with right now are what happens with volumes and ports attached to the
> server. Today we detach from the source and attach at the target, but that's
> assuming the storage backend and network are available to both hosts
> involved in the move of the server. Will that be the case across cells? I am
> assuming that depends on the network topology (are routed networks being
> used?) and storage backend (routed storage?). If the network and/or storage
> backend are not available across cells, how do we migrate volumes and ports?
> Cinder has a volume migrate API for admins but I do not know how nova would
> know the proper affinity per-cell to migrate the volume to the proper host
> (cinder does not have a routed storage concept like routed provider networks
> in neutron, correct?). And as far as I know, there is no such thing as port
> migration in Neutron.
>

Just speaking to iSCSI storage, I know some deployments do not route their
storage traffic. If this is the case, then both cells would need to have access
to the same subnet to still access the volume.

I'm also referring to the case where the migration is from one compute host to
another compute host, and not from one storage backend to another storage
backend.

I haven't gone through the workflow, but I thought shelve/unshelve could detach
the volume on shelving and reattach it on unshelve. In that workflow, assuming
the networking is in place to provide the connectivity, the nova compute host
would be connecting to the volume just like any other attach and should work
fine. The unknown or tricky part is making sure that there is the network
connectivity or routing in place for the compute host to be able to log in to
the storage target.

If it's the other scenario mentioned where the volume needs to be migrated from
one storage backend to another storage backend, then that may require a little
more work. The volume would need to be retype'd or migrated (storage migration)
from the original backend to the new backend.

Again, in this scenario at some point there needs to be network connectivity
between cells to copy over that data.

There is no storage-offloaded migration in this situation, so Cinder can't
currently optimize how that data gets from the original volume backend to the
new one. It would require a host copy of all the data on the volume (an often
slow and expensive operation) and it would require that the host doing the data
copy has access to both the original backend and then new backend.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [nova][cinder][neutron] Cross-cell cold migration

Gorka Eguileor
In reply to this post by Dan Smith-2
On 23/08, Dan Smith wrote:

> > I think Nova should never have to rely on Cinder's hosts/backends
> > information to do migrations or any other operation.
> >
> > In this case even if Nova had that info, it wouldn't be the solution.
> > Cinder would reject migrations if there's an incompatibility on the
> > Volume Type (AZ, Referenced backend, capabilities...)
>
> I think I'm missing a bunch of cinder knowledge required to fully grok
> this situation and probably need to do some reading. Is there some
> reason that a volume type can't exist in multiple backends or something?
> I guess I think of volume type as flavor, and the same definition in two
> places would be interchangeable -- is that not the case?
>

Hi,

I just know the basics of flavors, and they are kind of similar, though
I'm sure there are quite a few differences.

Sure, multiple storage arrays can meet the requirements of a Volume
Type, but then when you create the volume you don't know where it's
going to land. If your volume type is too generic you volume could land
somewhere your cell cannot reach.


> > I don't know anything about Nova cells, so I don't know the specifics of
> > how we could do the mapping between them and Cinder backends, but
> > considering the limited range of possibilities in Cinder I would say we
> > only have Volume Types and AZs to work a solution.
>
> I think the only mapping we need is affinity or distance. The point of
> needing to migrate the volume would purely be because moving cells
> likely means you moved physically farther away from where you were,
> potentially with different storage connections and networking. It
> doesn't *have* to mean that, but I think in reality it would. So the
> question I think Matt is looking to answer here is "how do we move an
> instance from a DC in building A to building C and make sure the
> volume gets moved to some storage local in the new building so we're
> not just transiting back to the original home for no reason?"
>
> Does that explanation help or are you saying that's fundamentally hard
> to do/orchestrate?
>
> Fundamentally, the cells thing doesn't even need to be part of the
> discussion, as the same rules would apply if we're just doing a normal
> migration but need to make sure that storage remains affined to compute.
>

We could probably work something out using the affinity filter, but
right now we don't have a way of doing what you need.

We could probably rework the migration to accept scheduler hints to be
used with the affinity filter and to accept calls with the host or the
hints, that way it could migrate a volume without knowing the
destination host and decide it based on affinity.

We may have to do more modifications, but it could be a way to do it.



> > I don't know how the Nova Placement works, but it could hold an
> > equivalency mapping of volume types to cells as in:
> >
> >  Cell#1        Cell#2
> >
> > VolTypeA <--> VolTypeD
> > VolTypeB <--> VolTypeE
> > VolTypeC <--> VolTypeF
> >
> > Then it could do volume retypes (allowing migration) and that would
> > properly move the volumes from one backend to another.
>
> The only way I can think that we could do this in placement would be if
> volume types were resource providers and we assigned them traits that
> had special meaning to nova indicating equivalence. Several of the words
> in that sentence are likely to freak out placement people, myself
> included :)
>
> So is the concern just that we need to know what volume types in one
> backend map to those in another so that when we do the migration we know
> what to ask for? Is "they are the same name" not enough? Going back to
> the flavor analogy, you could kinda compare two flavor definitions and
> have a good idea if they're equivalent or not...
>
> --Dan

In Cinder you don't get that from Volume Types, unless all your backends
have the same hardware and are configured exactly the same.

There can be some storage specific information there, which doesn't
correlate to anything on other hardware.  Volume types may refer to a
specific pool that has been configured in the array to use specific type
of disks.  But even the info on the type of disks is unknown to the
volume type.

I haven't checked the PTG agenda yet, but is there a meeting on this?
Because we may want to have one to try to understand the requirements
and figure out if there's a way to do it with current Cinder
functionality of if we'd need something new.

Cheers,
Gorka.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [nova][cinder][neutron] Cross-cell cold migration

Jay S Bryant-2


On 8/23/2018 12:07 PM, Gorka Eguileor wrote:

> On 23/08, Dan Smith wrote:
>>> I think Nova should never have to rely on Cinder's hosts/backends
>>> information to do migrations or any other operation.
>>>
>>> In this case even if Nova had that info, it wouldn't be the solution.
>>> Cinder would reject migrations if there's an incompatibility on the
>>> Volume Type (AZ, Referenced backend, capabilities...)
>> I think I'm missing a bunch of cinder knowledge required to fully grok
>> this situation and probably need to do some reading. Is there some
>> reason that a volume type can't exist in multiple backends or something?
>> I guess I think of volume type as flavor, and the same definition in two
>> places would be interchangeable -- is that not the case?
>>
> Hi,
>
> I just know the basics of flavors, and they are kind of similar, though
> I'm sure there are quite a few differences.
>
> Sure, multiple storage arrays can meet the requirements of a Volume
> Type, but then when you create the volume you don't know where it's
> going to land. If your volume type is too generic you volume could land
> somewhere your cell cannot reach.
>
>
>>> I don't know anything about Nova cells, so I don't know the specifics of
>>> how we could do the mapping between them and Cinder backends, but
>>> considering the limited range of possibilities in Cinder I would say we
>>> only have Volume Types and AZs to work a solution.
>> I think the only mapping we need is affinity or distance. The point of
>> needing to migrate the volume would purely be because moving cells
>> likely means you moved physically farther away from where you were,
>> potentially with different storage connections and networking. It
>> doesn't *have* to mean that, but I think in reality it would. So the
>> question I think Matt is looking to answer here is "how do we move an
>> instance from a DC in building A to building C and make sure the
>> volume gets moved to some storage local in the new building so we're
>> not just transiting back to the original home for no reason?"
>>
>> Does that explanation help or are you saying that's fundamentally hard
>> to do/orchestrate?
>>
>> Fundamentally, the cells thing doesn't even need to be part of the
>> discussion, as the same rules would apply if we're just doing a normal
>> migration but need to make sure that storage remains affined to compute.
>>
> We could probably work something out using the affinity filter, but
> right now we don't have a way of doing what you need.
>
> We could probably rework the migration to accept scheduler hints to be
> used with the affinity filter and to accept calls with the host or the
> hints, that way it could migrate a volume without knowing the
> destination host and decide it based on affinity.
>
> We may have to do more modifications, but it could be a way to do it.
>
>
>
>>> I don't know how the Nova Placement works, but it could hold an
>>> equivalency mapping of volume types to cells as in:
>>>
>>>   Cell#1        Cell#2
>>>
>>> VolTypeA <--> VolTypeD
>>> VolTypeB <--> VolTypeE
>>> VolTypeC <--> VolTypeF
>>>
>>> Then it could do volume retypes (allowing migration) and that would
>>> properly move the volumes from one backend to another.
>> The only way I can think that we could do this in placement would be if
>> volume types were resource providers and we assigned them traits that
>> had special meaning to nova indicating equivalence. Several of the words
>> in that sentence are likely to freak out placement people, myself
>> included :)
>>
>> So is the concern just that we need to know what volume types in one
>> backend map to those in another so that when we do the migration we know
>> what to ask for? Is "they are the same name" not enough? Going back to
>> the flavor analogy, you could kinda compare two flavor definitions and
>> have a good idea if they're equivalent or not...
>>
>> --Dan
> In Cinder you don't get that from Volume Types, unless all your backends
> have the same hardware and are configured exactly the same.
>
> There can be some storage specific information there, which doesn't
> correlate to anything on other hardware.  Volume types may refer to a
> specific pool that has been configured in the array to use specific type
> of disks.  But even the info on the type of disks is unknown to the
> volume type.
>
> I haven't checked the PTG agenda yet, but is there a meeting on this?
> Because we may want to have one to try to understand the requirements
> and figure out if there's a way to do it with current Cinder
> functionality of if we'd need something new.
Gorka,

I don't think that this has been put on the agenda yet.  Might be good
to add.  I don't think we have a cross project time officially planned
with Nova.  I will start that discussion with Melanie so that we can
cover the couple of cross projects subjects we have.

Jay

> Cheers,
> Gorka.
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [nova][cinder][neutron] Cross-cell cold migration

Matt Riedemann-3
In reply to this post by Sam Morrison
On 8/22/2018 9:14 PM, Sam Morrison wrote:
> I think in our case we’d only migrate between cells if we know the network and storage is accessible and would never do it if not.
> Thinking moving from old to new hardware at a cell level.

If it's done via the resize API at the top, initiated by a non-admin
user, how would you prevent it? We don't really know if we're going
across cell boundaries until the scheduler picks a host, and today we
restrict all move operations to within the same cell. But that's part of
the problem that needs addressing - how to tell the scheduler when it's
OK to get target hosts for a move from all cells rather than the cell
that the server is currently in.

>
> If storage and network isn’t available ideally it would fail at the api request.

Not sure this is something we can really tell beforehand in the API, but
maybe possible depending on whatever we come up with regarding volumes
and ports. I expect this is a whole new orchestrated task in the
(super)conductor when it happens. So while I think about using
shelve/unshelve from a compute operation standpoint, I don't want to try
and shoehorn this into existing conductor tasks.

>
> There is also ceph backed instances and so this is also something to take into account which nova would be responsible for.

Not everyone is using ceph and it's not really something the API is
aware of...at least not today - but long-term with shared storage
providers in placement we might be able to leverage this for
non-volume-backed instances, i.e. if we know the source and target host
are on the same shared storage, regardless of cell boundary, we could
just move rather than use snapshots (shelve). But I think phase1 is
easiest universally if we are using snapshots to get from cell 1 to cell 2.

>
> I’ll be in Denver so we can discuss more there too.

Awesome.

--

Thanks,

Matt

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [nova][cinder][neutron] Cross-cell cold migration

Matt Riedemann-3
In reply to this post by Gorka Eguileor
On 8/23/2018 12:07 PM, Gorka Eguileor wrote:
> I haven't checked the PTG agenda yet, but is there a meeting on this?
> Because we may want to have one to try to understand the requirements
> and figure out if there's a way to do it with current Cinder
> functionality of if we'd need something new.

I don't see any set schedule yet for topics like we've done in the past,
I'll ask Mel since time is getting short (~2 weeks out now). But I have
this as an item for discussion in the etherpad [1]. In previous PTGs, we
usually have 3 days for (mostly) vertical team stuff with Wednesday
being our big topics days split into morning and afternoon, e.g. cells
and placement, then Thursday is split into 1-2 hour cross-project
sessions, e.g. nova/cinder, nova/neutron, etc, and then Friday is the
miscellaneous everything else day for stuff on the etherpad.

[1] https://etherpad.openstack.org/p/nova-ptg-stein

--

Thanks,

Matt

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [nova][cinder][neutron] Cross-cell cold migration

Matt Riedemann-3
In reply to this post by Sean McGinnis
On 8/23/2018 10:22 AM, Sean McGinnis wrote:
> I haven't gone through the workflow, but I thought shelve/unshelve could detach
> the volume on shelving and reattach it on unshelve. In that workflow, assuming
> the networking is in place to provide the connectivity, the nova compute host
> would be connecting to the volume just like any other attach and should work
> fine. The unknown or tricky part is making sure that there is the network
> connectivity or routing in place for the compute host to be able to log in to
> the storage target.

Yeah that's also why I like shelve/unshelve as a start since it's doing
volume detach from the source host in the source cell and volume attach
to the target host in the target cell.

Host aggregates in Nova, as a grouping concept, are not restricted to
cells at all, so you could have hosts in the same aggregate which span
cells, so I'd think that's what operators would be doing if they have
network/storage spanning multiple cells. Having said that, host
aggregates are not exposed to non-admin end users, so again, if we rely
on a normal user to do this move operation via resize, the only way we
can restrict the instance to another host in the same aggregate is via
availability zones, which is the user-facing aggregate construct in
nova. I know Sam would care about this because NeCTAR sets
[cinder]/cross_az_attach=False in nova.conf so servers/volumes are
restricted to the same AZ, but that's not the default, and specifying an
AZ when you create a server is not required (although there is a config
option in nova which allows operators to define a default AZ for the
instance if the user didn't specify one).

Anyway, my point is, there are a lot of "ifs" if it's not an
operator/admin explicitly telling nova where to send the server if it's
moving across cells.

>
> If it's the other scenario mentioned where the volume needs to be migrated from
> one storage backend to another storage backend, then that may require a little
> more work. The volume would need to be retype'd or migrated (storage migration)
> from the original backend to the new backend.

Yeah, the thing with retype/volume migration that isn't great is it
triggers the swap_volume callback to the source host in nova, so if nova
was orchestrating the volume retype/move, we'd need to wait for the swap
volume to be done (not impossible) before proceeding, and only the
libvirt driver implements the swap volume API. I've always wondered,
what the hell do non-libvirt deployments do with respect to the volume
retype/migration APIs in Cinder? Just disable them via policy?

--

Thanks,

Matt

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [nova][cinder][neutron] Cross-cell cold migration

Matt Riedemann-3
+operators

On 8/24/2018 4:08 PM, Matt Riedemann wrote:

> On 8/23/2018 10:22 AM, Sean McGinnis wrote:
>> I haven't gone through the workflow, but I thought shelve/unshelve
>> could detach
>> the volume on shelving and reattach it on unshelve. In that workflow,
>> assuming
>> the networking is in place to provide the connectivity, the nova
>> compute host
>> would be connecting to the volume just like any other attach and
>> should work
>> fine. The unknown or tricky part is making sure that there is the network
>> connectivity or routing in place for the compute host to be able to
>> log in to
>> the storage target.
>
> Yeah that's also why I like shelve/unshelve as a start since it's doing
> volume detach from the source host in the source cell and volume attach
> to the target host in the target cell.
>
> Host aggregates in Nova, as a grouping concept, are not restricted to
> cells at all, so you could have hosts in the same aggregate which span
> cells, so I'd think that's what operators would be doing if they have
> network/storage spanning multiple cells. Having said that, host
> aggregates are not exposed to non-admin end users, so again, if we rely
> on a normal user to do this move operation via resize, the only way we
> can restrict the instance to another host in the same aggregate is via
> availability zones, which is the user-facing aggregate construct in
> nova. I know Sam would care about this because NeCTAR sets
> [cinder]/cross_az_attach=False in nova.conf so servers/volumes are
> restricted to the same AZ, but that's not the default, and specifying an
> AZ when you create a server is not required (although there is a config
> option in nova which allows operators to define a default AZ for the
> instance if the user didn't specify one).
>
> Anyway, my point is, there are a lot of "ifs" if it's not an
> operator/admin explicitly telling nova where to send the server if it's
> moving across cells.
>
>>
>> If it's the other scenario mentioned where the volume needs to be
>> migrated from
>> one storage backend to another storage backend, then that may require
>> a little
>> more work. The volume would need to be retype'd or migrated (storage
>> migration)
>> from the original backend to the new backend.
>
> Yeah, the thing with retype/volume migration that isn't great is it
> triggers the swap_volume callback to the source host in nova, so if nova
> was orchestrating the volume retype/move, we'd need to wait for the swap
> volume to be done (not impossible) before proceeding, and only the
> libvirt driver implements the swap volume API. I've always wondered,
> what the hell do non-libvirt deployments do with respect to the volume
> retype/migration APIs in Cinder? Just disable them via policy?
>


--

Thanks,

Matt

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [nova][cinder][neutron] Cross-cell cold migration

Gorka Eguileor
In reply to this post by Jay S Bryant-2
On 24/08, Jay S Bryant wrote:

>
>
> On 8/23/2018 12:07 PM, Gorka Eguileor wrote:
> > On 23/08, Dan Smith wrote:
> > > > I think Nova should never have to rely on Cinder's hosts/backends
> > > > information to do migrations or any other operation.
> > > >
> > > > In this case even if Nova had that info, it wouldn't be the solution.
> > > > Cinder would reject migrations if there's an incompatibility on the
> > > > Volume Type (AZ, Referenced backend, capabilities...)
> > > I think I'm missing a bunch of cinder knowledge required to fully grok
> > > this situation and probably need to do some reading. Is there some
> > > reason that a volume type can't exist in multiple backends or something?
> > > I guess I think of volume type as flavor, and the same definition in two
> > > places would be interchangeable -- is that not the case?
> > >
> > Hi,
> >
> > I just know the basics of flavors, and they are kind of similar, though
> > I'm sure there are quite a few differences.
> >
> > Sure, multiple storage arrays can meet the requirements of a Volume
> > Type, but then when you create the volume you don't know where it's
> > going to land. If your volume type is too generic you volume could land
> > somewhere your cell cannot reach.
> >
> >
> > > > I don't know anything about Nova cells, so I don't know the specifics of
> > > > how we could do the mapping between them and Cinder backends, but
> > > > considering the limited range of possibilities in Cinder I would say we
> > > > only have Volume Types and AZs to work a solution.
> > > I think the only mapping we need is affinity or distance. The point of
> > > needing to migrate the volume would purely be because moving cells
> > > likely means you moved physically farther away from where you were,
> > > potentially with different storage connections and networking. It
> > > doesn't *have* to mean that, but I think in reality it would. So the
> > > question I think Matt is looking to answer here is "how do we move an
> > > instance from a DC in building A to building C and make sure the
> > > volume gets moved to some storage local in the new building so we're
> > > not just transiting back to the original home for no reason?"
> > >
> > > Does that explanation help or are you saying that's fundamentally hard
> > > to do/orchestrate?
> > >
> > > Fundamentally, the cells thing doesn't even need to be part of the
> > > discussion, as the same rules would apply if we're just doing a normal
> > > migration but need to make sure that storage remains affined to compute.
> > >
> > We could probably work something out using the affinity filter, but
> > right now we don't have a way of doing what you need.
> >
> > We could probably rework the migration to accept scheduler hints to be
> > used with the affinity filter and to accept calls with the host or the
> > hints, that way it could migrate a volume without knowing the
> > destination host and decide it based on affinity.
> >
> > We may have to do more modifications, but it could be a way to do it.
> >
> >
> >
> > > > I don't know how the Nova Placement works, but it could hold an
> > > > equivalency mapping of volume types to cells as in:
> > > >
> > > >   Cell#1        Cell#2
> > > >
> > > > VolTypeA <--> VolTypeD
> > > > VolTypeB <--> VolTypeE
> > > > VolTypeC <--> VolTypeF
> > > >
> > > > Then it could do volume retypes (allowing migration) and that would
> > > > properly move the volumes from one backend to another.
> > > The only way I can think that we could do this in placement would be if
> > > volume types were resource providers and we assigned them traits that
> > > had special meaning to nova indicating equivalence. Several of the words
> > > in that sentence are likely to freak out placement people, myself
> > > included :)
> > >
> > > So is the concern just that we need to know what volume types in one
> > > backend map to those in another so that when we do the migration we know
> > > what to ask for? Is "they are the same name" not enough? Going back to
> > > the flavor analogy, you could kinda compare two flavor definitions and
> > > have a good idea if they're equivalent or not...
> > >
> > > --Dan
> > In Cinder you don't get that from Volume Types, unless all your backends
> > have the same hardware and are configured exactly the same.
> >
> > There can be some storage specific information there, which doesn't
> > correlate to anything on other hardware.  Volume types may refer to a
> > specific pool that has been configured in the array to use specific type
> > of disks.  But even the info on the type of disks is unknown to the
> > volume type.
> >
> > I haven't checked the PTG agenda yet, but is there a meeting on this?
> > Because we may want to have one to try to understand the requirements
> > and figure out if there's a way to do it with current Cinder
> > functionality of if we'd need something new.
> Gorka,
>
> I don't think that this has been put on the agenda yet.  Might be good to
> add.  I don't think we have a cross project time officially planned with
> Nova.  I will start that discussion with Melanie so that we can cover the
> couple of cross projects subjects we have.
>
> Jay

Thanks Jay!


>
> > Cheers,
> > Gorka.
> >
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: [hidden email]?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [nova][cinder][neutron] Cross-cell cold migration

MIguel Lavalle
Hi Matt,

Isn't multiple port binding what we need in the case of ports? In my mind, the big motivator for multiple port binding is the ability to change a port's backend

Best regards

Miguel

On Mon, Aug 27, 2018 at 4:32 AM, Gorka Eguileor <[hidden email]> wrote:
On 24/08, Jay S Bryant wrote:
>
>
> On 8/23/2018 12:07 PM, Gorka Eguileor wrote:
> > On 23/08, Dan Smith wrote:
> > > > I think Nova should never have to rely on Cinder's hosts/backends
> > > > information to do migrations or any other operation.
> > > >
> > > > In this case even if Nova had that info, it wouldn't be the solution.
> > > > Cinder would reject migrations if there's an incompatibility on the
> > > > Volume Type (AZ, Referenced backend, capabilities...)
> > > I think I'm missing a bunch of cinder knowledge required to fully grok
> > > this situation and probably need to do some reading. Is there some
> > > reason that a volume type can't exist in multiple backends or something?
> > > I guess I think of volume type as flavor, and the same definition in two
> > > places would be interchangeable -- is that not the case?
> > >
> > Hi,
> >
> > I just know the basics of flavors, and they are kind of similar, though
> > I'm sure there are quite a few differences.
> >
> > Sure, multiple storage arrays can meet the requirements of a Volume
> > Type, but then when you create the volume you don't know where it's
> > going to land. If your volume type is too generic you volume could land
> > somewhere your cell cannot reach.
> >
> >
> > > > I don't know anything about Nova cells, so I don't know the specifics of
> > > > how we could do the mapping between them and Cinder backends, but
> > > > considering the limited range of possibilities in Cinder I would say we
> > > > only have Volume Types and AZs to work a solution.
> > > I think the only mapping we need is affinity or distance. The point of
> > > needing to migrate the volume would purely be because moving cells
> > > likely means you moved physically farther away from where you were,
> > > potentially with different storage connections and networking. It
> > > doesn't *have* to mean that, but I think in reality it would. So the
> > > question I think Matt is looking to answer here is "how do we move an
> > > instance from a DC in building A to building C and make sure the
> > > volume gets moved to some storage local in the new building so we're
> > > not just transiting back to the original home for no reason?"
> > >
> > > Does that explanation help or are you saying that's fundamentally hard
> > > to do/orchestrate?
> > >
> > > Fundamentally, the cells thing doesn't even need to be part of the
> > > discussion, as the same rules would apply if we're just doing a normal
> > > migration but need to make sure that storage remains affined to compute.
> > >
> > We could probably work something out using the affinity filter, but
> > right now we don't have a way of doing what you need.
> >
> > We could probably rework the migration to accept scheduler hints to be
> > used with the affinity filter and to accept calls with the host or the
> > hints, that way it could migrate a volume without knowing the
> > destination host and decide it based on affinity.
> >
> > We may have to do more modifications, but it could be a way to do it.
> >
> >
> >
> > > > I don't know how the Nova Placement works, but it could hold an
> > > > equivalency mapping of volume types to cells as in:
> > > >
> > > >   Cell#1        Cell#2
> > > >
> > > > VolTypeA <--> VolTypeD
> > > > VolTypeB <--> VolTypeE
> > > > VolTypeC <--> VolTypeF
> > > >
> > > > Then it could do volume retypes (allowing migration) and that would
> > > > properly move the volumes from one backend to another.
> > > The only way I can think that we could do this in placement would be if
> > > volume types were resource providers and we assigned them traits that
> > > had special meaning to nova indicating equivalence. Several of the words
> > > in that sentence are likely to freak out placement people, myself
> > > included :)
> > >
> > > So is the concern just that we need to know what volume types in one
> > > backend map to those in another so that when we do the migration we know
> > > what to ask for? Is "they are the same name" not enough? Going back to
> > > the flavor analogy, you could kinda compare two flavor definitions and
> > > have a good idea if they're equivalent or not...
> > >
> > > --Dan
> > In Cinder you don't get that from Volume Types, unless all your backends
> > have the same hardware and are configured exactly the same.
> >
> > There can be some storage specific information there, which doesn't
> > correlate to anything on other hardware.  Volume types may refer to a
> > specific pool that has been configured in the array to use specific type
> > of disks.  But even the info on the type of disks is unknown to the
> > volume type.
> >
> > I haven't checked the PTG agenda yet, but is there a meeting on this?
> > Because we may want to have one to try to understand the requirements
> > and figure out if there's a way to do it with current Cinder
> > functionality of if we'd need something new.
> Gorka,
>
> I don't think that this has been put on the agenda yet.  Might be good to
> add.  I don't think we have a cross project time officially planned with
> Nova.  I will start that discussion with Melanie so that we can cover the
> couple of cross projects subjects we have.
>
> Jay

Thanks Jay!


>
> > Cheers,
> > Gorka.
> >
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [nova][cinder][neutron] Cross-cell cold migration

Matt Riedemann-3
On 8/27/2018 12:11 PM, Miguel Lavalle wrote:
> Isn't multiple port binding what we need in the case of ports? In my
> mind, the big motivator for multiple port binding is the ability to
> change a port's backend

Hmm, yes maybe. Nova's usage of multiple port bindings today is
restricted to live migration which isn't what we're supporting with the
initial cross-cell (cold) migration support, but it could be a
dependency if that's what we need.

What I was wondering is if there is a concept like a port spanning or
migrating across networks? I'm assuming there isn't, and I'm not even
sure if that would be required here. But it would mean there is an
implicit requirement that for cross-cell migration to work, neutron
networks need to span cells (similarly storage backends would need to
span cells).

--

Thanks,

Matt

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [nova][cinder][neutron] Cross-cell cold migration

melanie witt-3
In reply to this post by Jay S Bryant-2
On Fri, 24 Aug 2018 10:44:16 -0500, Jay S Bryant wrote:

>> I haven't checked the PTG agenda yet, but is there a meeting on this?
>> Because we may want to have one to try to understand the requirements
>> and figure out if there's a way to do it with current Cinder
>> functionality of if we'd need something new.
> Gorka,
>
> I don't think that this has been put on the agenda yet.  Might be good
> to add.  I don't think we have a cross project time officially planned
> with Nova.  I will start that discussion with Melanie so that we can
> cover the couple of cross projects subjects we have.

Just to update everyone, we've schedule Cinder/Nova cross project time
for Thursday 9am-11am at the PTG, please add topics starting at L134 in
the Cinder section:

https://etherpad.openstack.org/p/nova-ptg-stein

Cheers,
-melanie




__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [nova][cinder][neutron] Cross-cell cold migration

Matt Riedemann-3
In reply to this post by Matt Riedemann-3
On 8/27/2018 1:53 PM, Matt Riedemann wrote:

> On 8/27/2018 12:11 PM, Miguel Lavalle wrote:
>> Isn't multiple port binding what we need in the case of ports? In my
>> mind, the big motivator for multiple port binding is the ability to
>> change a port's backend
>
> Hmm, yes maybe. Nova's usage of multiple port bindings today is
> restricted to live migration which isn't what we're supporting with the
> initial cross-cell (cold) migration support, but it could be a
> dependency if that's what we need.
>
> What I was wondering is if there is a concept like a port spanning or
> migrating across networks? I'm assuming there isn't, and I'm not even
> sure if that would be required here. But it would mean there is an
> implicit requirement that for cross-cell migration to work, neutron
> networks need to span cells (similarly storage backends would need to
> span cells).

In thinking about this again (sleepless at 3am of course), port bindings
doesn't help us here if we're orchestrating the cross-cell move using
shelve offload, because in that case the port is unbound from the source
host - while the instance is shelved offloaded, it has no host. When we
unshelve in the new cell, we'd update the port binding. So there isn't
really a use in this flow for multiple port bindings on multiple hosts
(assuming we stick with using the shelve/unshelve idea here).

--

Thanks,

Matt

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Openstack-operators] [nova][cinder][neutron] Cross-cell cold migration

Matt Riedemann-3
In reply to this post by Matt Riedemann-3
On 8/29/2018 3:21 PM, Tim Bell wrote:
> Sounds like a good topic for PTG/Forum?

Yeah it's already on the PTG agenda [1][2]. I started the thread because
I wanted to get the ball rolling as early as possible, and with people
that won't attend the PTG and/or the Forum, to weigh in on not only the
known issues with cross-cell migration but also the things I'm not
thinking about.

[1] https://etherpad.openstack.org/p/nova-ptg-stein
[2] https://etherpad.openstack.org/p/nova-ptg-stein-cells

--

Thanks,

Matt

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Openstack-operators] [nova][cinder][neutron] Cross-cell cold migration

Gorka Eguileor
On 29/08, Matt Riedemann wrote:

> On 8/29/2018 3:21 PM, Tim Bell wrote:
> > Sounds like a good topic for PTG/Forum?
>
> Yeah it's already on the PTG agenda [1][2]. I started the thread because I
> wanted to get the ball rolling as early as possible, and with people that
> won't attend the PTG and/or the Forum, to weigh in on not only the known
> issues with cross-cell migration but also the things I'm not thinking about.
>
> [1] https://etherpad.openstack.org/p/nova-ptg-stein
> [2] https://etherpad.openstack.org/p/nova-ptg-stein-cells
>
> --
>
> Thanks,
>
> Matt
>

Should we also add the topic to the Thursday Cinder-Nova slot in case
there are some questions where the Cinder team can assist?

Cheers,
Gorka.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Openstack-operators] [nova][cinder][neutron] Cross-cell cold migration

Sean McGinnis
> >
> > Yeah it's already on the PTG agenda [1][2]. I started the thread because I
> > wanted to get the ball rolling as early as possible, and with people that
> > won't attend the PTG and/or the Forum, to weigh in on not only the known
> > issues with cross-cell migration but also the things I'm not thinking about.
> >
> > [1] https://etherpad.openstack.org/p/nova-ptg-stein
> > [2] https://etherpad.openstack.org/p/nova-ptg-stein-cells
> >
> > --
> >
> > Thanks,
> >
> > Matt
> >
>
> Should we also add the topic to the Thursday Cinder-Nova slot in case
> there are some questions where the Cinder team can assist?
>
> Cheers,
> Gorka.
>

Good idea. That will be a good time to circle back between the teams to see if
any Cinder needs come up that we can still have time to talk through and see if
we can get work started.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [nova][cinder][neutron] Cross-cell cold migration

Matt Riedemann-3
In reply to this post by Matt Riedemann-3
After hacking on the PoC for awhile [1] I have finally pushed up a spec
[2]. Behold it in all its dark glory!

[1] https://review.openstack.org/#/c/603930/
[2] https://review.openstack.org/#/c/616037/

On 8/22/2018 8:23 PM, Matt Riedemann wrote:

> Hi everyone,
>
> I have started an etherpad for cells topics at the Stein PTG [1]. The
> main issue in there right now is dealing with cross-cell cold migration
> in nova.
>
> At a high level, I am going off these requirements:
>
> * Cells can shard across flavors (and hardware type) so operators would
> like to move users off the old flavors/hardware (old cell) to new
> flavors in a new cell.
>
> * There is network isolation between compute hosts in different cells,
> so no ssh'ing the disk around like we do today. But the image service is
> global to all cells.
>
> Based on this, for the initial support for cross-cell cold migration, I
> am proposing that we leverage something like shelve offload/unshelve
> masquerading as resize. We shelve offload from the source cell and
> unshelve in the target cell. This should work for both volume-backed and
> non-volume-backed servers (we use snapshots for shelved offloaded
> non-volume-backed servers).
>
> There are, of course, some complications. The main ones that I need help
> with right now are what happens with volumes and ports attached to the
> server. Today we detach from the source and attach at the target, but
> that's assuming the storage backend and network are available to both
> hosts involved in the move of the server. Will that be the case across
> cells? I am assuming that depends on the network topology (are routed
> networks being used?) and storage backend (routed storage?). If the
> network and/or storage backend are not available across cells, how do we
> migrate volumes and ports? Cinder has a volume migrate API for admins
> but I do not know how nova would know the proper affinity per-cell to
> migrate the volume to the proper host (cinder does not have a routed
> storage concept like routed provider networks in neutron, correct?). And
> as far as I know, there is no such thing as port migration in Neutron.
>
> Could Placement help with the volume/port migration stuff? Neutron
> routed provider networks rely on placement aggregates to schedule the VM
> to a compute host in the same network segment as the port used to create
> the VM, however, if that segment does not span cells we are kind of
> stuck, correct?
>
> To summarize the issues as I see them (today):
>
> * How to deal with the targeted cell during scheduling? This is so we
> can even get out of the source cell in nova.
>
> * How does the API deal with the same instance being in two DBs at the
> same time during the move?
>
> * How to handle revert resize?
>
> * How are volumes and ports handled?
>
> I can get feedback from my company's operators based on what their
> deployment will look like for this, but that does not mean it will work
> for others, so I need as much feedback from operators, especially those
> running with multiple cells today, as possible. Thanks in advance.
>
> [1] https://etherpad.openstack.org/p/nova-ptg-stein-cells
>


--

Thanks,

Matt

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev