[heat] Making stack outputs static

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[heat] Making stack outputs static

Zane Bitter
History lesson: a long, long time ago we made a very big mistake. We
treated stack outputs as things that would be resolved dynamically when
you requested them, instead of having values fixed at the time the
template was created or updated. This makes performance of reading
outputs slow, especially for e.g. large stacks, because it requires
making ReST calls, and it can result in inconsistencies between Heat's
internal model of the world and what it actually outputs.

As unfortunate as this is, it's difficult to change the behaviour and be
certain that no existing users will get broken. For that reason, this
issue has never been addressed. Now is the time to address it.

Here's the tracker bug: https://bugs.launchpad.net/heat/+bug/1660831

It turns out that the correct fix is to store the attributes of a
resource in the DB - this accounts for the fact that outputs may contain
attributes of multiple resources, and that these resources might get
updated at different times. It also solves a related consistency issue,
which is that during a stack update a resource that is not updated may
nevertheless report new attribute values, and thus cause things
downstream to be updated, or to fail, unexpectedly (e.g.
https://bugzilla.redhat.com/show_bug.cgi?id=1430753#c13).

The proposal[1] is to make this change in Pike for convergence stacks
only. This is to allow some warning for existing users who might be
relying on the current behaviour - at least if they control their own
cloud then they can opt to keep convergence disabled, and even once they
opt to enable it for new stacks they can keep using existing stacks in
legacy mode until they are ready to convert them to convergence or
replace them. In addition, it avoids the difficulty of trying to get
consistency out of the legacy path's crazy backup-stack shenanigans -
there's basically no way to get the outputs to behave in exactly the
same way in the legacy path as they will in convergence.

This topic was raised at the Forum, and there was some feedback that:

1) There are users who require the old behaviour even after they move to
convergence.
2) Specifically, there are users who don't have public API endpoints for
services other than Heat, and who rely on Heat proxying requests to
other services to get any information at all about their resources o.O
3) There are users still using the legacy path (*cough*TripleO) that
want the performance benefits of quick output resolution.

The suggestion is that instead of tying the change to the convergence
flag, we should make it configurable by the user on a per-stack basis.

I am vehemently opposed to this suggestion.

It's a total cop-out to make the user decide. The existing behaviour is
clearly buggy and inconsistent. Users are not, and should not have to
be, sufficiently steeped in the inner workings of Heat to be able to
decide whether and when to subject themselves to random inconsistencies
and hope for the best. If we make the change the default then we'll
still break people, and if we don't we'll still be saying "OMG, you
forgot to enable the --not-suck option??!" 10 years from now.

Instead, this is what I'm proposing as the solution to the above feedback:

1) The 'show' attribute of each resource will be marked CACHE_NONE[2].
This ensures that the live data is always available via this attribute.
2) When showing a resource's attributes via the API (as opposed to
referencing them from within a template), always return live values.[3]
Since we only store the attribute values that are actually referenced in
the template anyway, we more or less have to do this if we want the
attributes output through this API to be consistent with each other.
3) Move to convergence. Seriously, the memory and database usage are
much improved, and there are even more memory improvements in the
pipeline,[4] and they might even get merged in Pike as long as we don't
have to stop and reimplement the attribute storage patches that they
depend on. If TripleO were to move to convergence in Queens, which I
believe is 100% feasible, then it would get the performance improvements
at least as soon as it would if we tried to implement attribute storage
in the legacy path.

Is anyone still dissatisfied? Speak now or... you know the drill.

cheers,
Zane.

[1]
https://review.openstack.org/#/q/status:open+project:openstack/heat+branch:master+topic:bug/1660831
[2] https://review.openstack.org/#/c/422983/33/heat/engine/resource.py
[3] https://review.openstack.org/472501
[4]
https://review.openstack.org/#/q/status:open+project:openstack/heat+topic:bp/stack-definition

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [heat] Making stack outputs static

Ben Nemec


On 06/09/2017 03:10 PM, Zane Bitter wrote:

> History lesson: a long, long time ago we made a very big mistake. We
> treated stack outputs as things that would be resolved dynamically when
> you requested them, instead of having values fixed at the time the
> template was created or updated. This makes performance of reading
> outputs slow, especially for e.g. large stacks, because it requires
> making ReST calls, and it can result in inconsistencies between Heat's
> internal model of the world and what it actually outputs.
>
> As unfortunate as this is, it's difficult to change the behaviour and be
> certain that no existing users will get broken. For that reason, this
> issue has never been addressed. Now is the time to address it.
>
> Here's the tracker bug: https://bugs.launchpad.net/heat/+bug/1660831
>
> It turns out that the correct fix is to store the attributes of a
> resource in the DB - this accounts for the fact that outputs may contain
> attributes of multiple resources, and that these resources might get
> updated at different times. It also solves a related consistency issue,
> which is that during a stack update a resource that is not updated may
> nevertheless report new attribute values, and thus cause things
> downstream to be updated, or to fail, unexpectedly (e.g.
> https://bugzilla.redhat.com/show_bug.cgi?id=1430753#c13).
>
> The proposal[1] is to make this change in Pike for convergence stacks
> only. This is to allow some warning for existing users who might be
> relying on the current behaviour - at least if they control their own
> cloud then they can opt to keep convergence disabled, and even once they
> opt to enable it for new stacks they can keep using existing stacks in
> legacy mode until they are ready to convert them to convergence or
> replace them. In addition, it avoids the difficulty of trying to get
> consistency out of the legacy path's crazy backup-stack shenanigans -
> there's basically no way to get the outputs to behave in exactly the
> same way in the legacy path as they will in convergence.
>
> This topic was raised at the Forum, and there was some feedback that:
>
> 1) There are users who require the old behaviour even after they move to
> convergence.
> 2) Specifically, there are users who don't have public API endpoints for
> services other than Heat, and who rely on Heat proxying requests to
> other services to get any information at all about their resources o.O
> 3) There are users still using the legacy path (*cough*TripleO) that
> want the performance benefits of quick output resolution.
>
> The suggestion is that instead of tying the change to the convergence
> flag, we should make it configurable by the user on a per-stack basis.
>
> I am vehemently opposed to this suggestion.
>
> It's a total cop-out to make the user decide. The existing behaviour is
> clearly buggy and inconsistent. Users are not, and should not have to
> be, sufficiently steeped in the inner workings of Heat to be able to
> decide whether and when to subject themselves to random inconsistencies
> and hope for the best. If we make the change the default then we'll
> still break people, and if we don't we'll still be saying "OMG, you
> forgot to enable the --not-suck option??!" 10 years from now.
>
> Instead, this is what I'm proposing as the solution to the above feedback:
>
> 1) The 'show' attribute of each resource will be marked CACHE_NONE[2].
> This ensures that the live data is always available via this attribute.
> 2) When showing a resource's attributes via the API (as opposed to
> referencing them from within a template), always return live values.[3]
> Since we only store the attribute values that are actually referenced in
> the template anyway, we more or less have to do this if we want the
> attributes output through this API to be consistent with each other.
> 3) Move to convergence. Seriously, the memory and database usage are
> much improved, and there are even more memory improvements in the
> pipeline,[4] and they might even get merged in Pike as long as we don't
> have to stop and reimplement the attribute storage patches that they
> depend on. If TripleO were to move to convergence in Queens, which I
> believe is 100% feasible, then it would get the performance improvements
> at least as soon as it would if we tried to implement attribute storage
> in the legacy path.

I think we wanted to move to convergence anyway so I don't see a problem
with this.  I know there was some discussion about starting to test with
convergence in tripleo-ci, does anyone know what, if anything, happened
with that?

>
> Is anyone still dissatisfied? Speak now or... you know the drill.
>
> cheers,
> Zane.
>
> [1]
> https://review.openstack.org/#/q/status:open+project:openstack/heat+branch:master+topic:bug/1660831
>
> [2] https://review.openstack.org/#/c/422983/33/heat/engine/resource.py
> [3] https://review.openstack.org/472501
> [4]
> https://review.openstack.org/#/q/status:open+project:openstack/heat+topic:bp/stack-definition
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [heat] Making stack outputs static

Steven Hardy
On Mon, Jun 12, 2017 at 6:18 PM, Ben Nemec <[hidden email]> wrote:

>
>
> On 06/09/2017 03:10 PM, Zane Bitter wrote:
>>
>> History lesson: a long, long time ago we made a very big mistake. We
>> treated stack outputs as things that would be resolved dynamically when
>> you requested them, instead of having values fixed at the time the
>> template was created or updated. This makes performance of reading
>> outputs slow, especially for e.g. large stacks, because it requires
>> making ReST calls, and it can result in inconsistencies between Heat's
>> internal model of the world and what it actually outputs.
>>
>> As unfortunate as this is, it's difficult to change the behaviour and be
>> certain that no existing users will get broken. For that reason, this
>> issue has never been addressed. Now is the time to address it.
>>
>> Here's the tracker bug: https://bugs.launchpad.net/heat/+bug/1660831
>>
>> It turns out that the correct fix is to store the attributes of a
>> resource in the DB - this accounts for the fact that outputs may contain
>> attributes of multiple resources, and that these resources might get
>> updated at different times. It also solves a related consistency issue,
>> which is that during a stack update a resource that is not updated may
>> nevertheless report new attribute values, and thus cause things
>> downstream to be updated, or to fail, unexpectedly (e.g.
>> https://bugzilla.redhat.com/show_bug.cgi?id=1430753#c13).
>>
>> The proposal[1] is to make this change in Pike for convergence stacks
>> only. This is to allow some warning for existing users who might be
>> relying on the current behaviour - at least if they control their own
>> cloud then they can opt to keep convergence disabled, and even once they
>> opt to enable it for new stacks they can keep using existing stacks in
>> legacy mode until they are ready to convert them to convergence or
>> replace them. In addition, it avoids the difficulty of trying to get
>> consistency out of the legacy path's crazy backup-stack shenanigans -
>> there's basically no way to get the outputs to behave in exactly the
>> same way in the legacy path as they will in convergence.
>>
>> This topic was raised at the Forum, and there was some feedback that:
>>
>> 1) There are users who require the old behaviour even after they move to
>> convergence.
>> 2) Specifically, there are users who don't have public API endpoints for
>> services other than Heat, and who rely on Heat proxying requests to
>> other services to get any information at all about their resources o.O
>> 3) There are users still using the legacy path (*cough*TripleO) that
>> want the performance benefits of quick output resolution.
>>
>> The suggestion is that instead of tying the change to the convergence
>> flag, we should make it configurable by the user on a per-stack basis.
>>
>> I am vehemently opposed to this suggestion.
>>
>> It's a total cop-out to make the user decide. The existing behaviour is
>> clearly buggy and inconsistent. Users are not, and should not have to
>> be, sufficiently steeped in the inner workings of Heat to be able to
>> decide whether and when to subject themselves to random inconsistencies
>> and hope for the best. If we make the change the default then we'll
>> still break people, and if we don't we'll still be saying "OMG, you
>> forgot to enable the --not-suck option??!" 10 years from now.
>>
>> Instead, this is what I'm proposing as the solution to the above feedback:
>>
>> 1) The 'show' attribute of each resource will be marked CACHE_NONE[2].
>> This ensures that the live data is always available via this attribute.
>> 2) When showing a resource's attributes via the API (as opposed to
>> referencing them from within a template), always return live values.[3]
>> Since we only store the attribute values that are actually referenced in
>> the template anyway, we more or less have to do this if we want the
>> attributes output through this API to be consistent with each other.
>> 3) Move to convergence. Seriously, the memory and database usage are
>> much improved, and there are even more memory improvements in the
>> pipeline,[4] and they might even get merged in Pike as long as we don't
>> have to stop and reimplement the attribute storage patches that they
>> depend on. If TripleO were to move to convergence in Queens, which I
>> believe is 100% feasible, then it would get the performance improvements
>> at least as soon as it would if we tried to implement attribute storage
>> in the legacy path.
>
>
> I think we wanted to move to convergence anyway so I don't see a problem
> with this.  I know there was some discussion about starting to test with
> convergence in tripleo-ci, does anyone know what, if anything, happened with
> that?

There's an experimental job that runs only on the heat repo
(gate-tripleo-ci-centos-7-ovb-nonha-convergence)

But yeah now seems like a good time to get something running more
regularly in tripleo-ci.

Steve

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [heat] Making stack outputs static

Zane Bitter
On 12/06/17 16:21, Steven Hardy wrote:
>> I think we wanted to move to convergence anyway so I don't see a problem
>> with this.  I know there was some discussion about starting to test with
>> convergence in tripleo-ci, does anyone know what, if anything, happened with
>> that?
> There's an experimental job that runs only on the heat repo
> (gate-tripleo-ci-centos-7-ovb-nonha-convergence)
>
> But yeah now seems like a good time to get something running more
> regularly in tripleo-ci.

+1, there's no reason not to run a non-voting job against tripleo itself
at this point IMHO. That would allow me to start tracking the memory use
over time.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [heat] Making stack outputs static

Ben Nemec


On 06/12/2017 03:53 PM, Zane Bitter wrote:

> On 12/06/17 16:21, Steven Hardy wrote:
>>> I think we wanted to move to convergence anyway so I don't see a problem
>>> with this.  I know there was some discussion about starting to test with
>>> convergence in tripleo-ci, does anyone know what, if anything,
>>> happened with
>>> that?
>> There's an experimental job that runs only on the heat repo
>> (gate-tripleo-ci-centos-7-ovb-nonha-convergence)
>>
>> But yeah now seems like a good time to get something running more
>> regularly in tripleo-ci.
>
> +1, there's no reason not to run a non-voting job against tripleo itself
> at this point IMHO. That would allow me to start tracking the memory use
> over time.

Do you have a strong preference multinode vs. ovb?  I would tend to
think we want this to be ovb since multinode stubs out a bunch of stuff,
but at the same time ovb capacity is limited.  It's better* now that
we've basically halved our ovb ci coverage so we could probably get away
with adding a job to a specific repo though (t-h-t seems logical for
this purpose).

*for some definition of "better"

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [heat][tripleo] Making stack outputs static

Zane Bitter
On 13/06/17 13:00, Ben Nemec wrote:

>
>
> On 06/12/2017 03:53 PM, Zane Bitter wrote:
>> On 12/06/17 16:21, Steven Hardy wrote:
>>>> I think we wanted to move to convergence anyway so I don't see a
>>>> problem
>>>> with this.  I know there was some discussion about starting to test
>>>> with
>>>> convergence in tripleo-ci, does anyone know what, if anything,
>>>> happened with
>>>> that?
>>> There's an experimental job that runs only on the heat repo
>>> (gate-tripleo-ci-centos-7-ovb-nonha-convergence)
>>>
>>> But yeah now seems like a good time to get something running more
>>> regularly in tripleo-ci.
>>
>> +1, there's no reason not to run a non-voting job against tripleo itself
>> at this point IMHO. That would allow me to start tracking the memory use
>> over time.
>
> Do you have a strong preference multinode vs. ovb?  I would tend to
> think we want this to be ovb since multinode stubs out a bunch of stuff,
> but at the same time ovb capacity is limited.  It's better* now that
> we've basically halved our ovb ci coverage so we could probably get away
> with adding a job to a specific repo though (t-h-t seems logical for
> this purpose).

The job I've been tracking memory usage on against t-h-t was
tripleo-ci-centos-7-ovb-nonha... which I see no longer exists (doh!). So
I guess we can pick whatever it makes sense to track over the long term.

We want something fairly representative, so that we can be confident to
flip the switch early in Queens development. I don't think it especially
matters whether we're using baremetal servers or VMs though... that
should be fairly transparent to Heat.

cheers,
Zane.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev