[ceilometer] The reset on the cumulative counter

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

[ceilometer] The reset on the cumulative counter

Jiang, Yunhong
Hi,
        Eglynn and I discussed about how to handle the cumulative counter on IRC and we want to continue the discussion in the ML to get more feedback. The IRC log is in http://pastebin.com/byKvhNcJ. Eglynn, please correct me if I make any mistake.

        The issue is the cpu_info[cpu_time] is reset to 0 when a domain is suspend/resume. There are already some discussion on it already, and several potential solution raised.
        Jd discussed this firstly in https://bugs.launchpad.net/ceilometer/+bug/1061817 and then it's discussed in ML at http://www.mail-archive.com/openstack at lists.launchpad.net/msg17781.html.
       
        There are several method on this, like JD's MAX()- FIRST(), or asalkeld's idea of store the offset. Both try to detect if there is reset happen by checking if there is counter reverse. (again, Eglynn, please correct me if I'm wrong).

        However, during the discussion, we think that if the suspend/resume cycle is very rapid, and the ceilometer polling frequency is set up so that the stats appear to be in a monotonic sequence, then ceilometer will failed to detect the reset. A malicious user can exploit such situation.

        For example, the ceilometer query every 4 minutes, and suspend/resume take 1 minutes (not exact length, just example). Then if user try to reset every 4 min, right after the ceilometer polling, and then execute 3 minutes. With such arrangement, ceilometer can't detect the reset. Yes, it's difficult to achieve this detect, but once achieved it, it will save a lot of cost.

        My proposal is to fix this issue from nova side as it's nova bug. Per my understanding, the reason is different view of openstack/libvirt. For openstack, the instance is a logic object, thus still exist after suspend/resumt, while for libvirt, it's a real object, qemu process, which does not exist after suspend/resume. Nova should translate libvirt's view to openstack's view.  And this issue happens to other functionality, like nova's get_diagnostic().  But I agree with Eglynn that there are several implementation difficulties to fix it from nova side.

        Hope to get feedback from more person.

Thanks
--jyh


Reply | Threaded
Open this post in threaded view
|

[ceilometer] The reset on the cumulative counter

Julien Danjou
On Mon, Nov 26 2012, Jiang, Yunhong wrote:

Hi,

[?]

> There are several method on this, like JD's MAX()- FIRST(), or
> asalkeld's idea of store the offset. Both try to detect if there is reset
> happen by checking if there is counter reverse. (again, Eglynn, please
> correct me if I'm wrong).

I've indeed already express my point of view of solving this issue, but
I can do it here again.:)

I'm against trying to do anything fancy on calculation in pollsters, and
want always see raw data sent to the collector. One promise of
Ceilometer has always been to do no aggregation of any kind.

The "final value" computing problem is a probably easily solved with the
good algorithm in the API, and more specifically via the storage backend
if it's capable of doing it (at least SQL is, and I'm pretty sure
MongoDB would be).

[?]

> My proposal is to fix this issue from nova side as it's nova bug.
> Per my understanding, the reason is different view of openstack/libvirt. For
> openstack, the instance is a logic object, thus still exist after
> suspend/resumt, while for libvirt, it's a real object, qemu process, which
> does not exist after suspend/resume. Nova should translate libvirt's view to
> openstack's view. And this issue happens to other functionality, like nova's
> get_diagnostic(). But I agree with Eglynn that there are several
> implementation difficulties to fix it from nova side.

Just to be clear, that's a totally different issue. The fact that you
want to fix this in Nova is understandable, but you're not unfortunately
not going to be able to fix it for every pollsters people will write in
the future.

So both problems have to be addressed:
1. how to deal with resetable counters
2. how to prevent counters to be reset

And I agree with you that 2. is more a Nova issue than a Ceilometer one,
since we want that value to be provided by Nova at some point and not by
libvirt directly IIUC, which is a good idea.

--
Julien Danjou
# Free Software hacker & freelance
# http://julien.danjou.info
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121126/3feb0f9a/attachment.pgp>

Reply | Threaded
Open this post in threaded view
|

[ceilometer] The reset on the cumulative counter

Doug Hellmann
On Mon, Nov 26, 2012 at 8:53 AM, Julien Danjou <julien at danjou.info> wrote:

> On Mon, Nov 26 2012, Jiang, Yunhong wrote:
>
> Hi,
>
> [?]
>
> >       There are several method on this, like JD's MAX()- FIRST(), or
> > asalkeld's idea of store the offset. Both try to detect if there is reset
> > happen by checking if there is counter reverse. (again, Eglynn, please
> > correct me if I'm wrong).
>
> I've indeed already express my point of view of solving this issue, but
> I can do it here again.:)
>
> I'm against trying to do anything fancy on calculation in pollsters, and
> want always see raw data sent to the collector. One promise of
> Ceilometer has always been to do no aggregation of any kind.
>
> The "final value" computing problem is a probably easily solved with the
> good algorithm in the API, and more specifically via the storage backend
> if it's capable of doing it (at least SQL is, and I'm pretty sure
> MongoDB would be).
>

I'm a little concerned about how either implementation would actually
perform when calculating the values on an API call. I've thought about
doing it when the counter data is reported, but I'm not sure there's an
atomic way to do the update cleanly.

If we do decide to handle the calculation during the API query, we need to
ensure all backends can support it. Did we have a SQL solution that wasn't
tied to the underlying database (I remember a Postgresql solution using
window functions, but don't know about one for MySQL)?


>
> [?]
>
> >       My proposal is to fix this issue from nova side as it's nova bug.
> > Per my understanding, the reason is different view of openstack/libvirt.
> For
> > openstack, the instance is a logic object, thus still exist after
> > suspend/resumt, while for libvirt, it's a real object, qemu process,
> which
> > does not exist after suspend/resume. Nova should translate libvirt's
> view to
> > openstack's view. And this issue happens to other functionality, like
> nova's
> > get_diagnostic(). But I agree with Eglynn that there are several
> > implementation difficulties to fix it from nova side.
>
> Just to be clear, that's a totally different issue. The fact that you
> want to fix this in Nova is understandable, but you're not unfortunately
> not going to be able to fix it for every pollsters people will write in
> the future.
>
> So both problems have to be addressed:
> 1. how to deal with resetable counters
> 2. how to prevent counters to be reset
>
> And I agree with you that 2. is more a Nova issue than a Ceilometer one,
> since we want that value to be provided by Nova at some point and not by
> libvirt directly IIUC, which is a good idea.
>
> --
> Julien Danjou
> # Free Software hacker & freelance
> # http://julien.danjou.info
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121126/a41107f4/attachment.html>

Reply | Threaded
Open this post in threaded view
|

[ceilometer] The reset on the cumulative counter

Eoghan Glynn
In reply to this post by Jiang, Yunhong


> Eglynn and I discussed about how to handle the cumulative counter on
> IRC and we want to continue the discussion in the ML to get more
> feedback. The IRC log is in http://pastebin.com/byKvhNcJ. Eglynn,
> please correct me if I make any mistake.
>
> The issue is the cpu_info[cpu_time] is reset to 0 when a domain is
> suspend/resume. There are already some discussion on it already,
> and several potential solution raised.
> Jd discussed this firstly in
> https://bugs.launchpad.net/ceilometer/+bug/1061817 and then it's
> discussed in ML at
> http://www.mail-archive.com/openstack at lists.launchpad.net/msg17781.html.
>
> There are several method on this, like JD's MAX()- FIRST(), or
> asalkeld's idea of store the offset. Both try to detect if there is
> reset happen by checking if there is counter reverse. (again,
> Eglynn, please correct me if I'm wrong).
>
> However, during the discussion, we think that if the suspend/resume
> cycle is very rapid, and the ceilometer polling frequency is set up
> so that the stats appear to be in a monotonic sequence, then
> ceilometer will failed to detect the reset. A malicious user can
> exploit such situation.
>
> For example, the ceilometer query every 4 minutes, and
> suspend/resume take 1 minutes (not exact length, just example).
> Then if user try to reset every 4 min, right after the ceilometer
> polling, and then execute 3 minutes. With such arrangement,
> ceilometer can't detect the reset. Yes, it's difficult to achieve
> this detect, but once achieved it, it will save a lot of cost.
>
> My proposal is to fix this issue from nova side as it's nova bug.
> Per my understanding, the reason is different view of
> openstack/libvirt. For openstack, the instance is a logic object,
> thus still exist after suspend/resumt, while for libvirt, it's a
> real object, qemu process, which does not exist after
> suspend/resume. Nova should translate libvirt's view to openstack's
> view.  And this issue happens to other functionality, like nova's
> get_diagnostic().  But I agree with Eglynn that there are several
> implementation difficulties to fix it from nova side.


Here's another datapoint to consider, the bandwidth usage accounting
code in nova (currently only implemented for the xenapi driver) uses
a fairly basic scheme to detect resets simply by remembering the last
value and checking if the sequence appears monotonic:

  https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2928

I don't know if this code is intended to produce rock-solid or just
"advisory" stats, but it's interesting that it has taken this simple
approach.

As discussed on IRC earlier, I'd be leery of going down the road of
enforcing a "fix" on the nova side, such that resets were detected and
adjusted for prior to being reported (apart from the complexity of
reliably storing the previous value, there are multiple different
counters that could potentially be reset in different ways, possibly
depending on the hypervisor driver etc.)

However doing the adjustment within the metering store would always
be possible (either by storing a running offset as well as the reported
values, or switching to a delta-based measurement), as would isolating
the local maxima for max-min style queries (though we'd need to bound
the computational cost of that approach).

Of course that doesn't help in terms of detecting the edge case
where the reset is occurring so rapidly that the post-reset value
overtakes the last known value *before* the next sample is
taken. AFAICS this could only be addressed in code that's aware
of all the possible cases where a reset can occur, and is also in
a position to trigger the appropriate actions to enable the
adjustment (e.g. nova-compute persisting the latest stats when it
receives a 'suspend_instance' RPC message).

Cheers,
Eoghan


Reply | Threaded
Open this post in threaded view
|

[ceilometer] The reset on the cumulative counter

Doug Hellmann
On Mon, Nov 26, 2012 at 12:11 PM, Eoghan Glynn <eglynn at redhat.com> wrote:

>
>
> >       Eglynn and I discussed about how to handle the cumulative counter
> on
> >       IRC and we want to continue the discussion in the ML to get more
> >       feedback. The IRC log is in http://pastebin.com/byKvhNcJ. Eglynn,
> >       please correct me if I make any mistake.
> >
> >       The issue is the cpu_info[cpu_time] is reset to 0 when a domain is
> >       suspend/resume. There are already some discussion on it already,
> >       and several potential solution raised.
> >       Jd discussed this firstly in
> >       https://bugs.launchpad.net/ceilometer/+bug/1061817 and then it's
> >       discussed in ML at
> >
> http://www.mail-archive.com/openstack at lists.launchpad.net/msg17781.html.
> >
> >       There are several method on this, like JD's MAX()- FIRST(), or
> >       asalkeld's idea of store the offset. Both try to detect if there is
> >       reset happen by checking if there is counter reverse. (again,
> >       Eglynn, please correct me if I'm wrong).
> >
> >       However, during the discussion, we think that if the suspend/resume
> >       cycle is very rapid, and the ceilometer polling frequency is set up
> >       so that the stats appear to be in a monotonic sequence, then
> >       ceilometer will failed to detect the reset. A malicious user can
> >       exploit such situation.
> >
> >       For example, the ceilometer query every 4 minutes, and
> >       suspend/resume take 1 minutes (not exact length, just example).
> >       Then if user try to reset every 4 min, right after the ceilometer
> >       polling, and then execute 3 minutes. With such arrangement,
> >       ceilometer can't detect the reset. Yes, it's difficult to achieve
> >       this detect, but once achieved it, it will save a lot of cost.
> >
> >       My proposal is to fix this issue from nova side as it's nova bug.
> >       Per my understanding, the reason is different view of
> >       openstack/libvirt. For openstack, the instance is a logic object,
> >       thus still exist after suspend/resumt, while for libvirt, it's a
> >       real object, qemu process, which does not exist after
> >       suspend/resume. Nova should translate libvirt's view to openstack's
> >       view.  And this issue happens to other functionality, like nova's
> >       get_diagnostic().  But I agree with Eglynn that there are several
> >       implementation difficulties to fix it from nova side.
>
>
> Here's another datapoint to consider, the bandwidth usage accounting
> code in nova (currently only implemented for the xenapi driver) uses
> a fairly basic scheme to detect resets simply by remembering the last
> value and checking if the sequence appears monotonic:
>
>
> https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2928
>
> I don't know if this code is intended to produce rock-solid or just
> "advisory" stats, but it's interesting that it has taken this simple
> approach.
>
> As discussed on IRC earlier, I'd be leery of going down the road of
> enforcing a "fix" on the nova side, such that resets were detected and
> adjusted for prior to being reported (apart from the complexity of
> reliably storing the previous value, there are multiple different
> counters that could potentially be reset in different ways, possibly
> depending on the hypervisor driver etc.)
>
> However doing the adjustment within the metering store would always
> be possible (either by storing a running offset as well as the reported
> values, or switching to a delta-based measurement), as would isolating
> the local maxima for max-min style queries (though we'd need to bound
> the computational cost of that approach).
>

I understand Julien's reluctance to have the pollsters emitting computed
data, but tracking values inside the pollster so we can send a delta
instead of the cumulative value we have now seems like a relatively
straightforward fix for this.


>
> Of course that doesn't help in terms of detecting the edge case
> where the reset is occurring so rapidly that the post-reset value
> overtakes the last known value *before* the next sample is
> taken. AFAICS this could only be addressed in code that's aware
> of all the possible cases where a reset can occur, and is also in
> a position to trigger the appropriate actions to enable the
> adjustment (e.g. nova-compute persisting the latest stats when it
> receives a 'suspend_instance' RPC message).
>

We're already plugging in to the notifier API to have nova do some polling
when instances are deleted. We could do the same on suspend, reboot, etc.
-- whatever introduces this race condition.

Doug


>
> Cheers,
> Eoghan
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121126/5177015b/attachment.html>

Reply | Threaded
Open this post in threaded view
|

[ceilometer] The reset on the cumulative counter

Julien Danjou
On Mon, Nov 26 2012, Doug Hellmann wrote:

> I understand Julien's reluctance to have the pollsters emitting computed
> data, but tracking values inside the pollster so we can send a delta
> instead of the cumulative value we have now seems like a relatively
> straightforward fix for this.

I may have a better idea.
The problem with doing that in the pollster, is that:
1. it's bad bad bad bad
2. it's inefficient since you may lose values on restart
   you won't know what the value was before
3. it's not sure all publishers targets will want delta

What about doing this with both this approaches:
- with a transformer if someone really wants to do this for a publisher
- for ceilometer-collector, so it inside the collector: it will always
  know what the last value is, since it's stored in the database. So it
  just has to fetch the last stored value from the db if it doesn't know
  it, compute the delta, and stores it.

My main concerns in this, is that it'll be stored as a delta. We'll have
to be clear that polling this kind of things, like delta in general
actually, can't be done multiple times in parrallel. Something that is
easily done -- though maybe not generally wanted -- with cumulative
counters.

Thoughts?

--
Julien Danjou
;; Free Software hacker & freelance
;; http://julien.danjou.info
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121127/83a8ec5e/attachment.pgp>

Reply | Threaded
Open this post in threaded view
|

[ceilometer] The reset on the cumulative counter

Jiang, Yunhong
> What about doing this with both this approaches:
> - with a transformer if someone really wants to do this for a publisher

Yes, this is feasible through transformer, and two implementation details are, firstly we need identify the same counter, (same counter name, same resources_id, same instance_id etc), secondly we need keep a cache for compare, we need clean them when like instance destroyed.

I'm considering to implement a cached_transformer base class, which can be used for cpu_utilization, summary several counter for CW's network in/out etc.

> - for ceilometer-collector, so it inside the collector: it will always
>   know what the last value is, since it's stored in the database. So it
>   just has to fetch the last stored value from the db if it doesn't know
>   it, compute the delta, and stores it.
>
> My main concerns in this, is that it'll be stored as a delta. We'll have
> to be clear that polling this kind of things, like delta in general
> actually, can't be done multiple times in parrallel. Something that is
> easily done -- though maybe not generally wanted -- with cumulative

I assume the data from same pollster in such situation. If two counter for same source comes from two pollster, the transformer can raise exception if needed.

--jyh



Reply | Threaded
Open this post in threaded view
|

[ceilometer] The reset on the cumulative counter

Julien Danjou
On Tue, Nov 27 2012, Jiang, Yunhong wrote:

> I assume the data from same pollster in such situation. If two counter for
> same source comes from two pollster, the transformer can raise exception if
> needed.

Yeah, but that doesn't cover the case where multiple ceilometer-agent
are running for examples.

--
Julien Danjou
/* Free Software hacker & freelance
   http://julien.danjou.info */
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121127/fad2cc3c/attachment.pgp>

Reply | Threaded
Open this post in threaded view
|

[ceilometer] The reset on the cumulative counter

Eoghan Glynn
In reply to this post by Julien Danjou


> - for ceilometer-collector, so it inside the collector: it will
>   always know what the last value is, since it's stored in the
>   database. So it just has to fetch the last stored value from
>   the db if it doesn't know it, compute the delta, and stores it.

Yep, that's pretty much what I meant by:

"However doing the adjustment within the metering store would always
be possible (either by storing a running offset as well as the reported
values, or switching to a delta-based measurement)"


> My main concerns in this, is that it'll be stored as a
> delta. We'll have to be clear that polling this kind of things,
> like delta in general actually, can't be done multiple times in
> parrallel. Something that is easily done -- though maybe not
> generally wanted -- with cumulative counters.

Hmmm, are you thinking of the case where there are multiple
ceilo agents running the same compute host, but duplicating
the pollsters they've loaded?

Seems more likely to me that multiple ceilo compute agents
would take a disjoint subset of the pollsters each, rather
than duplicating.

Or, by "polling ... done multiple times in parrallel" do you
mean querying the ceilo API in parallel?

Cheers,
Eoghan


Reply | Threaded
Open this post in threaded view
|

[ceilometer] The reset on the cumulative counter

Julien Danjou
On Tue, Nov 27 2012, Eoghan Glynn wrote:

> Hmmm, are you thinking of the case where there are multiple
> ceilo agents running the same compute host, but duplicating
> the pollsters they've loaded?
>
> Seems more likely to me that multiple ceilo compute agents
> would take a disjoint subset of the pollsters each, rather
> than duplicating.

That's what I'm thinking about, but, as just as something doable or that
could happen. It's not something I would advise for, that's all.

--
Julien Danjou
/* Free Software hacker & freelance
   http://julien.danjou.info */
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121127/ef325d7a/attachment.pgp>

Reply | Threaded
Open this post in threaded view
|

[ceilometer] The reset on the cumulative counter

Doug Hellmann
In reply to this post by Julien Danjou
On Tue, Nov 27, 2012 at 4:49 AM, Julien Danjou <julien at danjou.info> wrote:

> On Mon, Nov 26 2012, Doug Hellmann wrote:
>
> > I understand Julien's reluctance to have the pollsters emitting computed
> > data, but tracking values inside the pollster so we can send a delta
> > instead of the cumulative value we have now seems like a relatively
> > straightforward fix for this.
>
> I may have a better idea.
> The problem with doing that in the pollster, is that:
> 1. it's bad bad bad bad
> 2. it's inefficient since you may lose values on restart
>    you won't know what the value was before
>

We will if we persist them on the compute node.


> 3. it's not sure all publishers targets will want delta
>

True. We may need to send gauge and delta events (or combine the values in
a single event).


>
> What about doing this with both this approaches:
> - with a transformer if someone really wants to do this for a publisher
> - for ceilometer-collector, so it inside the collector: it will always
>   know what the last value is, since it's stored in the database. So it
>   just has to fetch the last stored value from the db if it doesn't know
>   it, compute the delta, and stores it.
>

That doesn't work if different collector instances receive events from the
same meter on the same resource in parallel (which could happen easily if
the queue gets backed up).


>
> My main concerns in this, is that it'll be stored as a delta. We'll have
> to be clear that polling this kind of things, like delta in general
> actually, can't be done multiple times in parrallel. Something that is
> easily done -- though maybe not generally wanted -- with cumulative
> counters.
>
> Thoughts?
>
> --
> Julien Danjou
> ;; Free Software hacker & freelance
> ;; http://julien.danjou.info
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121127/90243a4a/attachment.html>

Reply | Threaded
Open this post in threaded view
|

[ceilometer] The reset on the cumulative counter

Julien Danjou
On Tue, Nov 27 2012, Doug Hellmann wrote:

>> I may have a better idea.
>> The problem with doing that in the pollster, is that:
>> 1. it's bad bad bad bad
>> 2. it's inefficient since you may lose values on restart
>>    you won't know what the value was before
>>
>
> We will if we persist them on the compute node.

See point 1. :-)

> That doesn't work if different collector instances receive events from the
> same meter on the same resource in parallel (which could happen easily if
> the queue gets backed up).

It does if you don't try to cache locally but request the last value
Reply | Threaded
Open this post in threaded view
|

[ceilometer] The reset on the cumulative counter

Doug Hellmann
On Wed, Nov 28, 2012 at 6:13 AM, Julien Danjou <julien at danjou.info> wrote:

> On Tue, Nov 27 2012, Doug Hellmann wrote:
>
> >> I may have a better idea.
> >> The problem with doing that in the pollster, is that:
> >> 1. it's bad bad bad bad
> >> 2. it's inefficient since you may lose values on restart
> >>    you won't know what the value was before
> >>
> >
> > We will if we persist them on the compute node.
>
> See point 1. :-)
>
> > That doesn't work if different collector instances receive events from
> the
> > same meter on the same resource in parallel (which could happen easily if
> > the queue gets backed up).
>
> It does if you don't try to cache locally but request the last value
> From the storage backend, right?
>

There's a race condition, though. If you and I are processing events at the
same time, we get the same "last value" from the storage but then try to
update it to our current value.

Doug


>
> --
> Julien Danjou
> // Free Software hacker & freelance
> // http://julien.danjou.info
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121128/737c4a9a/attachment.html>

Reply | Threaded
Open this post in threaded view
|

[ceilometer] The reset on the cumulative counter

Julien Danjou
On Wed, Nov 28 2012, Doug Hellmann wrote:

> There's a race condition, though. If you and I are processing events at the
> same time, we get the same "last value" from the storage but then try to
> update it to our current value.

Right. Now that I think about it, that problems with multiple collector
would also exists if we try to transform cumulative to delta on
notifications.

--
Julien Danjou
/* Free Software hacker & freelance
   http://julien.danjou.info */
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121128/884e66c7/attachment.pgp>

Reply | Threaded
Open this post in threaded view
|

[ceilometer] The reset on the cumulative counter

Doug Hellmann
On Wed, Nov 28, 2012 at 7:04 AM, Julien Danjou <julien at danjou.info> wrote:

> On Wed, Nov 28 2012, Doug Hellmann wrote:
>
> > There's a race condition, though. If you and I are processing events at
> the
> > same time, we get the same "last value" from the storage but then try to
> > update it to our current value.
>
> Right. Now that I think about it, that problems with multiple collector
> would also exists if we try to transform cumulative to delta on
> notifications.
>

Right. That's why it has to be done at the source, where the data is being
collected.

I talked about this a bit with the folks from Yahoo at the summit. They're
using this approach in their own monitoring, and seeing good results. They
use a little sqlite db on the compute node to persist the last few samples
from the meters so they can detect resets and the agent emits delta values
instead of cumulative values.

We have a further advantage in that we're already plugging into nova to
trigger stats collection. If we add a few more notification hooks for cases
where these counters get reset, we can be sure of detecting the reset no
matter what the polling interval is.

I think applying both techniques would give us the results we want.

Doug


>
> --
> Julien Danjou
> /* Free Software hacker & freelance
>    http://julien.danjou.info */
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121128/1f0a10b3/attachment.html>

Reply | Threaded
Open this post in threaded view
|

[ceilometer] The reset on the cumulative counter

Eoghan Glynn
In reply to this post by Doug Hellmann

> > > That doesn't work if different collector instances receive
> > > events from the same meter on the same resource in
> > > parallel (which could happen easily if the queue gets
> > > backed up).
> >
> > It does if you don't try to cache locally but request the
> > last value From the storage backend, right?
>
> There's a race condition, though. If you and I are processing events
> at the same time, we get the same "last value" from the storage but
> then try to update it to our current value.

Yeah, I think it would be possible to disambiguate with the timestamp
as long as only two events are mis-ordered (as the older last_value
can always be recovered from the more recent last_value - delta).

However that approach wouldn't extend to t2,t0,t1 style misorderings
(if such a thing has a non-zero probability of occurring) and the
problem quickly becomes intractable.

So sounds like the delta would be calculated at source to be 100%
rock-solid.

Cheers,
Eoghan