[heat] Split tempest plugin from heat

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[heat] Split tempest plugin from heat

Rabi Mishra
Hi All,

As part of community goal[1] for Queens, we've completed the repo split and created the new project[2].

The next objective to is to use the new plugin in our jobs. As we've merged some changes after the split, I've synced them and fixed some minor issues before using it in the gate jobs.

These are very small changes and I would suggest we review/land them on priority (we don't have to keep syncing them again and again for broken jobs). We should probably *not approve* any changes to integration tests in heat before these go in.

- Changes in heat-tempest-plugin (sync  missing patches and fixes)
  https://review.openstack.org/#/q/project:openstack/heat-tempest-plugin+topic:sync_from_heat

- Use heat-tempest-plugin for integration jobs
  https://review.openstack.org/#/c/508112/

- Use heat-tempest-plugin for grenade job
  https://review.openstack.org/#/c/521246/

- Add heat integration jobs to heat-tempest-plugin check/gate queue
  https://review.openstack.org/#/c/521340/
  I guess this would result in some issues, where we can't add any changes to heat that breaks the existing tests, as the changes for both projects would have a circular dependency (not sure how it works atm with other plugins!).

- Remove plugin an integration tests from heat (This has -1 atm as releasenote job is broken, waiting for infra to fix it)
  https://review.openstack.org/#/c/521263/ 

I've also created an etherpad[3] to track these.

Also, the plugin project is expected to be branchless (we may not backport these job changes to stable branches soon though), we've to find  a  way to run additional tests for any new feature only on > <release/branch>. AFAIK, other projects check api microversions supported and without microversions in heat, may be we've find an alternate way.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [heat][qa] Split tempest plugin from heat

Zane Bitter
On 19/11/17 03:08, Rabi Mishra wrote:

> Hi All,
>
> As part of community goal[1] for Queens, we've completed the repo split
> and created the new project[2].
>
> The next objective to is to use the new plugin in our jobs. As we've
> merged some changes after the split, I've synced them and fixed some
> minor issues before using it in the gate jobs.
>
> These are very small changes and I would suggest we review/land them on
> priority (we don't have to keep syncing them again and again for broken
> jobs). We should probably *not approve* any changes to integration tests
> in heat before these go in.
>
> - Changes in heat-tempest-plugin (sync  missing patches and fixes)
> https://review.openstack.org/#/q/project:openstack/heat-tempest-plugin+topic:sync_from_heat
>
> - Use heat-tempest-plugin for integration jobs
> https://review.openstack.org/#/c/508112/
>
> - Use heat-tempest-plugin for grenade job
> https://review.openstack.org/#/c/521246/
>
> - Add heat integration jobs to heat-tempest-plugin check/gate queue
> https://review.openstack.org/#/c/521340/
>    I guess this would result in some issues, where we can't add any
> changes to heat that breaks the existing tests, as the changes for both
> projects would have a circular dependency (not sure how it works atm
> with other plugins!).
>
> - Remove plugin an integration tests from heat (This has -1 atm as
> releasenote job is broken, waiting for infra to fix it)
> https://review.openstack.org/#/c/521263/ 
> <https://review.openstack.org/#/c/521263/>
>
> I've also created an etherpad[3] to track these.
>
> Also, the plugin project is expected to be branchless (we may not
> backport these job changes to stable branches soon though), we've to
> find  a  way to run additional tests for any new feature only on >
> <release/branch>. AFAIK, other projects check api microversions
> supported and without microversions in heat, may be we've find an
> alternate way.

So from discussion on IRC today I learned that the plan agreed at the
PTG was *not* to move all of heat_integrationtests to a separate repo,
but just those tests that test the API and are likely to be used in the
trademark program for verifying clouds.

In my view that effectively means just the Gabbi tests.

However, what is actually happening is that *all* of the
integration/scenario tests are moving to the separate branchless repo.
This looks to me like it will be a disaster for developer and reviewer
productivity, and project quality[1]. (Other folks seem inclined to agree.)

The QA team assures us that the project-wide goal does not preclude
keeping tests that are designed for Heat CI purposes (rather than cloud
verification purposes) in-tree - even continuing to use tempest as a
testrunner. However, it's not at all clear how to reconcile that with
the goal description listing the evils of having an in-tree tempest
plugin.[2] This appears to be the reason for the change in plan.

Does anybody remember what the mechanism for dealing with this that was
agreed at the PTG was?

Basically we need a way to:

* Run a bunch of integration tests in Heat CI jobs
* against a full OpenStack cloud
* preferably using tempest as a testrunner, but this is optional
* without any danger of e.g. breaking tempest test discovery just
because Heat was installed on the same machine as tempest

How do we do it?

thanks,
Zane.

[1]
http://eavesdrop.openstack.org/irclogs/%23heat/%23heat.2017-11-29.log.html#t2017-11-29T15:17:00
[2]
https://governance.openstack.org/tc/goals/queens/split-tempest-plugins.html#split-tempest-plugins-into-separate-repos-projects

> --
> Regards,
> Rabi Mishra
>
> [1]
> https://governance.openstack.org/tc/goals/queens/split-tempest-plugins.html
> [2] https://git.openstack.org/cgit/openstack/heat-tempest-plugin
> [3] https://etherpad.openstack.org/p/heat-tempest-plugin
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [heat][qa] Split tempest plugin from heat

Rabi Mishra
On Wed, Nov 29, 2017 at 9:51 PM, Zane Bitter <[hidden email]> wrote:
On 19/11/17 03:08, Rabi Mishra wrote:
Hi All,

As part of community goal[1] for Queens, we've completed the repo split and created the new project[2].

The next objective to is to use the new plugin in our jobs. As we've merged some changes after the split, I've synced them and fixed some minor issues before using it in the gate jobs.

These are very small changes and I would suggest we review/land them on priority (we don't have to keep syncing them again and again for broken jobs). We should probably *not approve* any changes to integration tests in heat before these go in.

- Changes in heat-tempest-plugin (sync  missing patches and fixes)
https://review.openstack.org/#/q/project:openstack/heat-tempest-plugin+topic:sync_from_heat

- Use heat-tempest-plugin for integration jobs
https://review.openstack.org/#/c/508112/

- Use heat-tempest-plugin for grenade job
https://review.openstack.org/#/c/521246/

- Add heat integration jobs to heat-tempest-plugin check/gate queue
https://review.openstack.org/#/c/521340/
   I guess this would result in some issues, where we can't add any changes to heat that breaks the existing tests, as the changes for both projects would have a circular dependency (not sure how it works atm with other plugins!).

- Remove plugin an integration tests from heat (This has -1 atm as releasenote job is broken, waiting for infra to fix it)
https://review.openstack.org/#/c/521263/ <https://review.openstack.org/#/c/521263/>

I've also created an etherpad[3] to track these.

Also, the plugin project is expected to be branchless (we may not backport these job changes to stable branches soon though), we've to find  a  way to run additional tests for any new feature only on > <release/branch>. AFAIK, other projects check api microversions supported and without microversions in heat, may be we've find an alternate way.

So from discussion on IRC today I learned that the plan agreed at the PTG was *not* to move all of heat_integrationtests to a separate repo, but just those tests that test the API and are likely to be used in the trademark program for verifying clouds.


Well, the etherpad[1] for the session clearly mentions "Create the new repo and import API and *scenario* tests" and *not* "that tests the API and likely to be used in trademark programs".  What you mentioned above is probably the conclusion from all the discussion we had on IRC yesterday.

In my view that effectively means just the Gabbi tests.
 
However, what is actually happening is that *all* of the integration/scenario tests are moving to the separate branchless repo. This looks to me like it will be a disaster for developer and reviewer productivity, and project quality[1]. (Other folks seem inclined to agree.)

Just to make it clear, there had been several discussions on 'what to move'/'branchless or not' in meetings[2] and IRC discussions[3] and the change of plan to move functional tests was based on the following.

- All our tests are API driven. There is probably not much difference between functional and scenario (i.e. if we only move scenario tests, we would have the same kind of issues as we would with functional tests).

- All our tests run with tempest as test runner and use the same tempest configuration. Unless we run remaining in-tree tests without tempest, we would end-up with the same set problems which the goal wanted to address ( issues with in-tree tempest plugins).

- Current coverage of gabbi tests is very poor and we've never been serious to land the patches to complete the API coverage. One of the patches have been languishing in review queue since a year[4].

- As heat is the orchestration service, I would think testing heat from user/operator point of view is not only testing API and interoperability, but how it works with other services deployed and many of our functional tests[5] do that. 

As per my understanding, the issue of having tests in a separate repo and it's impact on developer productivity was discussed by the all concerned before accepting this as a community goal. Our problem is probably not different from some other projects[6].

It's little unfortunate that we've started discussing about these issues after most of stuff has been done. Having said that, it's never late to do the right thing that saves us from a disaster, if that's what it means:)

Given the option, I would prefer:

- We keep all tests in-tree and run them with tempest for the time being, which means we would not complete the goal.

- Complete the coverage for API tests, probably identify functional/scenario tests that are expected to be stable across branches and then move them out along with the API tests.

- Change the remaining in-tree tests to run without tempest at the gate, unless there is a solution to your question below.

[4] https://review.openstack.org/#/c/421914/
[5] https://github.com/openstack/heat/blob/master/heat_integrationtests/functional/test_lbaasv2.py

The QA team assures us that the project-wide goal does not preclude keeping tests that are designed for Heat CI purposes (rather than cloud verification purposes) in-tree - even continuing to use tempest as a testrunner. However, it's not at all clear how to reconcile that with the goal description listing the evils of having an in-tree tempest plugin.[2] This appears to be the reason for the change in plan.

Does anybody remember what the mechanism for dealing with this that was agreed at the PTG was?

Basically we need a way to:

* Run a bunch of integration tests in Heat CI jobs
* against a full OpenStack cloud
* preferably using tempest as a testrunner, but this is optional
* without any danger of e.g. breaking tempest test discovery just because Heat was installed on the same machine as tempest

How do we do it?

thanks,
Zane.

[1] http://eavesdrop.openstack.org/irclogs/%23heat/%23heat.2017-11-29.log.html#t2017-11-29T15:17:00
[2] https://governance.openstack.org/tc/goals/queens/split-tempest-plugins.html#split-tempest-plugins-into-separate-repos-projects

--
Regards,
Rabi Mishra

[1] https://governance.openstack.org/tc/goals/queens/split-tempest-plugins.html
[2] https://git.openstack.org/cgit/openstack/heat-tempest-plugin
[3] https://etherpad.openstack.org/p/heat-tempest-plugin


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@...enstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@...enstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Regards,
Rabi Mishra


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [heat][qa] Split tempest plugin from heat

Rico Lin-2
In reply to this post by Zane Bitter
>
> Does anybody remember what the mechanism for dealing with this that was agreed at the PTG was?

The etherpad [1] shows actions that we(me and Thomas) agree to do, and that's all for PTG discussion (the online attendee number is 0 for that session).
Following with a recap in meeting to share the task and include that etherpad (which Rabi's agreement on fallow). And with further implement and research (by Rabi), we face the test separation issue, as we discussed yesterday, Rabi reaches out for suggestion and opinions,  and I gave mine with agreeing to move all test to the new repo and giving my +2 review to those patches.
IMO, that's a question of whether we consider our tests as part of tempest plugin or we want to consider them as internal Heat CI tests. 

Right now what we have is following:
1. A new repo for heat-tempest-plugin with all tests in it.
2. Switch heat's functional and grenade jobs (in master) to use new heat tempest plugin.
3. a patch to propose to remove integration tests in heat.
4. needs a plan to match branchless for heat-tempest-plugin repo

>
> Basically we need a way to:
>
> * Run a bunch of integration tests in Heat CI jobs
> * against a full OpenStack cloud
> * preferably using tempest as a testrunner, but this is optional
> * without any danger of e.g. breaking tempest test discovery just because Heat was installed on the same machine as tempest
>
> How do we do it?

We now use tempest for all our integration tests but seems the question now is how can we let our the CI out of this process, so all developers in heat will have a better life while users/ops will also have a better life when testing against their OpenStack environment.

My propose is we separate our tests into two groups:
A group of tests that should only stay in Heat CI jobs, which we still can use tempest as test runner, but we should mark it(with a description, a tag, or any way) as for internal CI usage, which will not follow any TC goals since that's only for heat internal usage.

Another group of tests that make sense for us to expose to users and operators. These stay in heat tempest plugin which users/ops can use to test their own environment, and we will meet what the users/ops might expect when they testing with their own heat environment. Also means we should keep this repo branchless and fully tested with each branch. So we stick with that goal. API tests seem to be native in this group, but we have to keep in mind that it's not covering all cases, so we might need to consider to add more tests in tempest plugin (in long-term).

In turns of actions:

1. Create a new tempest plugin for internal CI usage (in heat repo) and mark it as internal(as a declare for we do not consider this as a tempest plugin for users/ops), so others will know that they should not use it for any other way(or use at their own risk).  
2. Modify the patch which proposes to remove integration tests in heat, and keeps tests those tests we decided as internal CI tests.
3. Allow heat gate to run tests from both tempest plugin( internal one and `heat-tempest-plugin`) so we will still run and check both works. Of course, we can keep the same group of tests on both sides(like a sync job) but don't see why that's the way we like.

We should have this discussion earlier (when a simple suggestion from any one of us will do), but since we still under developing cycle that means it's not too late for anything.
And TBH, as a developer(not a good one but still learning), I see this goal as pure painful plus useless for developers in heat, but I trust TCs and QA team when they think this is so important (on how this will help users/ops when testing their environment) that they decided to make this as a goal for our community.

I'm fine with whatever the solution we came out from here, and I'm happy to implement it together, but please keep in mind that we need all to help us track the progress, so we can have more opinions or suggestions for any issues we are/will face on these tasks.

Let's do this together and make it right at once!

Would really like to hear more suggestions on how we can deal with this or how other teams do, so please share with us if you(heat/QA member or not) see this mail.



__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [heat][qa] Split tempest plugin from heat

Zane Bitter
OK, an update on this. The current plan of record after the team meeting
today goes something like this:

1) Identify tests that are moving to the new plugin repo, and remove
them from the heat repo.
2) Start using the utils (base classes &c.) from the plugin repo in the
remaining heat tests.
3) Find a way to run both the plugin tests and the in-repo tests as part
of the same test run in the gate.
4) Remove the tests that are _not_ moving from the plugin repo.

There's been some interesting related discussion on
https://review.openstack.org/#/c/521602/ that I'd encourage everyone to
read. I like the idea of a configurable plugin system that's independent
of Tempest to solve (3), but we'll see what comes out of that
discussion. If that doesn't pan out then we can probably still do it the
old-fashioned way.

I made a first pass at categorising the tests based on the categories I
identified in that discussion here:

https://etherpad.openstack.org/p/heat-integration-test-categories

If other Heat folks could take a look and comment on/move stuff they
think is wrong, that would be helpful. In particular, I'm pretty
uncertain whether most of the things I listed under Interop Tests are
actually good candidates for interop testing, or if they should just be
regression tests.

thanks,
Zane.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev