[deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

Emilien Macchi-4
I missed [tripleo] tag.

On Thu, Jun 15, 2017 at 12:09 PM, Emilien Macchi <[hidden email]> wrote:

> If you haven't followed the "Configuration management with etcd /
> confd" thread [1], Doug found out that using confd to generate
> configuration files wouldn't work for the Cinder case where we don't
> know in advance of the deployment what settings to tell confd to look
> at.
> We are still looking for a generic way to generate *.conf files for
> OpenStack, that would be usable by Deployment tools and operators.
> Right now, Doug and I are investigating some tooling that would be
> useful to achieve this goal.
>
> Doug has prototyped an Ansible role that would generate configuration
> files by consumming 2 things:
>
> * Configuration schema, generated by Ben's work with Machine Readable
> Sample Config.
>   $ oslo-config-generator --namespace cinder --format yaml > cinder-schema.yaml
>
> It also needs: https://review.openstack.org/#/c/474306/ to generate
> some extra data not included in the original version.
>
> * Parameters values provided in config_data directly in the playbook:
>    config_data:
>      DEFAULT:
>        transport_url: rabbit://user:password@hostname
>        verbose: true
>
> There are 2 options disabled by default but which would be useful for
> production environments:
> * Set to true to always show all configuration values: config_show_defaults
> * Set to true to show the help text: config_show_help: true
>
> The Ansible module is available on github:
> https://github.com/dhellmann/oslo-config-ansible
>
> To try this out, just run:
>   $ ansible-playbook ./playbook.yml
>
> You can quickly see the output of cinder.conf:
>     https://clbin.com/HmS58
>
>
> What are the next steps:
>
> * Getting feedback from Deployment Tools and operators on the concept
> of this module.
>   Maybe this module could replace what is done by Kolla with
> merge_configs and OpenStack Ansible with config_template.
> * On the TripleO side, we would like to see if this module could
> replace the Puppet OpenStack modules that are now mostly used for
> generating configuration files for containers.
>   A transition path would be having Heat to generate Ansible vars
> files and give it to this module. We could integrate the playbook into
> a new task in the composable services, something like
>   "os_gen_config_tasks", a bit like we already have for upgrade tasks,
> also driven by Ansible.
> * Another similar option to what Doug did is to write a standalone
> tool that would generate configuration, and for Ansible users we would
> write a new module to use this tool.
>   Example:
>       Step 1. oslo-config-generator --namespace cinder --format yaml >
> cinder-schema.yaml (note this tool already exists)
>       Step 2. Create config_data.yaml in a specific format with
> parameters values for what we want to configure (note this format
> doesn't exist yet but look at what Doug did in the role, we could use
> the same kind of schema).
>       Step 3. oslo-gen-config -i config_data.yaml -s schema.yaml >
> cinder.conf (note this tool doesn't exist yet)
>
>       For Ansible users, we would write an Ansible module that would
> take in entry 2 files: the schema and the data. The module would just
> run the tool provided by oslo.config.
>       Example:
>           - name: Generate cinder.conf
>             oslo-gen-config: schema=cinder-schema.yaml
>                                        data=config_data.yaml
>
>
> Please bring feedback and thoughts, it's really important to know what
> folks from Installers think about this idea; again the ultimate goal
> is to provide a reference tool to generate configuration in OpenStack,
> in a way that scales and is friendly for our operators.
>
> Thanks,
>
> [1] http://lists.openstack.org/pipermail/openstack-dev/2017-June/118176.html
> --
> Emilien Macchi



--
Emilien Macchi

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

Jiří Stránský
On 15.6.2017 19:06, Emilien Macchi wrote:

> I missed [tripleo] tag.
>
> On Thu, Jun 15, 2017 at 12:09 PM, Emilien Macchi <[hidden email]> wrote:
>> If you haven't followed the "Configuration management with etcd /
>> confd" thread [1], Doug found out that using confd to generate
>> configuration files wouldn't work for the Cinder case where we don't
>> know in advance of the deployment what settings to tell confd to look
>> at.
>> We are still looking for a generic way to generate *.conf files for
>> OpenStack, that would be usable by Deployment tools and operators.
>> Right now, Doug and I are investigating some tooling that would be
>> useful to achieve this goal.
>>
>> Doug has prototyped an Ansible role that would generate configuration
>> files by consumming 2 things:
>>
>> * Configuration schema, generated by Ben's work with Machine Readable
>> Sample Config.
>>    $ oslo-config-generator --namespace cinder --format yaml > cinder-schema.yaml
>>
>> It also needs: https://review.openstack.org/#/c/474306/ to generate
>> some extra data not included in the original version.
>>
>> * Parameters values provided in config_data directly in the playbook:
>>     config_data:
>>       DEFAULT:
>>         transport_url: rabbit://user:password@hostname
>>         verbose: true
>>
>> There are 2 options disabled by default but which would be useful for
>> production environments:
>> * Set to true to always show all configuration values: config_show_defaults
>> * Set to true to show the help text: config_show_help: true
>>
>> The Ansible module is available on github:
>> https://github.com/dhellmann/oslo-config-ansible
>>
>> To try this out, just run:
>>    $ ansible-playbook ./playbook.yml
>>
>> You can quickly see the output of cinder.conf:
>>      https://clbin.com/HmS58
>>
>>
>> What are the next steps:
>>
>> * Getting feedback from Deployment Tools and operators on the concept
>> of this module.
>>    Maybe this module could replace what is done by Kolla with
>> merge_configs and OpenStack Ansible with config_template.
>> * On the TripleO side, we would like to see if this module could
>> replace the Puppet OpenStack modules that are now mostly used for
>> generating configuration files for containers.
>>    A transition path would be having Heat to generate Ansible vars
>> files and give it to this module. We could integrate the playbook into
>> a new task in the composable services, something like
>>    "os_gen_config_tasks", a bit like we already have for upgrade tasks,
>> also driven by Ansible.

This sounds good to me, though one issue i can presently see is that
Puppet modules sometimes contain quite a bit of data processing logic
("smart" variables which map 1-to-N rather than 1-to-1 to actual config
values, and often not just in openstack service configs, e.g.
puppet-nova also configures libvirt, etc.). Also we use some non-config
aspects from the Puppet modules (e.g. seeding Keystone
tenants/services/endpoints/...). We'd need to implement this
functionality elsewhere when replacing the Puppet modules. Not a
blocker, but something to keep in mind.

>> * Another similar option to what Doug did is to write a standalone
>> tool that would generate configuration, and for Ansible users we would
>> write a new module to use this tool.
>>    Example:
>>        Step 1. oslo-config-generator --namespace cinder --format yaml >
>> cinder-schema.yaml (note this tool already exists)
>>        Step 2. Create config_data.yaml in a specific format with
>> parameters values for what we want to configure (note this format
>> doesn't exist yet but look at what Doug did in the role, we could use
>> the same kind of schema).
>>        Step 3. oslo-gen-config -i config_data.yaml -s schema.yaml >
>> cinder.conf (note this tool doesn't exist yet)

+1 on standalone tool which can be used in different contexts (by
different higher level tools), this sounds generally useful.

>>
>>        For Ansible users, we would write an Ansible module that would
>> take in entry 2 files: the schema and the data. The module would just
>> run the tool provided by oslo.config.
>>        Example:
>>            - name: Generate cinder.conf
>>              oslo-gen-config: schema=cinder-schema.yaml
>>                                         data=config_data.yaml

+1 for module rather than a role. "Take these inputs and produce that
output" fits the module semantics better than role semantics IMO.

FWIW as i see it right now, this ^^ + ConfigMaps + immutable-config
containers could result in a nicer/safer/more-debuggable containerized
OpenStack setup than etcd + confd in daemon mode + mutable-config
containers.

>>
>>
>> Please bring feedback and thoughts, it's really important to know what
>> folks from Installers think about this idea; again the ultimate goal
>> is to provide a reference tool to generate configuration in OpenStack,
>> in a way that scales and is friendly for our operators.
>>
>> Thanks,
>>
>> [1] http://lists.openstack.org/pipermail/openstack-dev/2017-June/118176.html
>> --
>> Emilien Macchi
>
>
>

Have a good day,

Jirka

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

Emilien Macchi-4
On Fri, Jun 16, 2017 at 11:09 AM, Jiří Stránský <[hidden email]> wrote:

> On 15.6.2017 19:06, Emilien Macchi wrote:
>>
>> I missed [tripleo] tag.
>>
>> On Thu, Jun 15, 2017 at 12:09 PM, Emilien Macchi <[hidden email]>
>> wrote:
>>>
>>> If you haven't followed the "Configuration management with etcd /
>>> confd" thread [1], Doug found out that using confd to generate
>>> configuration files wouldn't work for the Cinder case where we don't
>>> know in advance of the deployment what settings to tell confd to look
>>> at.
>>> We are still looking for a generic way to generate *.conf files for
>>> OpenStack, that would be usable by Deployment tools and operators.
>>> Right now, Doug and I are investigating some tooling that would be
>>> useful to achieve this goal.
>>>
>>> Doug has prototyped an Ansible role that would generate configuration
>>> files by consumming 2 things:
>>>
>>> * Configuration schema, generated by Ben's work with Machine Readable
>>> Sample Config.
>>>    $ oslo-config-generator --namespace cinder --format yaml >
>>> cinder-schema.yaml
>>>
>>> It also needs: https://review.openstack.org/#/c/474306/ to generate
>>> some extra data not included in the original version.
>>>
>>> * Parameters values provided in config_data directly in the playbook:
>>>     config_data:
>>>       DEFAULT:
>>>         transport_url: rabbit://user:password@hostname
>>>         verbose: true
>>>
>>> There are 2 options disabled by default but which would be useful for
>>> production environments:
>>> * Set to true to always show all configuration values:
>>> config_show_defaults
>>> * Set to true to show the help text: config_show_help: true
>>>
>>> The Ansible module is available on github:
>>> https://github.com/dhellmann/oslo-config-ansible
>>>
>>> To try this out, just run:
>>>    $ ansible-playbook ./playbook.yml
>>>
>>> You can quickly see the output of cinder.conf:
>>>      https://clbin.com/HmS58
>>>
>>>
>>> What are the next steps:
>>>
>>> * Getting feedback from Deployment Tools and operators on the concept
>>> of this module.
>>>    Maybe this module could replace what is done by Kolla with
>>> merge_configs and OpenStack Ansible with config_template.
>>> * On the TripleO side, we would like to see if this module could
>>> replace the Puppet OpenStack modules that are now mostly used for
>>> generating configuration files for containers.
>>>    A transition path would be having Heat to generate Ansible vars
>>> files and give it to this module. We could integrate the playbook into
>>> a new task in the composable services, something like
>>>    "os_gen_config_tasks", a bit like we already have for upgrade tasks,
>>> also driven by Ansible.
>
>
> This sounds good to me, though one issue i can presently see is that Puppet
> modules sometimes contain quite a bit of data processing logic ("smart"
> variables which map 1-to-N rather than 1-to-1 to actual config values, and
> often not just in openstack service configs, e.g. puppet-nova also
> configures libvirt, etc.). Also we use some non-config aspects from the
> Puppet modules (e.g. seeding Keystone tenants/services/endpoints/...). We'd
> need to implement this functionality elsewhere when replacing the Puppet
> modules. Not a blocker, but something to keep in mind.

2 interesting things:

- For the logic that are done by puppet modules for some parameters:
yes I agree, this problem isn't solved now. This thread talks about
config management, with some data in entry, it's a very little step I
know, but it's on purpose.
  Once we figure how to do that, we can think about the data
generation and where to put the logic (I think the logic is too
opinionated to be in a common project, but I might be wrong).

- Things like libvirt, mysql, etc will be managed by something else
but Puppet I think; this is out of topic for now. For Keystone
resources, same thing, we could use some native python clients or
Ansible modules if we switch to Ansible, etc.

Again, topic is really on "give me an ini file".

>>> * Another similar option to what Doug did is to write a standalone
>>> tool that would generate configuration, and for Ansible users we would
>>> write a new module to use this tool.
>>>    Example:
>>>        Step 1. oslo-config-generator --namespace cinder --format yaml >
>>> cinder-schema.yaml (note this tool already exists)
>>>        Step 2. Create config_data.yaml in a specific format with
>>> parameters values for what we want to configure (note this format
>>> doesn't exist yet but look at what Doug did in the role, we could use
>>> the same kind of schema).
>>>        Step 3. oslo-gen-config -i config_data.yaml -s schema.yaml >
>>> cinder.conf (note this tool doesn't exist yet)
>
>
> +1 on standalone tool which can be used in different contexts (by different
> higher level tools), this sounds generally useful.

Ack, good feedback.

>>>
>>>        For Ansible users, we would write an Ansible module that would
>>> take in entry 2 files: the schema and the data. The module would just
>>> run the tool provided by oslo.config.
>>>        Example:
>>>            - name: Generate cinder.conf
>>>              oslo-gen-config: schema=cinder-schema.yaml
>>>                                         data=config_data.yaml
>
>
> +1 for module rather than a role. "Take these inputs and produce that
> output" fits the module semantics better than role semantics IMO.
>
> FWIW as i see it right now, this ^^ + ConfigMaps + immutable-config
> containers could result in a nicer/safer/more-debuggable containerized
> OpenStack setup than etcd + confd in daemon mode + mutable-config
> containers.

Good feedback, I'm also looking forward to hear from Kolla, OSA and Helm folks.

>>>
>>>
>>> Please bring feedback and thoughts, it's really important to know what
>>> folks from Installers think about this idea; again the ultimate goal
>>> is to provide a reference tool to generate configuration in OpenStack,
>>> in a way that scales and is friendly for our operators.
>>>
>>> Thanks,
>>>
>>> [1]
>>> http://lists.openstack.org/pipermail/openstack-dev/2017-June/118176.html
>>> --
>>> Emilien Macchi
>>
>>
>>
>>
>
> Have a good day,
>
> Jirka
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Emilien Macchi

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

Michał Jastrzębski
So I'm trying to figure out how to actually use it.

We (and any other container based deploy..) will run into some
chicken/egg problem - you need to deploy container to generate big
yaml with defaults, then you need to overload it with your
configurations, validate if they're not deprecated, run container with
this ansible role (or module...really doesn't matter), spit out final
confg, lay it down, deploy container again. And that will have to be
done for every host class (as configs might differ host to host). Imho
a bit too much for this to be appealing (but I might be wrong). I'd
much rather have:
1. Yaml as input to oslo.config instead of broken ini
2. Validator to throw an error if one of our regular,
template-rendered, configs is deprecated

We can run this validator in gate to have quick feedback when
something gets deprecated.

Thoughts?
Michal

On 16 June 2017 at 13:24, Emilien Macchi <[hidden email]> wrote:

> On Fri, Jun 16, 2017 at 11:09 AM, Jiří Stránský <[hidden email]> wrote:
>> On 15.6.2017 19:06, Emilien Macchi wrote:
>>>
>>> I missed [tripleo] tag.
>>>
>>> On Thu, Jun 15, 2017 at 12:09 PM, Emilien Macchi <[hidden email]>
>>> wrote:
>>>>
>>>> If you haven't followed the "Configuration management with etcd /
>>>> confd" thread [1], Doug found out that using confd to generate
>>>> configuration files wouldn't work for the Cinder case where we don't
>>>> know in advance of the deployment what settings to tell confd to look
>>>> at.
>>>> We are still looking for a generic way to generate *.conf files for
>>>> OpenStack, that would be usable by Deployment tools and operators.
>>>> Right now, Doug and I are investigating some tooling that would be
>>>> useful to achieve this goal.
>>>>
>>>> Doug has prototyped an Ansible role that would generate configuration
>>>> files by consumming 2 things:
>>>>
>>>> * Configuration schema, generated by Ben's work with Machine Readable
>>>> Sample Config.
>>>>    $ oslo-config-generator --namespace cinder --format yaml >
>>>> cinder-schema.yaml
>>>>
>>>> It also needs: https://review.openstack.org/#/c/474306/ to generate
>>>> some extra data not included in the original version.
>>>>
>>>> * Parameters values provided in config_data directly in the playbook:
>>>>     config_data:
>>>>       DEFAULT:
>>>>         transport_url: rabbit://user:password@hostname
>>>>         verbose: true
>>>>
>>>> There are 2 options disabled by default but which would be useful for
>>>> production environments:
>>>> * Set to true to always show all configuration values:
>>>> config_show_defaults
>>>> * Set to true to show the help text: config_show_help: true
>>>>
>>>> The Ansible module is available on github:
>>>> https://github.com/dhellmann/oslo-config-ansible
>>>>
>>>> To try this out, just run:
>>>>    $ ansible-playbook ./playbook.yml
>>>>
>>>> You can quickly see the output of cinder.conf:
>>>>      https://clbin.com/HmS58
>>>>
>>>>
>>>> What are the next steps:
>>>>
>>>> * Getting feedback from Deployment Tools and operators on the concept
>>>> of this module.
>>>>    Maybe this module could replace what is done by Kolla with
>>>> merge_configs and OpenStack Ansible with config_template.
>>>> * On the TripleO side, we would like to see if this module could
>>>> replace the Puppet OpenStack modules that are now mostly used for
>>>> generating configuration files for containers.
>>>>    A transition path would be having Heat to generate Ansible vars
>>>> files and give it to this module. We could integrate the playbook into
>>>> a new task in the composable services, something like
>>>>    "os_gen_config_tasks", a bit like we already have for upgrade tasks,
>>>> also driven by Ansible.
>>
>>
>> This sounds good to me, though one issue i can presently see is that Puppet
>> modules sometimes contain quite a bit of data processing logic ("smart"
>> variables which map 1-to-N rather than 1-to-1 to actual config values, and
>> often not just in openstack service configs, e.g. puppet-nova also
>> configures libvirt, etc.). Also we use some non-config aspects from the
>> Puppet modules (e.g. seeding Keystone tenants/services/endpoints/...). We'd
>> need to implement this functionality elsewhere when replacing the Puppet
>> modules. Not a blocker, but something to keep in mind.
>
> 2 interesting things:
>
> - For the logic that are done by puppet modules for some parameters:
> yes I agree, this problem isn't solved now. This thread talks about
> config management, with some data in entry, it's a very little step I
> know, but it's on purpose.
>   Once we figure how to do that, we can think about the data
> generation and where to put the logic (I think the logic is too
> opinionated to be in a common project, but I might be wrong).
>
> - Things like libvirt, mysql, etc will be managed by something else
> but Puppet I think; this is out of topic for now. For Keystone
> resources, same thing, we could use some native python clients or
> Ansible modules if we switch to Ansible, etc.
>
> Again, topic is really on "give me an ini file".
>
>>>> * Another similar option to what Doug did is to write a standalone
>>>> tool that would generate configuration, and for Ansible users we would
>>>> write a new module to use this tool.
>>>>    Example:
>>>>        Step 1. oslo-config-generator --namespace cinder --format yaml >
>>>> cinder-schema.yaml (note this tool already exists)
>>>>        Step 2. Create config_data.yaml in a specific format with
>>>> parameters values for what we want to configure (note this format
>>>> doesn't exist yet but look at what Doug did in the role, we could use
>>>> the same kind of schema).
>>>>        Step 3. oslo-gen-config -i config_data.yaml -s schema.yaml >
>>>> cinder.conf (note this tool doesn't exist yet)
>>
>>
>> +1 on standalone tool which can be used in different contexts (by different
>> higher level tools), this sounds generally useful.
>
> Ack, good feedback.
>
>>>>
>>>>        For Ansible users, we would write an Ansible module that would
>>>> take in entry 2 files: the schema and the data. The module would just
>>>> run the tool provided by oslo.config.
>>>>        Example:
>>>>            - name: Generate cinder.conf
>>>>              oslo-gen-config: schema=cinder-schema.yaml
>>>>                                         data=config_data.yaml
>>
>>
>> +1 for module rather than a role. "Take these inputs and produce that
>> output" fits the module semantics better than role semantics IMO.
>>
>> FWIW as i see it right now, this ^^ + ConfigMaps + immutable-config
>> containers could result in a nicer/safer/more-debuggable containerized
>> OpenStack setup than etcd + confd in daemon mode + mutable-config
>> containers.
>
> Good feedback, I'm also looking forward to hear from Kolla, OSA and Helm folks.
>
>>>>
>>>>
>>>> Please bring feedback and thoughts, it's really important to know what
>>>> folks from Installers think about this idea; again the ultimate goal
>>>> is to provide a reference tool to generate configuration in OpenStack,
>>>> in a way that scales and is friendly for our operators.
>>>>
>>>> Thanks,
>>>>
>>>> [1]
>>>> http://lists.openstack.org/pipermail/openstack-dev/2017-June/118176.html
>>>> --
>>>> Emilien Macchi
>>>
>>>
>>>
>>>
>>
>> Have a good day,
>>
>> Jirka
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: [hidden email]?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Emilien Macchi
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

Doug Hellmann-2
Excerpts from Michał Jastrzębski's message of 2017-06-16 15:50:54 -0700:
> So I'm trying to figure out how to actually use it.
>
> We (and any other container based deploy..) will run into some
> chicken/egg problem - you need to deploy container to generate big
> yaml with defaults, then you need to overload it with your

The config schema file (the "big YAML with defaults") should be part of
the packaged software, so the deployment tool shouldn't need to generate
it unless you're handling drivers that are not included in tree.

> configurations, validate if they're not deprecated, run container with

It doesn't do it today, but the thing that converts the input data to
the INI file could automatically translate old option names to their new
names.

> this ansible role (or module...really doesn't matter), spit out final

Why does the config file need to be generated inside a container?

> confg, lay it down, deploy container again. And that will have to be
> done for every host class (as configs might differ host to host). Imho
> a bit too much for this to be appealing (but I might be wrong). I'd
> much rather have:
> 1. Yaml as input to oslo.config instead of broken ini

I'm not opposed to switching to YAML, but it's a bit more involved than
just adding support in the parser. All of the work that has been done on
generating sample default files and documentation needs to be updated to
support YAML. We need a migration path to move everyone from INI to
YAML. And we need to update devstack and all of its plugins to edit the
new file format. There are probably more tasks involved in the
migration. I'm dealing with a couple of other projects right now, and
don't have time to plan all of that out myself. If someone else wants to
pick it up, I can help with reviews on the spec and code changes.

> 2. Validator to throw an error if one of our regular,
> template-rendered, configs is deprecated
>
> We can run this validator in gate to have quick feedback when
> something gets deprecated.
>
> Thoughts?
> Michal
>
> On 16 June 2017 at 13:24, Emilien Macchi <[hidden email]> wrote:
> > On Fri, Jun 16, 2017 at 11:09 AM, Jiří Stránský <[hidden email]> wrote:
> >> On 15.6.2017 19:06, Emilien Macchi wrote:
> >>>
> >>> I missed [tripleo] tag.
> >>>
> >>> On Thu, Jun 15, 2017 at 12:09 PM, Emilien Macchi <[hidden email]>
> >>> wrote:
> >>>>
> >>>> If you haven't followed the "Configuration management with etcd /
> >>>> confd" thread [1], Doug found out that using confd to generate
> >>>> configuration files wouldn't work for the Cinder case where we don't
> >>>> know in advance of the deployment what settings to tell confd to look
> >>>> at.
> >>>> We are still looking for a generic way to generate *.conf files for
> >>>> OpenStack, that would be usable by Deployment tools and operators.
> >>>> Right now, Doug and I are investigating some tooling that would be
> >>>> useful to achieve this goal.
> >>>>
> >>>> Doug has prototyped an Ansible role that would generate configuration
> >>>> files by consumming 2 things:
> >>>>
> >>>> * Configuration schema, generated by Ben's work with Machine Readable
> >>>> Sample Config.
> >>>>    $ oslo-config-generator --namespace cinder --format yaml >
> >>>> cinder-schema.yaml
> >>>>
> >>>> It also needs: https://review.openstack.org/#/c/474306/ to generate
> >>>> some extra data not included in the original version.
> >>>>
> >>>> * Parameters values provided in config_data directly in the playbook:
> >>>>     config_data:
> >>>>       DEFAULT:
> >>>>         transport_url: rabbit://user:password@hostname
> >>>>         verbose: true
> >>>>
> >>>> There are 2 options disabled by default but which would be useful for
> >>>> production environments:
> >>>> * Set to true to always show all configuration values:
> >>>> config_show_defaults
> >>>> * Set to true to show the help text: config_show_help: true
> >>>>
> >>>> The Ansible module is available on github:
> >>>> https://github.com/dhellmann/oslo-config-ansible
> >>>>
> >>>> To try this out, just run:
> >>>>    $ ansible-playbook ./playbook.yml
> >>>>
> >>>> You can quickly see the output of cinder.conf:
> >>>>      https://clbin.com/HmS58
> >>>>
> >>>>
> >>>> What are the next steps:
> >>>>
> >>>> * Getting feedback from Deployment Tools and operators on the concept
> >>>> of this module.
> >>>>    Maybe this module could replace what is done by Kolla with
> >>>> merge_configs and OpenStack Ansible with config_template.
> >>>> * On the TripleO side, we would like to see if this module could
> >>>> replace the Puppet OpenStack modules that are now mostly used for
> >>>> generating configuration files for containers.
> >>>>    A transition path would be having Heat to generate Ansible vars
> >>>> files and give it to this module. We could integrate the playbook into
> >>>> a new task in the composable services, something like
> >>>>    "os_gen_config_tasks", a bit like we already have for upgrade tasks,
> >>>> also driven by Ansible.
> >>
> >>
> >> This sounds good to me, though one issue i can presently see is that Puppet
> >> modules sometimes contain quite a bit of data processing logic ("smart"
> >> variables which map 1-to-N rather than 1-to-1 to actual config values, and
> >> often not just in openstack service configs, e.g. puppet-nova also
> >> configures libvirt, etc.). Also we use some non-config aspects from the
> >> Puppet modules (e.g. seeding Keystone tenants/services/endpoints/...). We'd
> >> need to implement this functionality elsewhere when replacing the Puppet
> >> modules. Not a blocker, but something to keep in mind.
> >
> > 2 interesting things:
> >
> > - For the logic that are done by puppet modules for some parameters:
> > yes I agree, this problem isn't solved now. This thread talks about
> > config management, with some data in entry, it's a very little step I
> > know, but it's on purpose.
> >   Once we figure how to do that, we can think about the data
> > generation and where to put the logic (I think the logic is too
> > opinionated to be in a common project, but I might be wrong).
> >
> > - Things like libvirt, mysql, etc will be managed by something else
> > but Puppet I think; this is out of topic for now. For Keystone
> > resources, same thing, we could use some native python clients or
> > Ansible modules if we switch to Ansible, etc.
> >
> > Again, topic is really on "give me an ini file".
> >
> >>>> * Another similar option to what Doug did is to write a standalone
> >>>> tool that would generate configuration, and for Ansible users we would
> >>>> write a new module to use this tool.
> >>>>    Example:
> >>>>        Step 1. oslo-config-generator --namespace cinder --format yaml >
> >>>> cinder-schema.yaml (note this tool already exists)
> >>>>        Step 2. Create config_data.yaml in a specific format with
> >>>> parameters values for what we want to configure (note this format
> >>>> doesn't exist yet but look at what Doug did in the role, we could use
> >>>> the same kind of schema).
> >>>>        Step 3. oslo-gen-config -i config_data.yaml -s schema.yaml >
> >>>> cinder.conf (note this tool doesn't exist yet)
> >>
> >>
> >> +1 on standalone tool which can be used in different contexts (by different
> >> higher level tools), this sounds generally useful.
> >
> > Ack, good feedback.
> >
> >>>>
> >>>>        For Ansible users, we would write an Ansible module that would
> >>>> take in entry 2 files: the schema and the data. The module would just
> >>>> run the tool provided by oslo.config.
> >>>>        Example:
> >>>>            - name: Generate cinder.conf
> >>>>              oslo-gen-config: schema=cinder-schema.yaml
> >>>>                                         data=config_data.yaml
> >>
> >>
> >> +1 for module rather than a role. "Take these inputs and produce that
> >> output" fits the module semantics better than role semantics IMO.
> >>
> >> FWIW as i see it right now, this ^^ + ConfigMaps + immutable-config
> >> containers could result in a nicer/safer/more-debuggable containerized
> >> OpenStack setup than etcd + confd in daemon mode + mutable-config
> >> containers.
> >
> > Good feedback, I'm also looking forward to hear from Kolla, OSA and Helm folks.
> >
> >>>>
> >>>>
> >>>> Please bring feedback and thoughts, it's really important to know what
> >>>> folks from Installers think about this idea; again the ultimate goal
> >>>> is to provide a reference tool to generate configuration in OpenStack,
> >>>> in a way that scales and is friendly for our operators.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> [1]
> >>>> http://lists.openstack.org/pipermail/openstack-dev/2017-June/118176.html
> >>>> --
> >>>> Emilien Macchi
> >>>
> >>>
> >>>
> >>>
> >>
> >> Have a good day,
> >>
> >> Jirka
> >>
> >> __________________________________________________________________________
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: [hidden email]?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> > --
> > Emilien Macchi
> >
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: [hidden email]?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

Michał Jastrzębski
On 19 June 2017 at 06:05, Doug Hellmann <[hidden email]> wrote:

> Excerpts from Michał Jastrzębski's message of 2017-06-16 15:50:54 -0700:
>> So I'm trying to figure out how to actually use it.
>>
>> We (and any other container based deploy..) will run into some
>> chicken/egg problem - you need to deploy container to generate big
>> yaml with defaults, then you need to overload it with your
>
> The config schema file (the "big YAML with defaults") should be part of
> the packaged software, so the deployment tool shouldn't need to generate
> it unless you're handling drivers that are not included in tree.

Right that's what I was missing, I guess we can generate these during
buildtime without big issues, then it will be embedded into container,
shouldn't be too hard change and would work for both source and
binary.
>> configurations, validate if they're not deprecated, run container with
>
> It doesn't do it today, but the thing that converts the input data to
> the INI file could automatically translate old option names to their new
> names.
>
>> this ansible role (or module...really doesn't matter), spit out final
>
> Why does the config file need to be generated inside a container?

Outside of container you don't have oslo or nova (python libs), so to
get access to these you need to do it inside container.

>> confg, lay it down, deploy container again. And that will have to be
>> done for every host class (as configs might differ host to host). Imho
>> a bit too much for this to be appealing (but I might be wrong). I'd
>> much rather have:
>> 1. Yaml as input to oslo.config instead of broken ini
>
> I'm not opposed to switching to YAML, but it's a bit more involved than
> just adding support in the parser. All of the work that has been done on
> generating sample default files and documentation needs to be updated to
> support YAML. We need a migration path to move everyone from INI to
> YAML. And we need to update devstack and all of its plugins to edit the
> new file format. There are probably more tasks involved in the
> migration. I'm dealing with a couple of other projects right now, and
> don't have time to plan all of that out myself. If someone else wants to
> pick it up, I can help with reviews on the spec and code changes.

Switching is a big no, everyone would hate us with emotion pure as
mountain spring water. It's to support both at same time, which makes
it slightly more complex. We could make full switch after few releases
of deprecation I guess. Anyway, agree, lots of work.

>
>> 2. Validator to throw an error if one of our regular,
>> template-rendered, configs is deprecated
>>
>> We can run this validator in gate to have quick feedback when
>> something gets deprecated.
>>
>> Thoughts?
>> Michal
>>
>> On 16 June 2017 at 13:24, Emilien Macchi <[hidden email]> wrote:
>> > On Fri, Jun 16, 2017 at 11:09 AM, Jiří Stránský <[hidden email]> wrote:
>> >> On 15.6.2017 19:06, Emilien Macchi wrote:
>> >>>
>> >>> I missed [tripleo] tag.
>> >>>
>> >>> On Thu, Jun 15, 2017 at 12:09 PM, Emilien Macchi <[hidden email]>
>> >>> wrote:
>> >>>>
>> >>>> If you haven't followed the "Configuration management with etcd /
>> >>>> confd" thread [1], Doug found out that using confd to generate
>> >>>> configuration files wouldn't work for the Cinder case where we don't
>> >>>> know in advance of the deployment what settings to tell confd to look
>> >>>> at.
>> >>>> We are still looking for a generic way to generate *.conf files for
>> >>>> OpenStack, that would be usable by Deployment tools and operators.
>> >>>> Right now, Doug and I are investigating some tooling that would be
>> >>>> useful to achieve this goal.
>> >>>>
>> >>>> Doug has prototyped an Ansible role that would generate configuration
>> >>>> files by consumming 2 things:
>> >>>>
>> >>>> * Configuration schema, generated by Ben's work with Machine Readable
>> >>>> Sample Config.
>> >>>>    $ oslo-config-generator --namespace cinder --format yaml >
>> >>>> cinder-schema.yaml
>> >>>>
>> >>>> It also needs: https://review.openstack.org/#/c/474306/ to generate
>> >>>> some extra data not included in the original version.
>> >>>>
>> >>>> * Parameters values provided in config_data directly in the playbook:
>> >>>>     config_data:
>> >>>>       DEFAULT:
>> >>>>         transport_url: rabbit://user:password@hostname
>> >>>>         verbose: true
>> >>>>
>> >>>> There are 2 options disabled by default but which would be useful for
>> >>>> production environments:
>> >>>> * Set to true to always show all configuration values:
>> >>>> config_show_defaults
>> >>>> * Set to true to show the help text: config_show_help: true
>> >>>>
>> >>>> The Ansible module is available on github:
>> >>>> https://github.com/dhellmann/oslo-config-ansible
>> >>>>
>> >>>> To try this out, just run:
>> >>>>    $ ansible-playbook ./playbook.yml
>> >>>>
>> >>>> You can quickly see the output of cinder.conf:
>> >>>>      https://clbin.com/HmS58
>> >>>>
>> >>>>
>> >>>> What are the next steps:
>> >>>>
>> >>>> * Getting feedback from Deployment Tools and operators on the concept
>> >>>> of this module.
>> >>>>    Maybe this module could replace what is done by Kolla with
>> >>>> merge_configs and OpenStack Ansible with config_template.
>> >>>> * On the TripleO side, we would like to see if this module could
>> >>>> replace the Puppet OpenStack modules that are now mostly used for
>> >>>> generating configuration files for containers.
>> >>>>    A transition path would be having Heat to generate Ansible vars
>> >>>> files and give it to this module. We could integrate the playbook into
>> >>>> a new task in the composable services, something like
>> >>>>    "os_gen_config_tasks", a bit like we already have for upgrade tasks,
>> >>>> also driven by Ansible.
>> >>
>> >>
>> >> This sounds good to me, though one issue i can presently see is that Puppet
>> >> modules sometimes contain quite a bit of data processing logic ("smart"
>> >> variables which map 1-to-N rather than 1-to-1 to actual config values, and
>> >> often not just in openstack service configs, e.g. puppet-nova also
>> >> configures libvirt, etc.). Also we use some non-config aspects from the
>> >> Puppet modules (e.g. seeding Keystone tenants/services/endpoints/...). We'd
>> >> need to implement this functionality elsewhere when replacing the Puppet
>> >> modules. Not a blocker, but something to keep in mind.
>> >
>> > 2 interesting things:
>> >
>> > - For the logic that are done by puppet modules for some parameters:
>> > yes I agree, this problem isn't solved now. This thread talks about
>> > config management, with some data in entry, it's a very little step I
>> > know, but it's on purpose.
>> >   Once we figure how to do that, we can think about the data
>> > generation and where to put the logic (I think the logic is too
>> > opinionated to be in a common project, but I might be wrong).
>> >
>> > - Things like libvirt, mysql, etc will be managed by something else
>> > but Puppet I think; this is out of topic for now. For Keystone
>> > resources, same thing, we could use some native python clients or
>> > Ansible modules if we switch to Ansible, etc.
>> >
>> > Again, topic is really on "give me an ini file".
>> >
>> >>>> * Another similar option to what Doug did is to write a standalone
>> >>>> tool that would generate configuration, and for Ansible users we would
>> >>>> write a new module to use this tool.
>> >>>>    Example:
>> >>>>        Step 1. oslo-config-generator --namespace cinder --format yaml >
>> >>>> cinder-schema.yaml (note this tool already exists)
>> >>>>        Step 2. Create config_data.yaml in a specific format with
>> >>>> parameters values for what we want to configure (note this format
>> >>>> doesn't exist yet but look at what Doug did in the role, we could use
>> >>>> the same kind of schema).
>> >>>>        Step 3. oslo-gen-config -i config_data.yaml -s schema.yaml >
>> >>>> cinder.conf (note this tool doesn't exist yet)
>> >>
>> >>
>> >> +1 on standalone tool which can be used in different contexts (by different
>> >> higher level tools), this sounds generally useful.
>> >
>> > Ack, good feedback.
>> >
>> >>>>
>> >>>>        For Ansible users, we would write an Ansible module that would
>> >>>> take in entry 2 files: the schema and the data. The module would just
>> >>>> run the tool provided by oslo.config.
>> >>>>        Example:
>> >>>>            - name: Generate cinder.conf
>> >>>>              oslo-gen-config: schema=cinder-schema.yaml
>> >>>>                                         data=config_data.yaml
>> >>
>> >>
>> >> +1 for module rather than a role. "Take these inputs and produce that
>> >> output" fits the module semantics better than role semantics IMO.
>> >>
>> >> FWIW as i see it right now, this ^^ + ConfigMaps + immutable-config
>> >> containers could result in a nicer/safer/more-debuggable containerized
>> >> OpenStack setup than etcd + confd in daemon mode + mutable-config
>> >> containers.
>> >
>> > Good feedback, I'm also looking forward to hear from Kolla, OSA and Helm folks.
>> >
>> >>>>
>> >>>>
>> >>>> Please bring feedback and thoughts, it's really important to know what
>> >>>> folks from Installers think about this idea; again the ultimate goal
>> >>>> is to provide a reference tool to generate configuration in OpenStack,
>> >>>> in a way that scales and is friendly for our operators.
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> [1]
>> >>>> http://lists.openstack.org/pipermail/openstack-dev/2017-June/118176.html
>> >>>> --
>> >>>> Emilien Macchi
>> >>>
>> >>>
>> >>>
>> >>>
>> >>
>> >> Have a good day,
>> >>
>> >> Jirka
>> >>
>> >> __________________________________________________________________________
>> >> OpenStack Development Mailing List (not for usage questions)
>> >> Unsubscribe: [hidden email]?subject:unsubscribe
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> >
>> >
>> > --
>> > Emilien Macchi
>> >
>> > __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe: [hidden email]?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

Bogdan Dobrelya-2
On 20.06.2017 17:27, Michał Jastrzębski wrote:

> On 19 June 2017 at 06:05, Doug Hellmann <[hidden email]> wrote:
>> Excerpts from Michał Jastrzębski's message of 2017-06-16 15:50:54 -0700:
>>> So I'm trying to figure out how to actually use it.
>>>
>>> We (and any other container based deploy..) will run into some
>>> chicken/egg problem - you need to deploy container to generate big
>>> yaml with defaults, then you need to overload it with your
>>
>> The config schema file (the "big YAML with defaults") should be part of
>> the packaged software, so the deployment tool shouldn't need to generate
>> it unless you're handling drivers that are not included in tree.
>
> Right that's what I was missing, I guess we can generate these during
> buildtime without big issues, then it will be embedded into container,
> shouldn't be too hard change and would work for both source and
> binary.
>>> configurations, validate if they're not deprecated, run container with
>>
>> It doesn't do it today, but the thing that converts the input data to
>> the INI file could automatically translate old option names to their new
>> names.
>>
>>> this ansible role (or module...really doesn't matter), spit out final
>>
>> Why does the config file need to be generated inside a container?
>
> Outside of container you don't have oslo or nova (python libs), so to
> get access to these you need to do it inside container.

That could be another container I suppose. Like those containers used
for build deps, there could be as well a container for config management
deps. Docker multi-stage [0] could help to achieve that smooth, w/o
impacting the service containers.

>
>>> confg, lay it down, deploy container again. And that will have to be
>>> done for every host class (as configs might differ host to host). Imho
>>> a bit too much for this to be appealing (but I might be wrong). I'd
>>> much rather have:
>>> 1. Yaml as input to oslo.config instead of broken ini
>>
>> I'm not opposed to switching to YAML, but it's a bit more involved than
>> just adding support in the parser. All of the work that has been done on
>> generating sample default files and documentation needs to be updated to
>> support YAML. We need a migration path to move everyone from INI to
>> YAML. And we need to update devstack and all of its plugins to edit the
>> new file format. There are probably more tasks involved in the
>> migration. I'm dealing with a couple of other projects right now, and
>> don't have time to plan all of that out myself. If someone else wants to
>> pick it up, I can help with reviews on the spec and code changes.
>
> Switching is a big no, everyone would hate us with emotion pure as
> mountain spring water. It's to support both at same time, which makes
> it slightly more complex. We could make full switch after few releases
> of deprecation I guess. Anyway, agree, lots of work.
>


--
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [deployment][kolla][openstack-ansible][openstack-helm][tripleo] ansible role to produce oslo.config files for openstack services

Flavio Percoco-2
In reply to this post by Emilien Macchi-4
On 15/06/17 13:06 -0400, Emilien Macchi wrote:

>I missed [tripleo] tag.
>
>On Thu, Jun 15, 2017 at 12:09 PM, Emilien Macchi <[hidden email]> wrote:
>> If you haven't followed the "Configuration management with etcd /
>> confd" thread [1], Doug found out that using confd to generate
>> configuration files wouldn't work for the Cinder case where we don't
>> know in advance of the deployment what settings to tell confd to look
>> at.
>> We are still looking for a generic way to generate *.conf files for
>> OpenStack, that would be usable by Deployment tools and operators.
>> Right now, Doug and I are investigating some tooling that would be
>> useful to achieve this goal.
>>
>> Doug has prototyped an Ansible role that would generate configuration
>> files by consumming 2 things:
>>
>> * Configuration schema, generated by Ben's work with Machine Readable
>> Sample Config.
>>   $ oslo-config-generator --namespace cinder --format yaml > cinder-schema.yaml
>>
>> It also needs: https://review.openstack.org/#/c/474306/ to generate
>> some extra data not included in the original version.
>>
>> * Parameters values provided in config_data directly in the playbook:
>>    config_data:
>>      DEFAULT:
>>        transport_url: rabbit://user:password@hostname
>>        verbose: true
>>
>> There are 2 options disabled by default but which would be useful for
>> production environments:
>> * Set to true to always show all configuration values: config_show_defaults
>> * Set to true to show the help text: config_show_help: true
>>
>> The Ansible module is available on github:
>> https://github.com/dhellmann/oslo-config-ansible
>>
>> To try this out, just run:
>>   $ ansible-playbook ./playbook.yml
>>
>> You can quickly see the output of cinder.conf:
>>     https://clbin.com/HmS58
>>
>>
>> What are the next steps:
>>
>> * Getting feedback from Deployment Tools and operators on the concept
>> of this module.
>>   Maybe this module could replace what is done by Kolla with
>> merge_configs and OpenStack Ansible with config_template.
>> * On the TripleO side, we would like to see if this module could
>> replace the Puppet OpenStack modules that are now mostly used for
>> generating configuration files for containers.
>>   A transition path would be having Heat to generate Ansible vars
>> files and give it to this module. We could integrate the playbook into
>> a new task in the composable services, something like
>>   "os_gen_config_tasks", a bit like we already have for upgrade tasks,
>> also driven by Ansible.
>> * Another similar option to what Doug did is to write a standalone
>> tool that would generate configuration, and for Ansible users we would
>> write a new module to use this tool.
>>   Example:
>>       Step 1. oslo-config-generator --namespace cinder --format yaml >
>> cinder-schema.yaml (note this tool already exists)
>>       Step 2. Create config_data.yaml in a specific format with
>> parameters values for what we want to configure (note this format
>> doesn't exist yet but look at what Doug did in the role, we could use
>> the same kind of schema).
>>       Step 3. oslo-gen-config -i config_data.yaml -s schema.yaml >
>> cinder.conf (note this tool doesn't exist yet)
>>
>>       For Ansible users, we would write an Ansible module that would
>> take in entry 2 files: the schema and the data. The module would just
>> run the tool provided by oslo.config.
>>       Example:
>>           - name: Generate cinder.conf
>>             oslo-gen-config: schema=cinder-schema.yaml
>>                                        data=config_data.yaml
>>
I finally caught up with this thread and got the time to get back to y'all.
Sorry.

I like the roles version more because it's flexible and easier to distribute. We
can upload it to galaxy, package it, etc. Distributing ansible modules is a bit
painful right now and you end up adding them as roles in the playbook for the
modules to be loaded.

I'm about to work on a prototype and I'll use option #1 and perhaps we can
discuss further the option #2.

Flavio

--
@flaper87
Flavio Percoco

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

signature.asc (879 bytes) Download Attachment