[devstack] etcd v3.2.0?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[devstack] etcd v3.2.0?

Tony Breeds
Hi All,
        I just push a review [1] to bump the minimum etcd version to
3.2.0 which works on intel and ppc64le.  I know we're pretty late in the
cycle to be making changes like this but releasing pike with a dependacy
on 3.1.x make it harder for users on ppc64le (not many but a few :D)

Yours Tony.

[1] https://review.openstack.org/474825

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [devstack] etcd v3.2.0?

Sean Dague-2
On 06/15/2017 10:06 PM, Tony Breeds wrote:
> Hi All,
> I just push a review [1] to bump the minimum etcd version to
> 3.2.0 which works on intel and ppc64le.  I know we're pretty late in the
> cycle to be making changes like this but releasing pike with a dependacy
> on 3.1.x make it harder for users on ppc64le (not many but a few :D)
>
> Yours Tony.
>
> [1] https://review.openstack.org/474825

It should be fine, no one is really using these much at this point.
However it looks like mirroring is not happening automatically? The
patch fails on not existing in the infra mirror.

        -Sean

--
Sean Dague
http://dague.net

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [devstack] etcd v3.2.0?

Mikhail Medvedev
On Fri, Jun 16, 2017 at 6:01 AM, Sean Dague <[hidden email]> wrote:

> On 06/15/2017 10:06 PM, Tony Breeds wrote:
>> Hi All,
>>       I just push a review [1] to bump the minimum etcd version to
>> 3.2.0 which works on intel and ppc64le.  I know we're pretty late in the
>> cycle to be making changes like this but releasing pike with a dependacy
>> on 3.1.x make it harder for users on ppc64le (not many but a few :D)
>>
>> Yours Tony.
>>
>> [1] https://review.openstack.org/474825
>
> It should be fine, no one is really using these much at this point.
> However it looks like mirroring is not happening automatically? The
> patch fails on not existing in the infra mirror.
>
>         -Sean
>

It appears so. Also, IIRC, infra mirror would only host x86 binaries.
Right now PowerKVM CI works by patching devstack-gate to override
infra etcd download url. The fix [2] still needs to get merged to make
it a bit easier to use d-g with your own etcd mirror.

[2] https://review.openstack.org/#/c/467437/

---
Mikhail Medvedev
IBM OpenStack CI for KVM on Power

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [devstack] etcd v3.2.0?

Davanum Srinivas
Mikhail,

I have a TODO on my list - " adding a job that looks for new releases
and uploads them to tarballs periodically "

Thanks,
-- Dims

On Fri, Jun 16, 2017 at 3:32 PM, Mikhail Medvedev <[hidden email]> wrote:

> On Fri, Jun 16, 2017 at 6:01 AM, Sean Dague <[hidden email]> wrote:
>> On 06/15/2017 10:06 PM, Tony Breeds wrote:
>>> Hi All,
>>>       I just push a review [1] to bump the minimum etcd version to
>>> 3.2.0 which works on intel and ppc64le.  I know we're pretty late in the
>>> cycle to be making changes like this but releasing pike with a dependacy
>>> on 3.1.x make it harder for users on ppc64le (not many but a few :D)
>>>
>>> Yours Tony.
>>>
>>> [1] https://review.openstack.org/474825
>>
>> It should be fine, no one is really using these much at this point.
>> However it looks like mirroring is not happening automatically? The
>> patch fails on not existing in the infra mirror.
>>
>>         -Sean
>>
>
> It appears so. Also, IIRC, infra mirror would only host x86 binaries.
> Right now PowerKVM CI works by patching devstack-gate to override
> infra etcd download url. The fix [2] still needs to get merged to make
> it a bit easier to use d-g with your own etcd mirror.
>
> [2] https://review.openstack.org/#/c/467437/
>
> ---
> Mikhail Medvedev
> IBM OpenStack CI for KVM on Power
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Davanum Srinivas :: https://twitter.com/dims

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [devstack] etcd v3.2.0?

Tony Breeds
On Fri, Jun 16, 2017 at 03:59:22PM -0400, Davanum Srinivas wrote:
> Mikhail,
>
> I have a TODO on my list - " adding a job that looks for new releases
> and uploads them to tarballs periodically "

If you point me to how things are added to that mirror I can work
towards that.

Tony.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [devstack] etcd v3.2.0?

Davanum Srinivas
On Sun, Jun 18, 2017 at 7:36 PM, Tony Breeds <[hidden email]> wrote:
> On Fri, Jun 16, 2017 at 03:59:22PM -0400, Davanum Srinivas wrote:
>> Mikhail,
>>
>> I have a TODO on my list - " adding a job that looks for new releases
>> and uploads them to tarballs periodically "
>
> If you point me to how things are added to that mirror I can work
> towards that.

Awesome! thanks Tony, some kolla jobs do that for example, but i think
this job is a better one to key off of:
http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/infra.yaml#n381

Outline of the work is - check if there are any new releases in github
downloads, if so download them using wget and then delegate to the scp
publisher (with keep-hierarchy) to create the new directories and
upload the file(s).

Thanks,
Dims

>
> Tony.
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



--
Davanum Srinivas :: https://twitter.com/dims

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [devstack] etcd v3.2.0?

Tony Breeds
On Sun, Jun 18, 2017 at 08:19:16PM -0400, Davanum Srinivas wrote:

> Awesome! thanks Tony, some kolla jobs do that for example, but i think
> this job is a better one to key off of:
> http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/infra.yaml#n381
>
> Outline of the work is - check if there are any new releases in github
> downloads, if so download them using wget and then delegate to the scp
> publisher (with keep-hierarchy) to create the new directories and
> upload the file(s).

So perhaps I'm dense but I can't see an easy way to get a list of
release artefacts from github in a form that wget can consume.  The best
I can see is via the API.  I've knocked up a quick'n'dirty mirror
script[1] but I really feel like I've gone off into the weeds.

You basically need to do:

git clone  && cd
virtualenv .venv
.venv/bin/pip install -U pip setuptools wheel
.venv/bin/pip install -r ./requirements.txt   # [2]
.venv/bin/python ./mirror-github-releases.py \
        'coreos/etcd::.*linux.*gz:etcd' \
        'coreos/etcd:6225411:.*linux.*gz:etcd'

This will in theory from the 3.2.0 (latest) release and look at the
3.1.7 release, see that it's already publically mirrored and move on.

It wouldn't be too hard to incorporate into a job.  Thoughts?

Yours Tony.

[1]  https://github.com/tbreeds/mirror-github-releases
[2] Yes of course I could publish it on pypi if we want to go down this
    path

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [devstack] etcd v3.2.0?

Davanum Srinivas
Tony,


On Sun, Jun 18, 2017 at 11:34 PM, Tony Breeds <[hidden email]> wrote:

> On Sun, Jun 18, 2017 at 08:19:16PM -0400, Davanum Srinivas wrote:
>
>> Awesome! thanks Tony, some kolla jobs do that for example, but i think
>> this job is a better one to key off of:
>> http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/infra.yaml#n381
>>
>> Outline of the work is - check if there are any new releases in github
>> downloads, if so download them using wget and then delegate to the scp
>> publisher (with keep-hierarchy) to create the new directories and
>> upload the file(s).
>
> So perhaps I'm dense but I can't see an easy way to get a list of
> release artefacts from github in a form that wget can consume.  The best
> I can see is via the API.  I've knocked up a quick'n'dirty mirror
> script[1] but I really feel like I've gone off into the weeds.
>
> You basically need to do:
>
> git clone  && cd
> virtualenv .venv
> .venv/bin/pip install -U pip setuptools wheel
> .venv/bin/pip install -r ./requirements.txt   # [2]
> .venv/bin/python ./mirror-github-releases.py \
>         'coreos/etcd::.*linux.*gz:etcd' \
>         'coreos/etcd:6225411:.*linux.*gz:etcd'

Works for me!

> This will in theory from the 3.2.0 (latest) release and look at the
> 3.1.7 release, see that it's already publically mirrored and move on.
>
> It wouldn't be too hard to incorporate into a job.  Thoughts?
>
> Yours Tony.
>
> [1]  https://github.com/tbreeds/mirror-github-releases
> [2] Yes of course I could publish it on pypi if we want to go down this
>     path
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



--
Davanum Srinivas :: https://twitter.com/dims

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [devstack] etcd v3.2.0?

Tony Breeds
On Mon, Jun 19, 2017 at 08:17:53AM -0400, Davanum Srinivas wrote:

> Tony,
>
>
> On Sun, Jun 18, 2017 at 11:34 PM, Tony Breeds <[hidden email]> wrote:
> > On Sun, Jun 18, 2017 at 08:19:16PM -0400, Davanum Srinivas wrote:
> >
> >> Awesome! thanks Tony, some kolla jobs do that for example, but i think
> >> this job is a better one to key off of:
> >> http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/infra.yaml#n381
> >>
> >> Outline of the work is - check if there are any new releases in github
> >> downloads, if so download them using wget and then delegate to the scp
> >> publisher (with keep-hierarchy) to create the new directories and
> >> upload the file(s).
> >
> > So perhaps I'm dense but I can't see an easy way to get a list of
> > release artefacts from github in a form that wget can consume.  The best
> > I can see is via the API.  I've knocked up a quick'n'dirty mirror
> > script[1] but I really feel like I've gone off into the weeds.
> >
> > You basically need to do:
> >
> > git clone  && cd
> > virtualenv .venv
> > .venv/bin/pip install -U pip setuptools wheel
> > .venv/bin/pip install -r ./requirements.txt   # [2]
> > .venv/bin/python ./mirror-github-releases.py \
> >         'coreos/etcd::.*linux.*gz:etcd' \
> >         'coreos/etcd:6225411:.*linux.*gz:etcd'
>
> Works for me!
Okay I'll put something more compleet together for infra review.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [devstack] etcd v3.2.0?

Tony Breeds
In reply to this post by Tony Breeds
On Fri, Jun 16, 2017 at 12:06:47PM +1000, Tony Breeds wrote:
> Hi All,
> I just push a review [1] to bump the minimum etcd version to
> 3.2.0 which works on intel and ppc64le.  I know we're pretty late in the
> cycle to be making changes like this but releasing pike with a dependacy
> on 3.1.x make it harder for users on ppc64le (not many but a few :D)
>
> Yours Tony.
>
> [1] https://review.openstack.org/474825

So this came up at the PTG and the current plan is:

Interim solution:
 1. Get the mirroring tool to the point it can be consumed by infra.
 2. setup a new zuulv3 job to run this and do the mirroring.

Middle term solution:
 1. Get etcd3 packages updated in debian/ubuntu create a PPA (or
    similar) for infra to consume.

Both of these are intended to be done during Queens.

Long term plan:
 1. Ensue the packages above are just there for 18.04 so we can put this
    behind us.

With hindsight I think we need to add something like "Current packages
can be consumed in our CI" as a requirement for anything added as a base
service.



> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Yours Tony.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

signature.asc (499 bytes) Download Attachment