[Openstack] [Trove] Trove instance got stuck in BUILD state

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[Openstack] [Trove] Trove instance got stuck in BUILD state

Syed Hussain

Hi,

 

I’m installing and configuring trove(DBaaS) for exisitng openstack setup.  

 

I have openstack setup and able to boot nova instances with following components:

1.       keystone

2.       glance

3.       neutron

4.       nova

5.       cinder

 

Followed below documentation for manual installation of trove:

http://docs.openstack.org/developer/trove/dev/manual_install.html  and few correction given in this mail thread https://www.mail-archive.com/openstack%40lists.openstack.org/msg05262.html .

 

Booted up a trove instance

trove create myTrove 7 --size=2 --databases=db3 --datastore_version mysql-5.5 --datastore mysql --nic net-id=752554ef-800c-46d8-b991-361db6c58226

 

Trove instance got created but is STUCK IN BUILD state.

 

cid:image003.jpg@01CF99FC.4F639B90

 

·         nova instance associated with db instance got created successfully.

·         Cinder volumes, security groups etc are also getting created successfully.

·         I checked nova, cinder logs everything looks fine but in trove-taskmanager.log below error got logged:

PollTimeOut: Polling request timed out

I am also unable to access mysql in the booted up trove instance . via : mysql –h <instance-IP>

·         Also I’m unable to delete this instance.

o    ERROR: Instance 23c8f4d5-4905-47d2-9992-13118dfa003f is not ready. (HTTP 422) (may be this is expected)

 

I’m a novice in Openstack but new to trove.

Thanks in advance and any help is greatly appreciaited.

               

Thanks & Regards,

Syed Afzal Hussain | Software Engineer | OpenStack

DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Openstack] [Trove] Trove instance got stuck in BUILD state

Denis Makogon



On Mon, Jul 7, 2014 at 2:33 PM, Syed Hussain <[hidden email]> wrote:

Hi,

 

I’m installing and configuring trove(DBaaS) for exisitng openstack setup.  

 

I have openstack setup and able to boot nova instances with following components:

1.       keystone

2.       glance

3.       neutron

4.       nova

5.       cinder

 

Followed below documentation for manual installation of trove:

http://docs.openstack.org/developer/trove/dev/manual_install.html  and few correction given in this mail thread https://www.mail-archive.com/openstack%40lists.openstack.org/msg05262.html .


Those docs are useless, since they are not reflecting significant step - creating custom Trove images. You need to create image with Trove installed in it, create upstart scriptto lauch Trove-guestagent with appropriate configuration files that comes to the compute instance through file injection.
Vanilla images are good, but they don't have Trove in it at all.

Here are some useful steps:
1. Create custom image with trove code in it (upstart scripts, etc).
2. Register datastore and associate given image with appropriate datastore/version.

FYI, Trove is not fully integrated with devstack, so, personally i'd suggest to use https://github.com/openstack/trove-integration  simple (3 clicks) Trove + DevStack deployment.
 

 

Booted up a trove instance

trove create myTrove 7 --size=2 --databases=db3 --datastore_version mysql-5.5 --datastore mysql --nic net-id=752554ef-800c-46d8-b991-361db6c58226

 

Trove instance got created but is STUCK IN BUILD state.

 

cid:image003.jpg@01CF99FC.4F639B90

 

·         nova instance associated with db instance got created successfully.

Correct.

·         Cinder volumes, security groups etc are also getting created successfully.

Correct.

·         I checked nova, cinder logs everything looks fine but in trove-taskmanager.log below error got logged:

PollTimeOut: Polling request timed out


Correct since Trove-guest agent service wasn't able to  report about its state.

I am also unable to access mysql in the booted up trove instance . via : mysql –h <instance-IP>

·         Also I’m unable to delete this instance.

o    ERROR: Instance 23c8f4d5-4905-47d2-9992-13118dfa003f is not ready. (HTTP 422) (may be this is expected)

Correct. You cannot modify/use instances that are remaining in BUILD state.
 

I’m a novice in Openstack but new to trove.

Thanks in advance and any help is greatly appreciaited.

               

Thanks & Regards,

Syed Afzal Hussain | Software Engineer | OpenStack

DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


I'd glad to help you with other question related to Trove deployment.


Best regards,
Denis Makogon

_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Openstack] [Trove] Trove instance got stuck in BUILD state

Amrith Kumar

Denis Makogon ([hidden email]) writes:

 

| Those docs are useless, since they are not reflecting significant step –

| creating custom Trove images. You need to create image with Trove

| installed in it, create upstart scriptto lauch Trove-guestagent with appropriate

| configuration files that comes to the compute instance through file injection.

| Vanilla images are good, but they don't have Trove in it at all.

 

I think it is totally ludicrous (and to all the technical writers who work on OpenStack, downright offensive) to say the “docs are useless”. Not only have I been able to install and successfully operate a OpenStack installation by (largely) following the documentation, but “trove-integration” and “redstack” are useful for developers but I would highly doubt that a production deployment of Trove would use ‘redstack’.

 

Syed, maybe you need to download a guest image for Trove, or maybe there is something else amiss with your setup. Happy to catch up with you on IRC and help you with that. Optionally, email me and I’ll give you a hand.

 

Good job on getting all the core services installed and running, and welcome to the OpenStack community.

 

-amrith

 

--

 

Amrith Kumar, CTO, Tesora

 

Phone: +1-978-563-9590

Twitter: @amrithkumar 

Skype: amrith.skype

Web: http://www.tesora.com

IRC: amrith @freenode #openstack-trove #tesora

 

 

 

From: Denis Makogon [mailto:[hidden email]]
Sent: Monday, July 07, 2014 8:00 AM
To: OpenStack Development Mailing List (not for usage questions)
Cc: Ram Nalluri
Subject: Re: [openstack-dev] [Openstack] [Trove] Trove instance got stuck in BUILD state

 

 

 

On Mon, Jul 7, 2014 at 2:33 PM, Syed Hussain <[hidden email]> wrote:

Hi,

 

I’m installing and configuring trove(DBaaS) for exisitng openstack setup.  

 

I have openstack setup and able to boot nova instances with following components:

1.       keystone

2.       glance

3.       neutron

4.       nova

5.       cinder

 

Followed below documentation for manual installation of trove:

http://docs.openstack.org/developer/trove/dev/manual_install.html  and few correction given in this mail thread https://www.mail-archive.com/openstack%40lists.openstack.org/msg05262.html .

 

Those docs are useless, since they are not reflecting significant step - creating custom Trove images. You need to create image with Trove installed in it, create upstart scriptto lauch Trove-guestagent with appropriate configuration files that comes to the compute instance through file injection.

Vanilla images are good, but they don't have Trove in it at all.

Here are some useful steps:

1. Create custom image with trove code in it (upstart scripts, etc).

2. Register datastore and associate given image with appropriate datastore/version.

FYI, Trove is not fully integrated with devstack, so, personally i'd suggest to use https://github.com/openstack/trove-integration  simple (3 clicks) Trove + DevStack deployment.

 

 

Booted up a trove instance

trove create myTrove 7 --size=2 --databases=db3 --datastore_version mysql-5.5 --datastore mysql --nic net-id=752554ef-800c-46d8-b991-361db6c58226

 

Trove instance got created but is STUCK IN BUILD state.

 

cid:image003.jpg@01CF99FC.4F639B90

 

·         nova instance associated with db instance got created successfully.

Correct.

·         Cinder volumes, security groups etc are also getting created successfully.

Correct.

·         I checked nova, cinder logs everything looks fine but in trove-taskmanager.log below error got logged:

PollTimeOut: Polling request timed out

 

Correct since Trove-guest agent service wasn't able to  report about its state.

I am also unable to access mysql in the booted up trove instance . via : mysql –h <instance-IP>

·         Also I’m unable to delete this instance.

o    ERROR: Instance 23c8f4d5-4905-47d2-9992-13118dfa003f is not ready. (HTTP 422) (may be this is expected)

Correct. You cannot modify/use instances that are remaining in BUILD state.
 

I’m a novice in Openstack but new to trove.

Thanks in advance and any help is greatly appreciaited.

               

Thanks & Regards,

Syed Afzal Hussain | Software Engineer | OpenStack

DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

 

I'd glad to help you with other question related to Trove deployment.

Best regards,
Denis Makogon


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

smime.p7s (8K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Openstack] [Trove] Trove instance got stuck in BUILD state

Denis Makogon



On Mon, Jul 7, 2014 at 3:40 PM, Amrith Kumar <[hidden email]> wrote:

Denis Makogon ([hidden email]) writes:

 

| Those docs are useless, since they are not reflecting significant step –

| creating custom Trove images. You need to create image with Trove

| installed in it, create upstart scriptto lauch Trove-guestagent with appropriate

| configuration files that comes to the compute instance through file injection.

| Vanilla images are good, but they don't have Trove in it at all.

 

I think it is totally ludicrous (and to all the technical writers who work on OpenStack, downright offensive) to say the “docs are useless”. Not only have I been able to install and successfully operate a OpenStack installation by (largely) following the documentation, but “trove-integration” and “redstack” are useful for developers but I would highly doubt that a production deployment of Trove would use ‘redstack’.


Amrith, those doc doesn't reflect any post-deployment steps, even more, doc still suggesting to use trove-cli that was deprecated long time ago. I do agree that trove-integration project can't be used as production deployment system, but first try-outs - more than enough.

 

Syed, maybe you need to download a guest image for Trove, or maybe there is something else amiss with your setup. Happy to catch up with you on IRC and help you with that. Optionally, email me and I’ll give you a hand.

 

Syed, i'd suggest to use heat-jeos tools to build custom images for Trove. Since it doesn't forces you to relay on any pre-baked images built for other production deployments.
 Or there's another way to accomplish Trove instances provisioning - you are able to use cloud-init mechanism (for more information see link - option for Trove-taskamanger service, each cloud-init script should be placed under {{cloud-init-script-location}}/{{datastore}} (/etc/trove/cloud-init/mysql, etc.)
 

Good job on getting all the core services installed and running, and welcome to the OpenStack community.

 

-amrith

 

--

 

Amrith Kumar, CTO, Tesora

 

Phone: <a href="tel:%2B1-978-563-9590" value="+19785639590" target="_blank">+1-978-563-9590

Twitter: @amrithkumar 

Skype: amrith.skype

Web: http://www.tesora.com

IRC: amrith @freenode #openstack-trove #tesora

 

 

 

From: Denis Makogon [mailto:[hidden email]]
Sent: Monday, July 07, 2014 8:00 AM
To: OpenStack Development Mailing List (not for usage questions)
Cc: Ram Nalluri
Subject: Re: [openstack-dev] [Openstack] [Trove] Trove instance got stuck in BUILD state

 

 

 

On Mon, Jul 7, 2014 at 2:33 PM, Syed Hussain <[hidden email]> wrote:

Hi,

 

I’m installing and configuring trove(DBaaS) for exisitng openstack setup.  

 

I have openstack setup and able to boot nova instances with following components:

1.       keystone

2.       glance

3.       neutron

4.       nova

5.       cinder

 

Followed below documentation for manual installation of trove:

http://docs.openstack.org/developer/trove/dev/manual_install.html  and few correction given in this mail thread https://www.mail-archive.com/openstack%40lists.openstack.org/msg05262.html .

 

Those docs are useless, since they are not reflecting significant step - creating custom Trove images. You need to create image with Trove installed in it, create upstart scriptto lauch Trove-guestagent with appropriate configuration files that comes to the compute instance through file injection.

Vanilla images are good, but they don't have Trove in it at all.

Here are some useful steps:

1. Create custom image with trove code in it (upstart scripts, etc).

2. Register datastore and associate given image with appropriate datastore/version.

FYI, Trove is not fully integrated with devstack, so, personally i'd suggest to use https://github.com/openstack/trove-integration  simple (3 clicks) Trove + DevStack deployment.

 

 

Booted up a trove instance

trove create myTrove 7 --size=2 --databases=db3 --datastore_version mysql-5.5 --datastore mysql --nic net-id=752554ef-800c-46d8-b991-361db6c58226

 

Trove instance got created but is STUCK IN BUILD state.

 

cid:image003.jpg@01CF99FC.4F639B90

 

·         nova instance associated with db instance got created successfully.

Correct.

·         Cinder volumes, security groups etc are also getting created successfully.

Correct.

·         I checked nova, cinder logs everything looks fine but in trove-taskmanager.log below error got logged:

PollTimeOut: Polling request timed out

 

Correct since Trove-guest agent service wasn't able to  report about its state.

I am also unable to access mysql in the booted up trove instance . via : mysql –h <instance-IP>

·         Also I’m unable to delete this instance.

o    ERROR: Instance 23c8f4d5-4905-47d2-9992-13118dfa003f is not ready. (HTTP 422) (may be this is expected)

Correct. You cannot modify/use instances that are remaining in BUILD state.
 

I’m a novice in Openstack but new to trove.

Thanks in advance and any help is greatly appreciaited.

               

Thanks & Regards,

Syed Afzal Hussain | Software Engineer | OpenStack

DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

 

I'd glad to help you with other question related to Trove deployment.

Best regards,
Denis Makogon


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Openstack] [Trove] Trove instance got stuck in BUILD state

Nikhil Manchanda-2

Denis Makogon writes:

> On Mon, Jul 7, 2014 at 3:40 PM, Amrith Kumar <[hidden email]> wrote:
>
>> Denis Makogon ([hidden email]) writes:
>>
> [...]
>>
>> I think it is totally ludicrous (and to all the technical writers who work
>> on OpenStack, downright offensive) to say the “docs are useless”. Not only
>> have I been able to install and successfully operate a OpenStack
>> installation by (largely) following the documentation, but
>> “trove-integration” and “redstack” are useful for developers but I would
>> highly doubt that a production deployment of Trove would use ‘redstack’.
>>
>
> Amrith, those doc doesn't reflect any post-deployment steps, even more, doc
> still suggesting to use trove-cli that was deprecated long time ago. I do
> agree that trove-integration project can't be used as production deployment
> system, but first try-outs - more than enough.
>

I think we're doing ourselves a great disservice here by calling our own
docs "useless", and yet not taking any steps to correct the issue at
hand. If the docs are missing certain key steps we need to ensure that
they are updated and improved as we go along. I, for one, am pretty
certain that the image building and post-deploy steps for trove do work,
since that's what we currently use in the image build job and the
integration tests, and they've been passing pretty consistently.

I'm going to add a "Scrub the Docs" agenda item to this week's Trove
meeting. I'd really like those of us who have a few cycles to clean-up
the docs we have, and make sure that information in them is
up-to-date. I can help lead this effort.


> FYI, Trove is not fully integrated with devstack, so, personally i'd
> suggest to use https://github.com/openstack/trove-integration[3]  simple (3
> clicks) Trove + DevStack deployment.

This is no longer true. We've recently done a lot of work here to get
Trove fully integrated with devstack. A trove guest-image is now
uploaded into glance as part of the trove install in devstack.
Appropriate default datastores, and versions are also created, making
the trove install fully usable.

If you're running trove functional tests, you probably want to stick
with trove-integration for now, since it sets up some test conf values
that are needed for the functional tests. But if all you need to do is
to take trove for an end-to-end test-run, a devstack install with the
trove services enabled should be sufficient.


Syed, if you're still having trouble with any of this, pleas feel free
to reach out to me or anyone else on #openstack-trove. We'd be happy
to assist you with any issues you might have encountered.

Hope this helps,

Thanks,
Nikhil

_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Openstack] [Trove] Trove instance got stuck in BUILD state

Mark Kirkwood
In reply to this post by Amrith Kumar
On 08/07/14 00:40, Amrith Kumar wrote:

>
>
> I think it is totally ludicrous (and to all the technical writers who work on OpenStack, downright offensive) to say the “docs are useless”. Not only have I been able to install and successfully operate a OpenStack installation by (largely) following the documentation, but “trove-integration” and “redstack” are useful for developers but I would highly doubt that a production deployment of Trove would use ‘redstack’.
>
>
>
> Syed, maybe you need to download a guest image for Trove, or maybe there is something else amiss with your setup. Happy to catch up with you on IRC and help you with that. Optionally, email me and I’ll give you a hand.
>
>
>

It is a bit harsh, to be sure. However critical areas are light/thin or
not covered at all - and this is bound to generate a bit of frustration
for folk wanting to use this feature.

In particular:

- guest image preparation
- guest file injection (/etc/guest_info) nova interaction
- dns requirements for guest image (self hostname resolv)
- swift backup config authorization
- api_extensions_path setting and how critical that is

There are probably more that I have forgotten (repressed perhaps...)!

Regards

Mark


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Openstack] [Trove] Trove instance got stuck in BUILD state

Denis Makogon
Mark, there are also no documentation about service tuning(no description of service related options, sample configs in Trove repo is not enough). So, I think we should extend your list of significant things to document. 

Thanks,
Denis M.

вторник, 8 июля 2014 г. пользователь Mark Kirkwood написал:
On 08/07/14 00:40, Amrith Kumar wrote:



I think it is totally ludicrous (and to all the technical writers who work on OpenStack, downright offensive) to say the “docs are useless”. Not only have I been able to install and successfully operate a OpenStack installation by (largely) following the documentation, but “trove-integration” and “redstack” are useful for developers but I would highly doubt that a production deployment of Trove would use ‘redstack’.



Syed, maybe you need to download a guest image for Trove, or maybe there is something else amiss with your setup. Happy to catch up with you on IRC and help you with that. Optionally, email me and I’ll give you a hand.




It is a bit harsh, to be sure. However critical areas are light/thin or not covered at all - and this is bound to generate a bit of frustration for folk wanting to use this feature.

In particular:

- guest image preparation
- guest file injection (/etc/guest_info) nova interaction
- dns requirements for guest image (self hostname resolv)
- swift backup config authorization
- api_extensions_path setting and how critical that is

There are probably more that I have forgotten (repressed perhaps...)!

Regards

Mark


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Openstack] [Trove] Trove instance got stuck in BUILD state

Mark Kirkwood
On 08/07/14 17:08, Denis Makogon wrote:
> Mark, there are also no documentation about service tuning(no description
> of service related options, sample configs in Trove repo is not enough).
> So, I think we should extend your list of significant things to document.

Right - I guess most of the tuning/config parameters could be better
documented too (I do recall seeing this mentioned for one of the Trove
meetings).

One other thing I recall is:

- mysql security install/setup in guest (mysql root password).

I had to struggle through all of these - and it took a lot of time,
because essentially the only viable way to debug each issue was:

- check in an equivalent devstack build
- read devstack setup code

or (if the issue was present in devstack as well)

- read the trove code and insert debug logging as appropriate

...and while this was a very interesting exercise, it was not a fast one!

Cheers

Mark

_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [Openstack] [Trove] Trove instance got stuck in BUILD state

Mark Kirkwood
In reply to this post by Denis Makogon
On 08/07/14 17:08, Denis Makogon wrote:
> Mark, there are also no documentation about service tuning(no description
> of service related options, sample configs in Trove repo is not enough).
> So, I think we should extend your list of significant things to document.

...and in case it might be helpful: here's my notes for installing
openstack/trove on Ubuntu 14.04 using the Ubuntu packages, and debugging
it (ahem). Some of the issues (virtio on bridges) are caused by it being
all on one node, but hopefully it is generally useful (I cover how to
build the guest image and get backups etc going). I make no claim for it
being the best/only way to force the beast into being :-) but I think it
works (for the logic devotees - sufficient not maybe not necessary)!

Regards

Mark


_______________________________________________
OpenStack-dev mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

README-OPENSTACK.gz (16K) Download Attachment