[masakari] problems starting up masakari instance monitoring in devstack @ master

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

[masakari] problems starting up masakari instance monitoring in devstack @ master

Waines, Greg

I am just getting started working with masakari.

I am working on master.

 

I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes.

 

I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors .

Specifically:

# install & startup monitors

cd

git clone https://github.com/openstack/masakari-monitors.git

cd masakari-monitors

sudo python setup.py install

cd

sudo mkdir /etc/masakarimonitors

sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list

cd ~/masakari-monitors/masakarimonitors/cmd

sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list &

sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf &

sudo /usr/bin/python ./instancemonitor.py &

 

However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file.

 

Is this the correct way to startup masakari instance monitoring ?

 

Greg.

 

 

 

 

My Masakari setup in Devstack

 

sudo useradd -s /bin/bash -d /opt/stack -m stack

echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack

sudo su - stack

 

git clone https://github.com/openstack-dev/devstack       

cd devstack

 

                                                local.conf file:

[[local|localrc]]

ADMIN_PASSWORD=admin

DATABASE_PASSWORD=admin

RABBIT_PASSWORD=admin

SERVICE_PASSWORD=admin

# setup Neutron services

disable_service n-net

enable_service q-svc

enable_service q-agt

enable_service q-dhcp

enable_service q-l3

enable_service q-meta

# ceilometer

enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer

enable_plugin aodh https://git.openstack.org/openstack/aodh

# heat

enable_plugin heat https://git.openstack.org/openstack/heat

# vitrage

enable_plugin vitrage https://git.openstack.org/openstack/vitrage

enable_plugin vitrage-dashboard https://git.openstack.org/openstack/vitrage-dashboard

# masakari

enable_plugin masakari git://git.openstack.org/openstack/masakari

 

 

./stack.sh

 

 

 


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [masakari]problems starting up masakari instance monitoring in devstack @ master

Rikimaru Honjo-2
Hello Greg,

Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf.
You can generate it by "tox -egenconfig".

hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script.
masakarimonitors.conf is a configuration file for monitors implemented by python that you installed.

And, we are preparing setting guides.
Please see it if you are good.

masakari:
https://review.openstack.org/#/c/489570/

masakari-monitors:
https://review.openstack.org/#/c/489095/

Best regards,

On 2017/12/06 22:48, Waines, Greg wrote:

> I am just getting started working with masakari.
> I am working on master.
>
> I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes.
>
> I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors .
> Specifically:
> # install & startup monitors
> cd
> git clone https://github.com/openstack/masakari-monitors.git
> cd masakari-monitors
> sudo python setup.py install
> cd
> sudo mkdir /etc/masakarimonitors
> sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf
> sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf
> sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list
> cd ~/masakari-monitors/masakarimonitors/cmd
> sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list &
> sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf &
> sudo /usr/bin/python ./instancemonitor.py &
>
> However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file.
>
> Is this the correct way to startup masakari instance monitoring ?
>
> Greg.
>
>
>
>
> My Masakari setup in Devstack
>
> sudo useradd -s /bin/bash -d /opt/stack -m stack
> echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack
> sudo su - stack
>
> git clone https://github.com/openstack-dev/devstack
> cd devstack
>
>                                                  local.conf file:
> [[local|localrc]]
> ADMIN_PASSWORD=admin
> DATABASE_PASSWORD=admin
> RABBIT_PASSWORD=admin
> SERVICE_PASSWORD=admin
> # setup Neutron services
> disable_service n-net
> enable_service q-svc
> enable_service q-agt
> enable_service q-dhcp
> enable_service q-l3
> enable_service q-meta
> # ceilometer
> enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer
> enable_plugin aodh https://git.openstack.org/openstack/aodh
> # heat
> enable_plugin heat https://git.openstack.org/openstack/heat
> # vitrage
> enable_plugin vitrage https://git.openstack.org/openstack/vitrage
> enable_plugin vitrage-dashboard https://git.openstack.org/openstack/vitrage-dashboard
> # masakari
> enable_plugin masakari git://git.openstack.org/openstack/masakari
>
>
> ./stack.sh
>
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: [hidden email]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

--
_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
Rikimaru Honjo
E-mail:[hidden email]



__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [masakari] problems starting up masakari instance monitoring in devstack @ master

Rikimaru Honjo-2
Hello Greg,

I forgot to tell you.

Please use process_list.yaml instead of proc.list.sample.

On 2017/12/07 14:03, Rikimaru Honjo wrote:

> Hello Greg,
>
> Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf.
> You can generate it by "tox -egenconfig".
>
> hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script.
> masakarimonitors.conf is a configuration file for monitors implemented by python that you installed.
>
> And, we are preparing setting guides.
> Please see it if you are good.
>
> masakari:
> https://review.openstack.org/#/c/489570/
>
> masakari-monitors:
> https://review.openstack.org/#/c/489095/
>
> Best regards,
>
> On 2017/12/06 22:48, Waines, Greg wrote:
>> I am just getting started working with masakari.
>> I am working on master.
>>
>> I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes.
>>
>> I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors .
>> Specifically:
>> # install & startup monitors
>> cd
>> git clone https://github.com/openstack/masakari-monitors.git
>> cd masakari-monitors
>> sudo python setup.py install
>> cd
>> sudo mkdir /etc/masakarimonitors
>> sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf
>> sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf
>> sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list
>> cd ~/masakari-monitors/masakarimonitors/cmd
>> sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list &
>> sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf &
>> sudo /usr/bin/python ./instancemonitor.py &
>>
>> However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file.
>>
>> Is this the correct way to startup masakari instance monitoring ?
>>
>> Greg.
>>
>>
>>
>>
>> My Masakari setup in Devstack
>>
>> sudo useradd -s /bin/bash -d /opt/stack -m stack
>> echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack
>> sudo su - stack
>>
>> git clone https://github.com/openstack-dev/devstack
>> cd devstack
>>
>>                                                  local.conf file:
>> [[local|localrc]]
>> ADMIN_PASSWORD=admin
>> DATABASE_PASSWORD=admin
>> RABBIT_PASSWORD=admin
>> SERVICE_PASSWORD=admin
>> # setup Neutron services
>> disable_service n-net
>> enable_service q-svc
>> enable_service q-agt
>> enable_service q-dhcp
>> enable_service q-l3
>> enable_service q-meta
>> # ceilometer
>> enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer
>> enable_plugin aodh https://git.openstack.org/openstack/aodh
>> # heat
>> enable_plugin heat https://git.openstack.org/openstack/heat
>> # vitrage
>> enable_plugin vitrage https://git.openstack.org/openstack/vitrage
>> enable_plugin vitrage-dashboard https://git.openstack.org/openstack/vitrage-dashboard
>> # masakari
>> enable_plugin masakari git://git.openstack.org/openstack/masakari
>>
>>
>> ./stack.sh
>>
>>
>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: [hidden email]?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>

--
_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
Rikimaru Honjo
E-mail:[hidden email]


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [masakari] problems starting up masakari instance monitoring in devstack @ master

Waines, Greg

thanks Honjo,

 

I’ll try this out now.

 

I’m assuming that it is ok to start ONLY the instance-monitor if that’s all i want to test.

Is that correct ?

 

In that case, following the instructions you pointed me at,

I would do the following:

·         setup devstack

o    i.e. with ‘enable_plugin masakari git://git.openstack.org/openstack/masakari’

·          

·         ( don’t need corosync and pacemaker )

·          

·         install & startup client

o    cd

o    git clone https://github.com/openstack/python-masakariclient.git

o    cd python-masakariclient

o    sudo python setup.py build

o    sudo python setup.py install

·          

·         install & startup instance monitor

o    cd

o    git clone https://github.com/openstack/masakari-monitors.git

o    sudo mkdir /etc/masakarimonitors

o    cd masakari-monitors

o    sudo python setup.py build

o    sudo python setup.py install

o     

o    sudo masakari-instancemonitor &

 

Is this correct ?

 

Greg.

 

 

 

 

From: Rikimaru Honjo <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, December 7, 2017 at 12:24 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Hello Greg,

 

I forgot to tell you.

 

Please use process_list.yaml instead of proc.list.sample.

 

On 2017/12/07 14:03, Rikimaru Honjo wrote:

Hello Greg,

Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf.

You can generate it by "tox -egenconfig".

hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script.

masakarimonitors.conf is a configuration file for monitors implemented by python that you installed.

And, we are preparing setting guides.

Please see it if you are good.

masakari:

masakari-monitors:

Best regards,

On 2017/12/06 22:48, Waines, Greg wrote:

I am just getting started working with masakari.

I am working on master.

 

I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes.

 

I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors .

Specifically:

# install & startup monitors

cd

cd masakari-monitors

sudo python setup.py install

cd

sudo mkdir /etc/masakarimonitors

sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list

cd ~/masakari-monitors/masakarimonitors/cmd

sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list &

sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf &

sudo /usr/bin/python ./instancemonitor.py &

 

However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file.

 

Is this the correct way to startup masakari instance monitoring ?

 

Greg.

 

 

 

 

My Masakari setup in Devstack

 

sudo useradd -s /bin/bash -d /opt/stack -m stack

echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack

sudo su - stack

 

cd devstack

 

                                                 local.conf file:

[[local|localrc]]

ADMIN_PASSWORD=admin

DATABASE_PASSWORD=admin

RABBIT_PASSWORD=admin

SERVICE_PASSWORD=admin

# setup Neutron services

disable_service n-net

enable_service q-svc

enable_service q-agt

enable_service q-dhcp

enable_service q-l3

enable_service q-meta

# ceilometer

# heat

# vitrage

# masakari

enable_plugin masakari git://git.openstack.org/openstack/masakari

 

 

./stack.sh

 

 

 

 

 

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]?subject:unsubscribe

 

 

--

_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

Rikimaru Honjo

 

 

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]?subject:unsubscribe

 


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [masakari] problems starting up masakari instance monitoring in devstack @ master

Waines, Greg

Honjo,

 

Assuming the masakari instance monitor is up and running,

What is the actual USE CASE / TEST CASE for demonstrating instance monitoring ?

 

e.g. is it ?

·         launch a VM

·         ps -ef | fgrep qemu        // to find the qemu process containing the running VM

·         kill -9 <qemu-pid>

·          

·         Logs ?

o    will there be any logs from instance monitor that it detected a failure ?

·          

·         instance-monitor will report VM failure to masakari-engine

·         masakari-engine

o    Logs ?

§  will  there be any logs from masakari-engine that it received notification of failure ?

o    then will masakari-engine make the appropriate NOVA calls to recover the VM ?

 

 

let me know if this is correct,

Greg.

 

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:17 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

thanks Honjo,

 

I’ll try this out now.

 

I’m assuming that it is ok to start ONLY the instance-monitor if that’s all i want to test.

Is that correct ?

 

In that case, following the instructions you pointed me at,

I would do the following:

·         setup devstack

o   i.e. with ‘enable_plugin masakari git://git.openstack.org/openstack/masakari’

·          

·         ( don’t need corosync and pacemaker )

·          

·         install & startup client

o   cd

o   git clone https://github.com/openstack/python-masakariclient.git

o   cd python-masakariclient

o   sudo python setup.py build

o   sudo python setup.py install

·          

·         install & startup instance monitor

o   cd

o   git clone https://github.com/openstack/masakari-monitors.git

o   sudo mkdir /etc/masakarimonitors

o   cd masakari-monitors

o   sudo python setup.py build

o   sudo python setup.py install

o    

o   sudo masakari-instancemonitor &

 

Is this correct ?

 

Greg.

 

 

 

 

From: Rikimaru Honjo <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, December 7, 2017 at 12:24 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Hello Greg,

 

I forgot to tell you.

 

Please use process_list.yaml instead of proc.list.sample.

 

On 2017/12/07 14:03, Rikimaru Honjo wrote:

Hello Greg,

Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf.

You can generate it by "tox -egenconfig".

hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script.

masakarimonitors.conf is a configuration file for monitors implemented by python that you installed.

And, we are preparing setting guides.

Please see it if you are good.

masakari:

masakari-monitors:

Best regards,

On 2017/12/06 22:48, Waines, Greg wrote:

I am just getting started working with masakari.

I am working on master.

 

I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes.

 

I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors .

Specifically:

# install & startup monitors

cd

cd masakari-monitors

sudo python setup.py install

cd

sudo mkdir /etc/masakarimonitors

sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list

cd ~/masakari-monitors/masakarimonitors/cmd

sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list &

sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf &

sudo /usr/bin/python ./instancemonitor.py &

 

However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file.

 

Is this the correct way to startup masakari instance monitoring ?

 

Greg.

 

 

 

 

My Masakari setup in Devstack

 

sudo useradd -s /bin/bash -d /opt/stack -m stack

echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack

sudo su - stack

 

cd devstack

 

                                                 local.conf file:

[[local|localrc]]

ADMIN_PASSWORD=admin

DATABASE_PASSWORD=admin

RABBIT_PASSWORD=admin

SERVICE_PASSWORD=admin

# setup Neutron services

disable_service n-net

enable_service q-svc

enable_service q-agt

enable_service q-dhcp

enable_service q-l3

enable_service q-meta

# ceilometer

# heat

# vitrage

# masakari

enable_plugin masakari git://git.openstack.org/openstack/masakari

 

 

./stack.sh

 

 

 

 

 

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]?subject:unsubscribe

 

 

--

_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

Rikimaru Honjo

 

 

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]?subject:unsubscribe

 


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [masakari] problems starting up masakari instance monitoring in devstack @ master

Waines, Greg
In reply to this post by Waines, Greg

Actually, i’m guessing this is wrong because masakari-instancemonitor fails to start when i use these instructions.

 

stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &

[1] 22959

stack@devstack-masakari:~/masakari-monitors$ Traceback (most recent call last):

  File "/usr/local/bin/masakari-instancemonitor", line 10, in <module>

    sys.exit(main())

  File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/cmd/instancemonitor.py", line 31, in main

    config.parse_args(sys.argv)

  File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/config.py", line 32, in parse_args

    default_config_files=default_config_files)

  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2495, in __call__

    self._check_required_opts()

  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 3134, in _check_required_opts

    raise RequiredOptError(opt.name, group)

oslo_config.cfg.RequiredOptError: value required for option auth-url in group [api]

 

[1]+  Exit 1                  sudo masakari-instancemonitor

 

Greg.

 

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:17 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

thanks Honjo,

 

I’ll try this out now.

 

I’m assuming that it is ok to start ONLY the instance-monitor if that’s all i want to test.

Is that correct ?

 

In that case, following the instructions you pointed me at,

I would do the following:

·         setup devstack

o   i.e. with ‘enable_plugin masakari git://git.openstack.org/openstack/masakari’

·          

·         ( don’t need corosync and pacemaker )

·          

·         install & startup client

o   cd

o   git clone https://github.com/openstack/python-masakariclient.git

o   cd python-masakariclient

o   sudo python setup.py build

o   sudo python setup.py install

·          

·         install & startup instance monitor

o   cd

o   git clone https://github.com/openstack/masakari-monitors.git

o   sudo mkdir /etc/masakarimonitors

o   cd masakari-monitors

o   sudo python setup.py build

o   sudo python setup.py install

o    

o   sudo masakari-instancemonitor &

 

Is this correct ?

 

Greg.

 

 

 

 

From: Rikimaru Honjo <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, December 7, 2017 at 12:24 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Hello Greg,

 

I forgot to tell you.

 

Please use process_list.yaml instead of proc.list.sample.

 

On 2017/12/07 14:03, Rikimaru Honjo wrote:

Hello Greg,

Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf.

You can generate it by "tox -egenconfig".

hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script.

masakarimonitors.conf is a configuration file for monitors implemented by python that you installed.

And, we are preparing setting guides.

Please see it if you are good.

masakari:

masakari-monitors:

Best regards,

On 2017/12/06 22:48, Waines, Greg wrote:

I am just getting started working with masakari.

I am working on master.

 

I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes.

 

I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors .

Specifically:

# install & startup monitors

cd

cd masakari-monitors

sudo python setup.py install

cd

sudo mkdir /etc/masakarimonitors

sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list

cd ~/masakari-monitors/masakarimonitors/cmd

sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list &

sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf &

sudo /usr/bin/python ./instancemonitor.py &

 

However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file.

 

Is this the correct way to startup masakari instance monitoring ?

 

Greg.

 

 

 

 

My Masakari setup in Devstack

 

sudo useradd -s /bin/bash -d /opt/stack -m stack

echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack

sudo su - stack

 

cd devstack

 

                                                 local.conf file:

[[local|localrc]]

ADMIN_PASSWORD=admin

DATABASE_PASSWORD=admin

RABBIT_PASSWORD=admin

SERVICE_PASSWORD=admin

# setup Neutron services

disable_service n-net

enable_service q-svc

enable_service q-agt

enable_service q-dhcp

enable_service q-l3

enable_service q-meta

# ceilometer

# heat

# vitrage

# masakari

enable_plugin masakari git://git.openstack.org/openstack/masakari

 

 

./stack.sh

 

 

 

 

 

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]?subject:unsubscribe

 

 

--

_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

Rikimaru Honjo

 

 

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]?subject:unsubscribe

 


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [masakari] problems starting up masakari instance monitoring in devstack @ master

Waines, Greg

Went back and started up corosync and pacemaker and now masakari-instancemonitor starts successfully.

 

stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &

[1] 25973

stack@devstack-masakari:~/masakari-monitors$ 2017-12-11 12:47:49.483 25974 INFO masakarimonitors.service [-] Starting masakarimonitors-instancemonitor

 

stack@devstack-masakari:~/masakari-monitors

 

I don’t see any log file in /var/log for masakari-instancemonitor or masakari-engine ?

 

All processes seem to be up and running fine:

stack@devstack-masakari:~/masakari-monitors$ ps -ef | fgrep masakari

stack    11625     1  0 12:29 ?        00:00:10 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

stack    11778     1  0 12:29 ?        00:00:02 /usr/bin/python /usr/local/bin/masakari-engine --config-file=/etc/masakari/masakari.conf --debug

stack    12005 11625  0 12:29 ?        00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

stack    12006 11625  0 12:29 ?        00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

root     19336     1  0 12:19 ?        00:00:05 /opt/stack/bin/etcd --name devstack-masakari --data-dir /opt/stack/data/etcd --initial-cluster-state new --initial-cluster-token etcd-cluster-01 --initial-cluster devstack-masakari=http://10.10.10.7:2380 --initial-advertise-peer-urls http://10.10.10.7:2380 --advertise-client-urls http://10.10.10.7:2379 --listen-peer-urls http://0.0.0.0:2380 --listen-client-urls http://10.10.10.7:2379

root     25973 25760  0 12:47 pts/0    00:00:00 sudo masakari-instancemonitor

root     25974 25973  0 12:47 pts/0    00:00:00 /usr/bin/python /usr/local/bin/masakari-instancemonitor

stack    26123 25760  0 12:50 pts/0    00:00:00 grep -F --color=auto masakari

rabbitmq 26477 26348  0 12:06 ?        00:00:15 /usr/lib/erlang/erts-7.3/bin/beam.smp -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@devstack-masakari -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@devstack-masakari.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@devstack-masakari-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@devstack-masakari-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@devstack-masakari" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672

stack@devstack-masakari:~/masakari-monitors

 

 

Greg.

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:38 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Actually, i’m guessing this is wrong because masakari-instancemonitor fails to start when i use these instructions.

 

stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &

[1] 22959

stack@devstack-masakari:~/masakari-monitors$ Traceback (most recent call last):

  File "/usr/local/bin/masakari-instancemonitor", line 10, in <module>

    sys.exit(main())

  File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/cmd/instancemonitor.py", line 31, in main

    config.parse_args(sys.argv)

  File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/config.py", line 32, in parse_args

    default_config_files=default_config_files)

  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2495, in __call__

    self._check_required_opts()

  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 3134, in _check_required_opts

    raise RequiredOptError(opt.name, group)

oslo_config.cfg.RequiredOptError: value required for option auth-url in group [api]

 

[1]+  Exit 1                  sudo masakari-instancemonitor

 

Greg.

 

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:17 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

thanks Honjo,

 

I’ll try this out now.

 

I’m assuming that it is ok to start ONLY the instance-monitor if that’s all i want to test.

Is that correct ?

 

In that case, following the instructions you pointed me at,

I would do the following:

·         setup devstack

o   i.e. with ‘enable_plugin masakari git://git.openstack.org/openstack/masakari’

·          

·         ( don’t need corosync and pacemaker )

·          

·         install & startup client

o   cd

o   git clone https://github.com/openstack/python-masakariclient.git

o   cd python-masakariclient

o   sudo python setup.py build

o   sudo python setup.py install

·          

·         install & startup instance monitor

o   cd

o   git clone https://github.com/openstack/masakari-monitors.git

o   sudo mkdir /etc/masakarimonitors

o   cd masakari-monitors

o   sudo python setup.py build

o   sudo python setup.py install

o    

o   sudo masakari-instancemonitor &

 

Is this correct ?

 

Greg.

 

 

 

 

From: Rikimaru Honjo <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, December 7, 2017 at 12:24 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Hello Greg,

 

I forgot to tell you.

 

Please use process_list.yaml instead of proc.list.sample.

 

On 2017/12/07 14:03, Rikimaru Honjo wrote:

Hello Greg,

Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf.

You can generate it by "tox -egenconfig".

hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script.

masakarimonitors.conf is a configuration file for monitors implemented by python that you installed.

And, we are preparing setting guides.

Please see it if you are good.

masakari:

masakari-monitors:

Best regards,

On 2017/12/06 22:48, Waines, Greg wrote:

I am just getting started working with masakari.

I am working on master.

 

I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes.

 

I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors .

Specifically:

# install & startup monitors

cd

cd masakari-monitors

sudo python setup.py install

cd

sudo mkdir /etc/masakarimonitors

sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list

cd ~/masakari-monitors/masakarimonitors/cmd

sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list &

sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf &

sudo /usr/bin/python ./instancemonitor.py &

 

However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file.

 

Is this the correct way to startup masakari instance monitoring ?

 

Greg.

 

 

 

 

My Masakari setup in Devstack

 

sudo useradd -s /bin/bash -d /opt/stack -m stack

echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack

sudo su - stack

 

cd devstack

 

                                                 local.conf file:

[[local|localrc]]

ADMIN_PASSWORD=admin

DATABASE_PASSWORD=admin

RABBIT_PASSWORD=admin

SERVICE_PASSWORD=admin

# setup Neutron services

disable_service n-net

enable_service q-svc

enable_service q-agt

enable_service q-dhcp

enable_service q-l3

enable_service q-meta

# ceilometer

# heat

# vitrage

# masakari

enable_plugin masakari git://git.openstack.org/openstack/masakari

 

 

./stack.sh

 

 

 

 

 

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]?subject:unsubscribe

 

 

--

_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

Rikimaru Honjo

 

 

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]?subject:unsubscribe

 


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [masakari] problems starting up masakari instance monitoring in devstack @ master

Waines, Greg

I started up a VM and then deleted the VM ... and got some errors that make me think i don’t have the masakari client setup correctly.

Any ideas ?

Greg.

 

i.e.

stack@devstack-masakari:~/devstack$ nova list

 

 

+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+

| ID                                   | Name        | Status | Task State | Power State | Networks                                                |

+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+

| de8922c6-450d-4e2e-954d-ee4bd05ab909 | vm-1-cirros | ACTIVE | -          | Running     | private=10.0.0.10, fd1a:8f71:1a96:0:f816:3eff:fe5e:f315 |

+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+

stack@devstack-masakari:~/devstack

stack@devstack-masakari:~/devstack$ nova delete vm-1-cirros

Request to delete server vm-1-cirros has been accepted.

stack@devstack-masakari:~/devstack$ 2017-12-11 12:58:28.319 25974 INFO masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event: type=VM, hostname=devstack-masakari, uuid=de8922c6-450d-4e2e-954d-ee4bd05ab909, time=2017-12-11 12:58:28.318781, event_id=LIFECYCLE, detail=STOPPED_DESTROYED)

2017-12-11 12:58:28.319 25974 INFO masakarimonitors.ha.masakari [-] Send a notification. {'notification': {'hostname': 'devstack-masakari', 'type': 'VM', 'payload': {'instance_uuid': 'de8922c6-450d-4e2e-954d-ee4bd05ab909', 'vir_domain_event': 'STOPPED_DESTROYED', 'event': 'LIFECYCLE'}, 'generated_time': datetime.datetime(2017, 12, 11, 12, 58, 28, 318781)}}

2017-12-11 12:58:28.351 25974 WARNING masakarimonitors.ha.masakari [-] Retry sending a notification. (HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.): HttpException: HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.

... several times ... and then eventually

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari Traceback (most recent call last):

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/ha/masakari.py", line 91, in send_notification

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     payload=event['notification']['payload'])

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/masakariclient/sdk/ha/v1/_proxy.py", line 65, in create_notification

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self._create(_notification.Notification, **attrs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/proxy2.py", line 194, in _create

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return res.create(self._session)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/resource2.py", line 588, in create

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     json=request.body, headers=request.headers)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.request(url, 'POST', **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 64, in map_exceptions_wrapper

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return func(*args, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 352, in request

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return super(Session, self).request(*args, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in request

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     auth_headers = self.get_auth_headers(auth)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in get_auth_headers

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return auth.get_headers(self, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     token = self.get_token(session)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.get_access(session).auth_token

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     self.auth_ref = self.get_auth_ref(session)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self._plugin.get_auth_ref(session, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 165, in get_auth_ref

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     authenticated=False, log=False, **rkwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.request(url, 'POST', **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 66, in map_exceptions_wrapper

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     raise exceptions.from_exception(e)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari HttpException: HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-3640dd4f-b9d9-4a10-ae37-43a0e202393f), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari 

stack@devstack-masakari:~/devstack

 

 

Greg.

 

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:52 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Went back and started up corosync and pacemaker and now masakari-instancemonitor starts successfully.

 

stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &

[1] 25973

stack@devstack-masakari:~/masakari-monitors$ 2017-12-11 12:47:49.483 25974 INFO masakarimonitors.service [-] Starting masakarimonitors-instancemonitor

 

stack@devstack-masakari:~/masakari-monitors

 

I don’t see any log file in /var/log for masakari-instancemonitor or masakari-engine ?

 

All processes seem to be up and running fine:

stack@devstack-masakari:~/masakari-monitors$ ps -ef | fgrep masakari

stack    11625     1  0 12:29 ?        00:00:10 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

stack    11778     1  0 12:29 ?        00:00:02 /usr/bin/python /usr/local/bin/masakari-engine --config-file=/etc/masakari/masakari.conf --debug

stack    12005 11625  0 12:29 ?        00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

stack    12006 11625  0 12:29 ?        00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

root     19336     1  0 12:19 ?        00:00:05 /opt/stack/bin/etcd --name devstack-masakari --data-dir /opt/stack/data/etcd --initial-cluster-state new --initial-cluster-token etcd-cluster-01 --initial-cluster devstack-masakari=http://10.10.10.7:2380 --initial-advertise-peer-urls http://10.10.10.7:2380 --advertise-client-urls http://10.10.10.7:2379 --listen-peer-urls http://0.0.0.0:2380 --listen-client-urls http://10.10.10.7:2379

root     25973 25760  0 12:47 pts/0    00:00:00 sudo masakari-instancemonitor

root     25974 25973  0 12:47 pts/0    00:00:00 /usr/bin/python /usr/local/bin/masakari-instancemonitor

stack    26123 25760  0 12:50 pts/0    00:00:00 grep -F --color=auto masakari

rabbitmq 26477 26348  0 12:06 ?        00:00:15 /usr/lib/erlang/erts-7.3/bin/beam.smp -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@devstack-masakari -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@devstack-masakari.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@devstack-masakari-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@devstack-masakari-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@devstack-masakari" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672

stack@devstack-masakari:~/masakari-monitors

 

 

Greg.

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:38 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Actually, i’m guessing this is wrong because masakari-instancemonitor fails to start when i use these instructions.

 

stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &

[1] 22959

stack@devstack-masakari:~/masakari-monitors$ Traceback (most recent call last):

  File "/usr/local/bin/masakari-instancemonitor", line 10, in <module>

    sys.exit(main())

  File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/cmd/instancemonitor.py", line 31, in main

    config.parse_args(sys.argv)

  File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/config.py", line 32, in parse_args

    default_config_files=default_config_files)

  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2495, in __call__

    self._check_required_opts()

  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 3134, in _check_required_opts

    raise RequiredOptError(opt.name, group)

oslo_config.cfg.RequiredOptError: value required for option auth-url in group [api]

 

[1]+  Exit 1                  sudo masakari-instancemonitor

 

Greg.

 

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:17 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

thanks Honjo,

 

I’ll try this out now.

 

I’m assuming that it is ok to start ONLY the instance-monitor if that’s all i want to test.

Is that correct ?

 

In that case, following the instructions you pointed me at,

I would do the following:

·         setup devstack

o   i.e. with ‘enable_plugin masakari git://git.openstack.org/openstack/masakari’

·          

·         ( don’t need corosync and pacemaker )

·          

·         install & startup client

o   cd

o   git clone https://github.com/openstack/python-masakariclient.git

o   cd python-masakariclient

o   sudo python setup.py build

o   sudo python setup.py install

·          

·         install & startup instance monitor

o   cd

o   git clone https://github.com/openstack/masakari-monitors.git

o   sudo mkdir /etc/masakarimonitors

o   cd masakari-monitors

o   sudo python setup.py build

o   sudo python setup.py install

o    

o   sudo masakari-instancemonitor &

 

Is this correct ?

 

Greg.

 

 

 

 

From: Rikimaru Honjo <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, December 7, 2017 at 12:24 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Hello Greg,

 

I forgot to tell you.

 

Please use process_list.yaml instead of proc.list.sample.

 

On 2017/12/07 14:03, Rikimaru Honjo wrote:

Hello Greg,

Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf.

You can generate it by "tox -egenconfig".

hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script.

masakarimonitors.conf is a configuration file for monitors implemented by python that you installed.

And, we are preparing setting guides.

Please see it if you are good.

masakari:

masakari-monitors:

Best regards,

On 2017/12/06 22:48, Waines, Greg wrote:

I am just getting started working with masakari.

I am working on master.

 

I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes.

 

I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors .

Specifically:

# install & startup monitors

cd

cd masakari-monitors

sudo python setup.py install

cd

sudo mkdir /etc/masakarimonitors

sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list

cd ~/masakari-monitors/masakarimonitors/cmd

sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list &

sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf &

sudo /usr/bin/python ./instancemonitor.py &

 

However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file.

 

Is this the correct way to startup masakari instance monitoring ?

 

Greg.

 

 

 

 

My Masakari setup in Devstack

 

sudo useradd -s /bin/bash -d /opt/stack -m stack

echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack

sudo su - stack

 

cd devstack

 

                                                 local.conf file:

[[local|localrc]]

ADMIN_PASSWORD=admin

DATABASE_PASSWORD=admin

RABBIT_PASSWORD=admin

SERVICE_PASSWORD=admin

# setup Neutron services

disable_service n-net

enable_service q-svc

enable_service q-agt

enable_service q-dhcp

enable_service q-l3

enable_service q-meta

# ceilometer

# heat

# vitrage

# masakari

enable_plugin masakari git://git.openstack.org/openstack/masakari

 

 

./stack.sh

 

 

 

 

 

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]?subject:unsubscribe

 

 

--

_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

Rikimaru Honjo

 

 

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]?subject:unsubscribe

 


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [masakari] problems starting up masakari instance monitoring in devstack @ master

Waines, Greg

Any thoughts on what i don’t have setup correctly wrt the masakari client from the errors below ?

Greg.

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 8:05 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

I started up a VM and then deleted the VM ... and got some errors that make me think i don’t have the masakari client setup correctly.

Any ideas ?

Greg.

 

i.e.

stack@devstack-masakari:~/devstack$ nova list

 

 

+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+

| ID                                   | Name        | Status | Task State | Power State | Networks                                                |

+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+

| de8922c6-450d-4e2e-954d-ee4bd05ab909 | vm-1-cirros | ACTIVE | -          | Running     | private=10.0.0.10, fd1a:8f71:1a96:0:f816:3eff:fe5e:f315 |

+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+

stack@devstack-masakari:~/devstack

stack@devstack-masakari:~/devstack$ nova delete vm-1-cirros

Request to delete server vm-1-cirros has been accepted.

stack@devstack-masakari:~/devstack$ 2017-12-11 12:58:28.319 25974 INFO masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event: type=VM, hostname=devstack-masakari, uuid=de8922c6-450d-4e2e-954d-ee4bd05ab909, time=2017-12-11 12:58:28.318781, event_id=LIFECYCLE, detail=STOPPED_DESTROYED)

2017-12-11 12:58:28.319 25974 INFO masakarimonitors.ha.masakari [-] Send a notification. {'notification': {'hostname': 'devstack-masakari', 'type': 'VM', 'payload': {'instance_uuid': 'de8922c6-450d-4e2e-954d-ee4bd05ab909', 'vir_domain_event': 'STOPPED_DESTROYED', 'event': 'LIFECYCLE'}, 'generated_time': datetime.datetime(2017, 12, 11, 12, 58, 28, 318781)}}

2017-12-11 12:58:28.351 25974 WARNING masakarimonitors.ha.masakari [-] Retry sending a notification. (HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.): HttpException: HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.

... several times ... and then eventually

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari Traceback (most recent call last):

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/ha/masakari.py", line 91, in send_notification

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     payload=event['notification']['payload'])

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/masakariclient/sdk/ha/v1/_proxy.py", line 65, in create_notification

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self._create(_notification.Notification, **attrs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/proxy2.py", line 194, in _create

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return res.create(self._session)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/resource2.py", line 588, in create

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     json=request.body, headers=request.headers)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.request(url, 'POST', **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 64, in map_exceptions_wrapper

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return func(*args, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 352, in request

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return super(Session, self).request(*args, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in request

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     auth_headers = self.get_auth_headers(auth)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in get_auth_headers

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return auth.get_headers(self, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     token = self.get_token(session)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.get_access(session).auth_token

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     self.auth_ref = self.get_auth_ref(session)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self._plugin.get_auth_ref(session, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 165, in get_auth_ref

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     authenticated=False, log=False, **rkwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.request(url, 'POST', **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 66, in map_exceptions_wrapper

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     raise exceptions.from_exception(e)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari HttpException: HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-3640dd4f-b9d9-4a10-ae37-43a0e202393f), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari 

stack@devstack-masakari:~/devstack

 

 

Greg.

 

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:52 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Went back and started up corosync and pacemaker and now masakari-instancemonitor starts successfully.

 

stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &

[1] 25973

stack@devstack-masakari:~/masakari-monitors$ 2017-12-11 12:47:49.483 25974 INFO masakarimonitors.service [-] Starting masakarimonitors-instancemonitor

 

stack@devstack-masakari:~/masakari-monitors

 

I don’t see any log file in /var/log for masakari-instancemonitor or masakari-engine ?

 

All processes seem to be up and running fine:

stack@devstack-masakari:~/masakari-monitors$ ps -ef | fgrep masakari

stack    11625     1  0 12:29 ?        00:00:10 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

stack    11778     1  0 12:29 ?        00:00:02 /usr/bin/python /usr/local/bin/masakari-engine --config-file=/etc/masakari/masakari.conf --debug

stack    12005 11625  0 12:29 ?        00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

stack    12006 11625  0 12:29 ?        00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

root     19336     1  0 12:19 ?        00:00:05 /opt/stack/bin/etcd --name devstack-masakari --data-dir /opt/stack/data/etcd --initial-cluster-state new --initial-cluster-token etcd-cluster-01 --initial-cluster devstack-masakari=http://10.10.10.7:2380 --initial-advertise-peer-urls http://10.10.10.7:2380 --advertise-client-urls http://10.10.10.7:2379 --listen-peer-urls http://0.0.0.0:2380 --listen-client-urls http://10.10.10.7:2379

root     25973 25760  0 12:47 pts/0    00:00:00 sudo masakari-instancemonitor

root     25974 25973  0 12:47 pts/0    00:00:00 /usr/bin/python /usr/local/bin/masakari-instancemonitor

stack    26123 25760  0 12:50 pts/0    00:00:00 grep -F --color=auto masakari

rabbitmq 26477 26348  0 12:06 ?        00:00:15 /usr/lib/erlang/erts-7.3/bin/beam.smp -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@devstack-masakari -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@devstack-masakari.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@devstack-masakari-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@devstack-masakari-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@devstack-masakari" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672

stack@devstack-masakari:~/masakari-monitors

 

 

Greg.

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:38 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Actually, i’m guessing this is wrong because masakari-instancemonitor fails to start when i use these instructions.

 

stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &

[1] 22959

stack@devstack-masakari:~/masakari-monitors$ Traceback (most recent call last):

  File "/usr/local/bin/masakari-instancemonitor", line 10, in <module>

    sys.exit(main())

  File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/cmd/instancemonitor.py", line 31, in main

    config.parse_args(sys.argv)

  File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/config.py", line 32, in parse_args

    default_config_files=default_config_files)

  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2495, in __call__

    self._check_required_opts()

  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 3134, in _check_required_opts

    raise RequiredOptError(opt.name, group)

oslo_config.cfg.RequiredOptError: value required for option auth-url in group [api]

 

[1]+  Exit 1                  sudo masakari-instancemonitor

 

Greg.

 

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:17 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

thanks Honjo,

 

I’ll try this out now.

 

I’m assuming that it is ok to start ONLY the instance-monitor if that’s all i want to test.

Is that correct ?

 

In that case, following the instructions you pointed me at,

I would do the following:

·         setup devstack

o   i.e. with ‘enable_plugin masakari git://git.openstack.org/openstack/masakari’

·          

·         ( don’t need corosync and pacemaker )

·          

·         install & startup client

o   cd

o   git clone https://github.com/openstack/python-masakariclient.git

o   cd python-masakariclient

o   sudo python setup.py build

o   sudo python setup.py install

·          

·         install & startup instance monitor

o   cd

o   git clone https://github.com/openstack/masakari-monitors.git

o   sudo mkdir /etc/masakarimonitors

o   cd masakari-monitors

o   sudo python setup.py build

o   sudo python setup.py install

o    

o   sudo masakari-instancemonitor &

 

Is this correct ?

 

Greg.

 

 

 

 

From: Rikimaru Honjo <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, December 7, 2017 at 12:24 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Hello Greg,

 

I forgot to tell you.

 

Please use process_list.yaml instead of proc.list.sample.

 

On 2017/12/07 14:03, Rikimaru Honjo wrote:

Hello Greg,

Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf.

You can generate it by "tox -egenconfig".

hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script.

masakarimonitors.conf is a configuration file for monitors implemented by python that you installed.

And, we are preparing setting guides.

Please see it if you are good.

masakari:

masakari-monitors:

Best regards,

On 2017/12/06 22:48, Waines, Greg wrote:

I am just getting started working with masakari.

I am working on master.

 

I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes.

 

I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors .

Specifically:

# install & startup monitors

cd

cd masakari-monitors

sudo python setup.py install

cd

sudo mkdir /etc/masakarimonitors

sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list

cd ~/masakari-monitors/masakarimonitors/cmd

sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list &

sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf &

sudo /usr/bin/python ./instancemonitor.py &

 

However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file.

 

Is this the correct way to startup masakari instance monitoring ?

 

Greg.

 

 

 

 

My Masakari setup in Devstack

 

sudo useradd -s /bin/bash -d /opt/stack -m stack

echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack

sudo su - stack

 

cd devstack

 

                                                 local.conf file:

[[local|localrc]]

ADMIN_PASSWORD=admin

DATABASE_PASSWORD=admin

RABBIT_PASSWORD=admin

SERVICE_PASSWORD=admin

# setup Neutron services

disable_service n-net

enable_service q-svc

enable_service q-agt

enable_service q-dhcp

enable_service q-l3

enable_service q-meta

# ceilometer

# heat

# vitrage

# masakari

enable_plugin masakari git://git.openstack.org/openstack/masakari

 

 

./stack.sh

 

 

 

 

 

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]?subject:unsubscribe

 

 

--

_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

Rikimaru Honjo

 

 

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]?subject:unsubscribe

 


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [masakari] problems starting up masakari instance monitoring in devstack @ master

Waines, Greg

So i believe the error is:

HttpException: HttpException: Expecting to find domain in project.

 

I have attached the /etc/masakarimonitors/masakarimonitors.conf file and the /etc/masakari/masakari.conf file .

 

See the domain related attributes in each file below.

 

Is the Default vs default causing this problem ?

Are there other domain related attributes that should be set to default ?

 

stack@devstack-masakari:~$ fgrep -i domain /etc/masakari/masakari.conf 

project_domain_name = Default

user_domain_name = Default

 

stack@devstack-masakari:~$ fgrep -i domain /etc/masakarimonitors/masakarimonitors.conf 

#logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s

# Domain ID to scope to (string value)

#domain_id = <None>

# Domain name to scope to (string value)

#domain_name = <None>

# Domain ID containing project (string value)

#project_domain_id = <None>

# Domain name containing project (string value)

#project_domain_name = <None>

project_domain_name = default

# Optional domain ID to use with v3 and v2 parameters. It will be used for both

# the user and project domain in v3 and ignored in v2 authentication. (string

#default_domain_id = <None>

# Optional domain name to use with v3 API and v2 parameters. It will be used for

# both the user and project domain in v3 and ignored in v2 authentication.

#default_domain_name = <None>

# User's domain id (string value)

#user_domain_id = <None>

# User's domain name (string value)

#user_domain_name = <None>

# Indicate whether this resource may be shared with the domain received in the

stack@devstack-masakari:~

 

 

Greg.

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Tuesday, December 12, 2017 at 7:19 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Any thoughts on what i don’t have setup correctly wrt the masakari client from the errors below ?

Greg.

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 8:05 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

I started up a VM and then deleted the VM ... and got some errors that make me think i don’t have the masakari client setup correctly.

Any ideas ?

Greg.

 

i.e.

stack@devstack-masakari:~/devstack$ nova list

 

 

+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+

| ID                                   | Name        | Status | Task State | Power State | Networks                                                |

+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+

| de8922c6-450d-4e2e-954d-ee4bd05ab909 | vm-1-cirros | ACTIVE | -          | Running     | private=10.0.0.10, fd1a:8f71:1a96:0:f816:3eff:fe5e:f315 |

+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+

stack@devstack-masakari:~/devstack

stack@devstack-masakari:~/devstack$ nova delete vm-1-cirros

Request to delete server vm-1-cirros has been accepted.

stack@devstack-masakari:~/devstack$ 2017-12-11 12:58:28.319 25974 INFO masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event: type=VM, hostname=devstack-masakari, uuid=de8922c6-450d-4e2e-954d-ee4bd05ab909, time=2017-12-11 12:58:28.318781, event_id=LIFECYCLE, detail=STOPPED_DESTROYED)

2017-12-11 12:58:28.319 25974 INFO masakarimonitors.ha.masakari [-] Send a notification. {'notification': {'hostname': 'devstack-masakari', 'type': 'VM', 'payload': {'instance_uuid': 'de8922c6-450d-4e2e-954d-ee4bd05ab909', 'vir_domain_event': 'STOPPED_DESTROYED', 'event': 'LIFECYCLE'}, 'generated_time': datetime.datetime(2017, 12, 11, 12, 58, 28, 318781)}}

2017-12-11 12:58:28.351 25974 WARNING masakarimonitors.ha.masakari [-] Retry sending a notification. (HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.): HttpException: HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.

... several times ... and then eventually

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari Traceback (most recent call last):

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/ha/masakari.py", line 91, in send_notification

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     payload=event['notification']['payload'])

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/masakariclient/sdk/ha/v1/_proxy.py", line 65, in create_notification

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self._create(_notification.Notification, **attrs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/proxy2.py", line 194, in _create

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return res.create(self._session)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/resource2.py", line 588, in create

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     json=request.body, headers=request.headers)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.request(url, 'POST', **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 64, in map_exceptions_wrapper

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return func(*args, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 352, in request

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return super(Session, self).request(*args, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in request

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     auth_headers = self.get_auth_headers(auth)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in get_auth_headers

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return auth.get_headers(self, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     token = self.get_token(session)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.get_access(session).auth_token

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     self.auth_ref = self.get_auth_ref(session)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self._plugin.get_auth_ref(session, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 165, in get_auth_ref

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     authenticated=False, log=False, **rkwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.request(url, 'POST', **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 66, in map_exceptions_wrapper

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     raise exceptions.from_exception(e)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari HttpException: HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-3640dd4f-b9d9-4a10-ae37-43a0e202393f), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari 

stack@devstack-masakari:~/devstack

 

 

Greg.

 

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:52 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Went back and started up corosync and pacemaker and now masakari-instancemonitor starts successfully.

 

stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &

[1] 25973

stack@devstack-masakari:~/masakari-monitors$ 2017-12-11 12:47:49.483 25974 INFO masakarimonitors.service [-] Starting masakarimonitors-instancemonitor

 

stack@devstack-masakari:~/masakari-monitors

 

I don’t see any log file in /var/log for masakari-instancemonitor or masakari-engine ?

 

All processes seem to be up and running fine:

stack@devstack-masakari:~/masakari-monitors$ ps -ef | fgrep masakari

stack    11625     1  0 12:29 ?        00:00:10 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

stack    11778     1  0 12:29 ?        00:00:02 /usr/bin/python /usr/local/bin/masakari-engine --config-file=/etc/masakari/masakari.conf --debug

stack    12005 11625  0 12:29 ?        00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

stack    12006 11625  0 12:29 ?        00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

root     19336     1  0 12:19 ?        00:00:05 /opt/stack/bin/etcd --name devstack-masakari --data-dir /opt/stack/data/etcd --initial-cluster-state new --initial-cluster-token etcd-cluster-01 --initial-cluster devstack-masakari=http://10.10.10.7:2380 --initial-advertise-peer-urls http://10.10.10.7:2380 --advertise-client-urls http://10.10.10.7:2379 --listen-peer-urls http://0.0.0.0:2380 --listen-client-urls http://10.10.10.7:2379

root     25973 25760  0 12:47 pts/0    00:00:00 sudo masakari-instancemonitor

root     25974 25973  0 12:47 pts/0    00:00:00 /usr/bin/python /usr/local/bin/masakari-instancemonitor

stack    26123 25760  0 12:50 pts/0    00:00:00 grep -F --color=auto masakari

rabbitmq 26477 26348  0 12:06 ?        00:00:15 /usr/lib/erlang/erts-7.3/bin/beam.smp -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@devstack-masakari -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@devstack-masakari.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@devstack-masakari-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@devstack-masakari-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@devstack-masakari" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672

stack@devstack-masakari:~/masakari-monitors

 

 

Greg.

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:38 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Actually, i’m guessing this is wrong because masakari-instancemonitor fails to start when i use these instructions.

 

stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &

[1] 22959

stack@devstack-masakari:~/masakari-monitors$ Traceback (most recent call last):

  File "/usr/local/bin/masakari-instancemonitor", line 10, in <module>

    sys.exit(main())

  File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/cmd/instancemonitor.py", line 31, in main

    config.parse_args(sys.argv)

  File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/config.py", line 32, in parse_args

    default_config_files=default_config_files)

  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2495, in __call__

    self._check_required_opts()

  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 3134, in _check_required_opts

    raise RequiredOptError(opt.name, group)

oslo_config.cfg.RequiredOptError: value required for option auth-url in group [api]

 

[1]+  Exit 1                  sudo masakari-instancemonitor

 

Greg.

 

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:17 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

thanks Honjo,

 

I’ll try this out now.

 

I’m assuming that it is ok to start ONLY the instance-monitor if that’s all i want to test.

Is that correct ?

 

In that case, following the instructions you pointed me at,

I would do the following:

·         setup devstack

o   i.e. with ‘enable_plugin masakari git://git.openstack.org/openstack/masakari’

·          

·         ( don’t need corosync and pacemaker )

·          

·         install & startup client

o   cd

o   git clone https://github.com/openstack/python-masakariclient.git

o   cd python-masakariclient

o   sudo python setup.py build

o   sudo python setup.py install

·          

·         install & startup instance monitor

o   cd

o   git clone https://github.com/openstack/masakari-monitors.git

o   sudo mkdir /etc/masakarimonitors

o   cd masakari-monitors

o   sudo python setup.py build

o   sudo python setup.py install

o    

o   sudo masakari-instancemonitor &

 

Is this correct ?

 

Greg.

 

 

 

 

From: Rikimaru Honjo <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, December 7, 2017 at 12:24 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Hello Greg,

 

I forgot to tell you.

 

Please use process_list.yaml instead of proc.list.sample.

 

On 2017/12/07 14:03, Rikimaru Honjo wrote:

Hello Greg,

Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf.

You can generate it by "tox -egenconfig".

hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script.

masakarimonitors.conf is a configuration file for monitors implemented by python that you installed.

And, we are preparing setting guides.

Please see it if you are good.

masakari:

masakari-monitors:

Best regards,

On 2017/12/06 22:48, Waines, Greg wrote:

I am just getting started working with masakari.

I am working on master.

 

I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes.

 

I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors .

Specifically:

# install & startup monitors

cd

cd masakari-monitors

sudo python setup.py install

cd

sudo mkdir /etc/masakarimonitors

sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list

cd ~/masakari-monitors/masakarimonitors/cmd

sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list &

sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf &

sudo /usr/bin/python ./instancemonitor.py &

 

However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file.

 

Is this the correct way to startup masakari instance monitoring ?

 

Greg.

 

 

 

 

My Masakari setup in Devstack

 

sudo useradd -s /bin/bash -d /opt/stack -m stack

echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack

sudo su - stack

 

cd devstack

 

                                                 local.conf file:

[[local|localrc]]

ADMIN_PASSWORD=admin

DATABASE_PASSWORD=admin

RABBIT_PASSWORD=admin

SERVICE_PASSWORD=admin

# setup Neutron services

disable_service n-net

enable_service q-svc

enable_service q-agt

enable_service q-dhcp

enable_service q-l3

enable_service q-meta

# ceilometer

# heat

# vitrage

# masakari

enable_plugin masakari git://git.openstack.org/openstack/masakari

 

 

./stack.sh

 

 

 

 

 

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]?subject:unsubscribe

 

 

--

_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

Rikimaru Honjo

 

 

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]?subject:unsubscribe

 


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

masakarimonitors.conf (18K) Download Attachment
masakari.conf (18K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [masakari] problems starting up masakari instance monitoring in devstack @ master

Waines, Greg

ok ... i changed all the domain related attributes in /etc/masakarimonitors/masakarimonitors.conf to be set to ‘default’ .

I seemed to get further,

but now get the following error when instancemonitor tries to send a notification:

      Bad Request (HTTP 400) (Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari could not be found.

however:

- devstack-masakari is in the /etc/hosts

- nova hypervisor-list shows devstack-masakari as a hypervisor hostname

- i am running both hostmonitor, processmonitor and instancemonitor

 

any ideas ?

see details below,

Greg.

 

2017-12-13 13:50:34.353 7637 INFO masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event: type=VM, hostname=devstack-masakari, uuid=6ae0b09b-3e93-4f0c-b81b-fa140636f267, time=2017-12-13 13:50:34.353037, event_id=LIFECYCLE, detail=STOPPED_DESTROYED)

2017-12-13 13:50:34.353 7637 INFO masakarimonitors.ha.masakari [-] Send a notification. {'notification': {'hostname': 'devstack-masakari', 'type': 'VM', 'payload': {'instance_uuid': '6ae0b09b-3e93-4f0c-b81b-fa140636f267', 'vir_domain_event': 'STOPPED_DESTROYED', 'event': 'LIFECYCLE'}, 'generated_time': datetime.datetime(2017, 12, 13, 13, 50, 34, 353037)}}

2017-12-13 13:50:34.461 7637 WARNING masakarimonitors.ha.masakari [-] Retry sending a notification. (HttpException: Bad Request (HTTP 400) (Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari could not be found.): HttpException: HttpException: Bad Request (HTTP 400) (Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari could not be found.

^C2017-12-13 13:50:42.462 7637 INFO oslo_service.service [-] Caught SIGINT signal, instantaneous exiting

root@devstack-masakari:~# 

root@devstack-masakari:~# ping devstack-masakari

PING localhost (127.0.0.1) 56(84) bytes of data.

64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.049 ms

64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.022 ms

^C

--- localhost ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 999ms

rtt min/avg/max/mdev = 0.022/0.035/0.049/0.014 ms

root@devstack-masakari:~# 

 

stack@devstack-masakari:~$ nova hypervisor-list

+--------------------------------------+---------------------+-------+---------+

| ID                                   | Hypervisor hostname | State | Status  |

+--------------------------------------+---------------------+-------+---------+

| 5fb1b09b-e5f5-465a-828a-2101135ff700 | devstack-masakari   | up    | enabled |

+--------------------------------------+---------------------+-------+---------+

stack@devstack-masakari:~

 

 

 

 

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Wednesday, December 13, 2017 at 8:17 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

So i believe the error is:

HttpException: HttpException: Expecting to find domain in project.

 

I have attached the /etc/masakarimonitors/masakarimonitors.conf file and the /etc/masakari/masakari.conf file .

 

See the domain related attributes in each file below.

 

Is the Default vs default causing this problem ?

Are there other domain related attributes that should be set to default ?

 

stack@devstack-masakari:~$ fgrep -i domain /etc/masakari/masakari.conf 

project_domain_name = Default

user_domain_name = Default

 

stack@devstack-masakari:~$ fgrep -i domain /etc/masakarimonitors/masakarimonitors.conf 

#logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s

# Domain ID to scope to (string value)

#domain_id = <None>

# Domain name to scope to (string value)

#domain_name = <None>

# Domain ID containing project (string value)

#project_domain_id = <None>

# Domain name containing project (string value)

#project_domain_name = <None>

project_domain_name = default

# Optional domain ID to use with v3 and v2 parameters. It will be used for both

# the user and project domain in v3 and ignored in v2 authentication. (string

#default_domain_id = <None>

# Optional domain name to use with v3 API and v2 parameters. It will be used for

# both the user and project domain in v3 and ignored in v2 authentication.

#default_domain_name = <None>

# User's domain id (string value)

#user_domain_id = <None>

# User's domain name (string value)

#user_domain_name = <None>

# Indicate whether this resource may be shared with the domain received in the

stack@devstack-masakari:~

 

 

Greg.

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Tuesday, December 12, 2017 at 7:19 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Any thoughts on what i don’t have setup correctly wrt the masakari client from the errors below ?

Greg.

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 8:05 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

I started up a VM and then deleted the VM ... and got some errors that make me think i don’t have the masakari client setup correctly.

Any ideas ?

Greg.

 

i.e.

stack@devstack-masakari:~/devstack$ nova list

 

 

+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+

| ID                                   | Name        | Status | Task State | Power State | Networks                                                |

+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+

| de8922c6-450d-4e2e-954d-ee4bd05ab909 | vm-1-cirros | ACTIVE | -          | Running     | private=10.0.0.10, fd1a:8f71:1a96:0:f816:3eff:fe5e:f315 |

+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+

stack@devstack-masakari:~/devstack

stack@devstack-masakari:~/devstack$ nova delete vm-1-cirros

Request to delete server vm-1-cirros has been accepted.

stack@devstack-masakari:~/devstack$ 2017-12-11 12:58:28.319 25974 INFO masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event: type=VM, hostname=devstack-masakari, uuid=de8922c6-450d-4e2e-954d-ee4bd05ab909, time=2017-12-11 12:58:28.318781, event_id=LIFECYCLE, detail=STOPPED_DESTROYED)

2017-12-11 12:58:28.319 25974 INFO masakarimonitors.ha.masakari [-] Send a notification. {'notification': {'hostname': 'devstack-masakari', 'type': 'VM', 'payload': {'instance_uuid': 'de8922c6-450d-4e2e-954d-ee4bd05ab909', 'vir_domain_event': 'STOPPED_DESTROYED', 'event': 'LIFECYCLE'}, 'generated_time': datetime.datetime(2017, 12, 11, 12, 58, 28, 318781)}}

2017-12-11 12:58:28.351 25974 WARNING masakarimonitors.ha.masakari [-] Retry sending a notification. (HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.): HttpException: HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.

... several times ... and then eventually

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari Traceback (most recent call last):

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/ha/masakari.py", line 91, in send_notification

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     payload=event['notification']['payload'])

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/masakariclient/sdk/ha/v1/_proxy.py", line 65, in create_notification

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self._create(_notification.Notification, **attrs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/proxy2.py", line 194, in _create

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return res.create(self._session)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/resource2.py", line 588, in create

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     json=request.body, headers=request.headers)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.request(url, 'POST', **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 64, in map_exceptions_wrapper

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return func(*args, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 352, in request

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return super(Session, self).request(*args, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in request

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     auth_headers = self.get_auth_headers(auth)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in get_auth_headers

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return auth.get_headers(self, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     token = self.get_token(session)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.get_access(session).auth_token

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     self.auth_ref = self.get_auth_ref(session)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self._plugin.get_auth_ref(session, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 165, in get_auth_ref

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     authenticated=False, log=False, **rkwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.request(url, 'POST', **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 66, in map_exceptions_wrapper

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     raise exceptions.from_exception(e)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari HttpException: HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-3640dd4f-b9d9-4a10-ae37-43a0e202393f), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari 

stack@devstack-masakari:~/devstack

 

 

Greg.

 

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:52 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Went back and started up corosync and pacemaker and now masakari-instancemonitor starts successfully.

 

stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &

[1] 25973

stack@devstack-masakari:~/masakari-monitors$ 2017-12-11 12:47:49.483 25974 INFO masakarimonitors.service [-] Starting masakarimonitors-instancemonitor

 

stack@devstack-masakari:~/masakari-monitors

 

I don’t see any log file in /var/log for masakari-instancemonitor or masakari-engine ?

 

All processes seem to be up and running fine:

stack@devstack-masakari:~/masakari-monitors$ ps -ef | fgrep masakari

stack    11625     1  0 12:29 ?        00:00:10 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

stack    11778     1  0 12:29 ?        00:00:02 /usr/bin/python /usr/local/bin/masakari-engine --config-file=/etc/masakari/masakari.conf --debug

stack    12005 11625  0 12:29 ?        00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

stack    12006 11625  0 12:29 ?        00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

root     19336     1  0 12:19 ?        00:00:05 /opt/stack/bin/etcd --name devstack-masakari --data-dir /opt/stack/data/etcd --initial-cluster-state new --initial-cluster-token etcd-cluster-01 --initial-cluster devstack-masakari=http://10.10.10.7:2380 --initial-advertise-peer-urls http://10.10.10.7:2380 --advertise-client-urls http://10.10.10.7:2379 --listen-peer-urls http://0.0.0.0:2380 --listen-client-urls http://10.10.10.7:2379

root     25973 25760  0 12:47 pts/0    00:00:00 sudo masakari-instancemonitor

root     25974 25973  0 12:47 pts/0    00:00:00 /usr/bin/python /usr/local/bin/masakari-instancemonitor

stack    26123 25760  0 12:50 pts/0    00:00:00 grep -F --color=auto masakari

rabbitmq 26477 26348  0 12:06 ?        00:00:15 /usr/lib/erlang/erts-7.3/bin/beam.smp -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@devstack-masakari -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@devstack-masakari.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@devstack-masakari-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@devstack-masakari-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@devstack-masakari" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672

stack@devstack-masakari:~/masakari-monitors

 

 

Greg.

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:38 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Actually, i’m guessing this is wrong because masakari-instancemonitor fails to start when i use these instructions.

 

stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &

[1] 22959

stack@devstack-masakari:~/masakari-monitors$ Traceback (most recent call last):

  File "/usr/local/bin/masakari-instancemonitor", line 10, in <module>

    sys.exit(main())

  File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/cmd/instancemonitor.py", line 31, in main

    config.parse_args(sys.argv)

  File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/config.py", line 32, in parse_args

    default_config_files=default_config_files)

  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2495, in __call__

    self._check_required_opts()

  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 3134, in _check_required_opts

    raise RequiredOptError(opt.name, group)

oslo_config.cfg.RequiredOptError: value required for option auth-url in group [api]

 

[1]+  Exit 1                  sudo masakari-instancemonitor

 

Greg.

 

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:17 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

thanks Honjo,

 

I’ll try this out now.

 

I’m assuming that it is ok to start ONLY the instance-monitor if that’s all i want to test.

Is that correct ?

 

In that case, following the instructions you pointed me at,

I would do the following:

·         setup devstack

o   i.e. with ‘enable_plugin masakari git://git.openstack.org/openstack/masakari’

·          

·         ( don’t need corosync and pacemaker )

·          

·         install & startup client

o   cd

o   git clone https://github.com/openstack/python-masakariclient.git

o   cd python-masakariclient

o   sudo python setup.py build

o   sudo python setup.py install

·          

·         install & startup instance monitor

o   cd

o   git clone https://github.com/openstack/masakari-monitors.git

o   sudo mkdir /etc/masakarimonitors

o   cd masakari-monitors

o   sudo python setup.py build

o   sudo python setup.py install

o    

o   sudo masakari-instancemonitor &

 

Is this correct ?

 

Greg.

 

 

 

 

From: Rikimaru Honjo <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, December 7, 2017 at 12:24 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Hello Greg,

 

I forgot to tell you.

 

Please use process_list.yaml instead of proc.list.sample.

 

On 2017/12/07 14:03, Rikimaru Honjo wrote:

Hello Greg,

Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf.

You can generate it by "tox -egenconfig".

hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script.

masakarimonitors.conf is a configuration file for monitors implemented by python that you installed.

And, we are preparing setting guides.

Please see it if you are good.

masakari:

masakari-monitors:

Best regards,

On 2017/12/06 22:48, Waines, Greg wrote:

I am just getting started working with masakari.

I am working on master.

 

I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes.

 

I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors .

Specifically:

# install & startup monitors

cd

cd masakari-monitors

sudo python setup.py install

cd

sudo mkdir /etc/masakarimonitors

sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list

cd ~/masakari-monitors/masakarimonitors/cmd

sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list &

sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf &

sudo /usr/bin/python ./instancemonitor.py &

 

However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file.

 

Is this the correct way to startup masakari instance monitoring ?

 

Greg.

 

 

 

 

My Masakari setup in Devstack

 

sudo useradd -s /bin/bash -d /opt/stack -m stack

echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack

sudo su - stack

 

cd devstack

 

                                                 local.conf file:

[[local|localrc]]

ADMIN_PASSWORD=admin

DATABASE_PASSWORD=admin

RABBIT_PASSWORD=admin

SERVICE_PASSWORD=admin

# setup Neutron services

disable_service n-net

enable_service q-svc

enable_service q-agt

enable_service q-dhcp

enable_service q-l3

enable_service q-meta

# ceilometer

# heat

# vitrage

# masakari

enable_plugin masakari git://git.openstack.org/openstack/masakari

 

 

./stack.sh

 

 

 

 

 

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]?subject:unsubscribe

 

 

--

_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

Rikimaru Honjo

 

 

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]?subject:unsubscribe

 


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [masakari] problems starting up masakari instance monitoring in devstack @ master

Bhor, Dinesh

Hi Greg,


Looks like you don’t have “devstack-masakari” host registered in Masakari database.


Currently operator have to add all the hosts from failover-segment manually to Masakari database.


You can use below command:


Register failover-segment to Masakari database:

openstack segment create <name> <recovery_method> <service_type>


Register hosts under the created segment to Masakari database:

openstack segment host create <name> <type> <control_attributes> <segment-id>



There is a work in progress on auto compute node registration.

Please refer: https://blueprints.launchpad.net/masakari-monitors/+spec/auto-compute-node-registration



Thank you,

Dinesh Bhor


From: Waines, Greg <[hidden email]>
Sent: 13 December 2017 19:26:32
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master
 

ok ... i changed all the domain related attributes in /etc/masakarimonitors/masakarimonitors.conf to be set to ‘default’ .

I seemed to get further,

but now get the following error when instancemonitor tries to send a notification:

      Bad Request (HTTP 400) (Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari could not be found.

however:

- devstack-masakari is in the /etc/hosts

- nova hypervisor-list shows devstack-masakari as a hypervisor hostname

- i am running both hostmonitor, processmonitor and instancemonitor

 

any ideas ?

see details below,

Greg.

 

2017-12-13 13:50:34.353 7637 INFO masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event: type=VM, hostname=devstack-masakari, uuid=6ae0b09b-3e93-4f0c-b81b-fa140636f267, time=2017-12-13 13:50:34.353037, event_id=LIFECYCLE, detail=STOPPED_DESTROYED)

2017-12-13 13:50:34.353 7637 INFO masakarimonitors.ha.masakari [-] Send a notification. {'notification': {'hostname': 'devstack-masakari', 'type': 'VM', 'payload': {'instance_uuid': '6ae0b09b-3e93-4f0c-b81b-fa140636f267', 'vir_domain_event': 'STOPPED_DESTROYED', 'event': 'LIFECYCLE'}, 'generated_time': datetime.datetime(2017, 12, 13, 13, 50, 34, 353037)}}

2017-12-13 13:50:34.461 7637 WARNING masakarimonitors.ha.masakari [-] Retry sending a notification. (HttpException: Bad Request (HTTP 400) (Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari could not be found.): HttpException: HttpException: Bad Request (HTTP 400) (Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari could not be found.

^C2017-12-13 13:50:42.462 7637 INFO oslo_service.service [-] Caught SIGINT signal, instantaneous exiting

root@devstack-masakari:~# 

root@devstack-masakari:~# ping devstack-masakari

PING localhost (127.0.0.1) 56(84) bytes of data.

64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.049 ms

64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.022 ms

^C

--- localhost ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 999ms

rtt min/avg/max/mdev = 0.022/0.035/0.049/0.014 ms

root@devstack-masakari:~# 

 

stack@devstack-masakari:~$ nova hypervisor-list

+--------------------------------------+---------------------+-------+---------+

| ID                                   | Hypervisor hostname | State | Status  |

+--------------------------------------+---------------------+-------+---------+

| 5fb1b09b-e5f5-465a-828a-2101135ff700 | devstack-masakari   | up    | enabled |

+--------------------------------------+---------------------+-------+---------+

stack@devstack-masakari:~

 

 

 

 

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Wednesday, December 13, 2017 at 8:17 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

So i believe the error is:

HttpException: HttpException: Expecting to find domain in project.

 

I have attached the /etc/masakarimonitors/masakarimonitors.conf file and the /etc/masakari/masakari.conf file .

 

See the domain related attributes in each file below.

 

Is the Default vs default causing this problem ?

Are there other domain related attributes that should be set to default ?

 

stack@devstack-masakari:~$ fgrep -i domain /etc/masakari/masakari.conf 

project_domain_name = Default

user_domain_name = Default

 

stack@devstack-masakari:~$ fgrep -i domain /etc/masakarimonitors/masakarimonitors.conf 

#logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s

# Domain ID to scope to (string value)

#domain_id = <None>

# Domain name to scope to (string value)

#domain_name = <None>

# Domain ID containing project (string value)

#project_domain_id = <None>

# Domain name containing project (string value)

#project_domain_name = <None>

project_domain_name = default

# Optional domain ID to use with v3 and v2 parameters. It will be used for both

# the user and project domain in v3 and ignored in v2 authentication. (string

#default_domain_id = <None>

# Optional domain name to use with v3 API and v2 parameters. It will be used for

# both the user and project domain in v3 and ignored in v2 authentication.

#default_domain_name = <None>

# User's domain id (string value)

#user_domain_id = <None>

# User's domain name (string value)

#user_domain_name = <None>

# Indicate whether this resource may be shared with the domain received in the

stack@devstack-masakari:~

 

 

Greg.

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Tuesday, December 12, 2017 at 7:19 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Any thoughts on what i don’t have setup correctly wrt the masakari client from the errors below ?

Greg.

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 8:05 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

I started up a VM and then deleted the VM ... and got some errors that make me think i don’t have the masakari client setup correctly.

Any ideas ?

Greg.

 

i.e.

stack@devstack-masakari:~/devstack$ nova list

 

 

+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+

| ID                                   | Name        | Status | Task State | Power State | Networks                                                |

+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+

| de8922c6-450d-4e2e-954d-ee4bd05ab909 | vm-1-cirros | ACTIVE | -          | Running     | private=10.0.0.10, fd1a:8f71:1a96:0:f816:3eff:fe5e:f315 |

+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+

stack@devstack-masakari:~/devstack

stack@devstack-masakari:~/devstack$ nova delete vm-1-cirros

Request to delete server vm-1-cirros has been accepted.

stack@devstack-masakari:~/devstack$ 2017-12-11 12:58:28.319 25974 INFO masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event: type=VM, hostname=devstack-masakari, uuid=de8922c6-450d-4e2e-954d-ee4bd05ab909, time=2017-12-11 12:58:28.318781, event_id=LIFECYCLE, detail=STOPPED_DESTROYED)

2017-12-11 12:58:28.319 25974 INFO masakarimonitors.ha.masakari [-] Send a notification. {'notification': {'hostname': 'devstack-masakari', 'type': 'VM', 'payload': {'instance_uuid': 'de8922c6-450d-4e2e-954d-ee4bd05ab909', 'vir_domain_event': 'STOPPED_DESTROYED', 'event': 'LIFECYCLE'}, 'generated_time': datetime.datetime(2017, 12, 11, 12, 58, 28, 318781)}}

2017-12-11 12:58:28.351 25974 WARNING masakarimonitors.ha.masakari [-] Retry sending a notification. (HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.): HttpException: HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.

... several times ... and then eventually

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari Traceback (most recent call last):

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/ha/masakari.py", line 91, in send_notification

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     payload=event['notification']['payload'])

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/masakariclient/sdk/ha/v1/_proxy.py", line 65, in create_notification

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self._create(_notification.Notification, **attrs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/proxy2.py", line 194, in _create

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return res.create(self._session)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/resource2.py", line 588, in create

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     json=request.body, headers=request.headers)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.request(url, 'POST', **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 64, in map_exceptions_wrapper

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return func(*args, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 352, in request

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return super(Session, self).request(*args, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in request

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     auth_headers = self.get_auth_headers(auth)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in get_auth_headers

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return auth.get_headers(self, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     token = self.get_token(session)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.get_access(session).auth_token

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     self.auth_ref = self.get_auth_ref(session)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self._plugin.get_auth_ref(session, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 165, in get_auth_ref

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     authenticated=False, log=False, **rkwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.request(url, 'POST', **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 66, in map_exceptions_wrapper

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     raise exceptions.from_exception(e)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari HttpException: HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-3640dd4f-b9d9-4a10-ae37-43a0e202393f), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari 

stack@devstack-masakari:~/devstack

 

 

Greg.

 

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:52 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Went back and started up corosync and pacemaker and now masakari-instancemonitor starts successfully.

 

stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &

[1] 25973

stack@devstack-masakari:~/masakari-monitors$ 2017-12-11 12:47:49.483 25974 INFO masakarimonitors.service [-] Starting masakarimonitors-instancemonitor

 

stack@devstack-masakari:~/masakari-monitors

 

I don’t see any log file in /var/log for masakari-instancemonitor or masakari-engine ?

 

All processes seem to be up and running fine:

stack@devstack-masakari:~/masakari-monitors$ ps -ef | fgrep masakari

stack    11625     1  0 12:29 ?        00:00:10 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

stack    11778     1  0 12:29 ?        00:00:02 /usr/bin/python /usr/local/bin/masakari-engine --config-file=/etc/masakari/masakari.conf --debug

stack    12005 11625  0 12:29 ?        00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

stack    12006 11625  0 12:29 ?        00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

root     19336     1  0 12:19 ?        00:00:05 /opt/stack/bin/etcd --name devstack-masakari --data-dir /opt/stack/data/etcd --initial-cluster-state new --initial-cluster-token etcd-cluster-01 --initial-cluster devstack-masakari=http://10.10.10.7:2380 --initial-advertise-peer-urls http://10.10.10.7:2380 --advertise-client-urls http://10.10.10.7:2379 --listen-peer-urls http://0.0.0.0:2380 --listen-client-urls http://10.10.10.7:2379

root     25973 25760  0 12:47 pts/0    00:00:00 sudo masakari-instancemonitor

root     25974 25973  0 12:47 pts/0    00:00:00 /usr/bin/python /usr/local/bin/masakari-instancemonitor

stack    26123 25760  0 12:50 pts/0    00:00:00 grep -F --color=auto masakari

rabbitmq 26477 26348  0 12:06 ?        00:00:15 /usr/lib/erlang/erts-7.3/bin/beam.smp -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@devstack-masakari -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@devstack-masakari.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@devstack-masakari-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@devstack-masakari-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@devstack-masakari" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672

stack@devstack-masakari:~/masakari-monitors

 

 

Greg.

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:38 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Actually, i’m guessing this is wrong because masakari-instancemonitor fails to start when i use these instructions.

 

stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &

[1] 22959

stack@devstack-masakari:~/masakari-monitors$ Traceback (most recent call last):

  File "/usr/local/bin/masakari-instancemonitor", line 10, in <module>

    sys.exit(main())

  File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/cmd/instancemonitor.py", line 31, in main

    config.parse_args(sys.argv)

  File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/config.py", line 32, in parse_args

    default_config_files=default_config_files)

  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2495, in __call__

    self._check_required_opts()

  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 3134, in _check_required_opts

    raise RequiredOptError(opt.name, group)

oslo_config.cfg.RequiredOptError: value required for option auth-url in group [api]

 

[1]+  Exit 1                  sudo masakari-instancemonitor

 

Greg.

 

 

 

From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:17 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

thanks Honjo,

 

I’ll try this out now.

 

I’m assuming that it is ok to start ONLY the instance-monitor if that’s all i want to test.

Is that correct ?

 

In that case, following the instructions you pointed me at,

I would do the following:

·         setup devstack

o   i.e. with ‘enable_plugin masakari git://git.openstack.org/openstack/masakari’

·          

·         ( don’t need corosync and pacemaker )

·          

·         install & startup client

o   cd

o   git clone https://github.com/openstack/python-masakariclient.git

o   cd python-masakariclient

o   sudo python setup.py build

o   sudo python setup.py install

·          

·         install & startup instance monitor

o   cd

o   git clone https://github.com/openstack/masakari-monitors.git

o   sudo mkdir /etc/masakarimonitors

o   cd masakari-monitors

o   sudo python setup.py build

o   sudo python setup.py install

o    

o   sudo masakari-instancemonitor &

 

Is this correct ?

 

Greg.

 

 

 

 

From: Rikimaru Honjo <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, December 7, 2017 at 12:24 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

 

Hello Greg,

 

I forgot to tell you.

 

Please use process_list.yaml instead of proc.list.sample.

 

On 2017/12/07 14:03, Rikimaru Honjo wrote:

Hello Greg,

Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf.

You can generate it by "tox -egenconfig".

hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script.

masakarimonitors.conf is a configuration file for monitors implemented by python that you installed.

And, we are preparing setting guides.

Please see it if you are good.

masakari:

masakari-monitors:

Best regards,

On 2017/12/06 22:48, Waines, Greg wrote:

I am just getting started working with masakari.

I am working on master.

 

I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes.

 

I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors .

Specifically:

# install & startup monitors

cd

cd masakari-monitors

sudo python setup.py install

cd

sudo mkdir /etc/masakarimonitors

sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list

cd ~/masakari-monitors/masakarimonitors/cmd

sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list &

sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf &

sudo /usr/bin/python ./instancemonitor.py &

 

However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file.

 

Is this the correct way to startup masakari instance monitoring ?

 

Greg.

 

 

 

 

My Masakari setup in Devstack

 

sudo useradd -s /bin/bash -d /opt/stack -m stack

echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack

sudo su - stack

 

cd devstack

 

                                                 local.conf file:

[[local|localrc]]

ADMIN_PASSWORD=admin

DATABASE_PASSWORD=admin

RABBIT_PASSWORD=admin

SERVICE_PASSWORD=admin

# setup Neutron services

disable_service n-net

enable_service q-svc

enable_service q-agt

enable_service q-dhcp

enable_service q-l3

enable_service q-meta

# ceilometer

# heat

# vitrage

# masakari

enable_plugin masakari git://git.openstack.org/openstack/masakari

 

 

./stack.sh

 

 

 

 

 

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]?subject:unsubscribe

 

 

--

_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

Rikimaru Honjo

 

 

__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]?subject:unsubscribe

 


______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Reply | Threaded
Open this post in threaded view
|

Re: [masakari] problems starting up masakari instance monitoring in devstack @ master

Kwan, Louie
Hi Dinesh,

Thanks for the info.

'recovery_method' (choose from 'auto', 'reserved_host', 'auto_priority', 'rh_priority')

What should be put for <service_type>

Same as the segment host

What should be put as <type> <control_attributes> <segment-id>, it seems it accepts any arbitrary values.

Any more pointers. Much appreciated.

Thank you.
Louie Kwan
________________________________________
From: Bhor, Dinesh [[hidden email]]
Sent: Wednesday, December 13, 2017 11:22 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master

Hi Greg,


Looks like you don’t have “devstack-masakari” host registered in Masakari database.


Currently operator have to add all the hosts from failover-segment manually to Masakari database.


You can use below command:


Register failover-segment to Masakari database:

openstack segment create <name> <recovery_method> <service_type>


Register hosts under the created segment to Masakari database:

openstack segment host create <name> <type> <control_attributes> <segment-id>



There is a work in progress on auto compute node registration.

Please refer: https://blueprints.launchpad.net/masakari-monitors/+spec/auto-compute-node-registration



Thank you,

Dinesh Bhor

________________________________
From: Waines, Greg <[hidden email]>
Sent: 13 December 2017 19:26:32
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master


ok ... i changed all the domain related attributes in /etc/masakarimonitors/masakarimonitors.conf to be set to ‘default’ .

I seemed to get further,

but now get the following error when instancemonitor tries to send a notification:

      Bad Request (HTTP 400) (Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari could not be found.

however:

- devstack-masakari is in the /etc/hosts

- nova hypervisor-list shows devstack-masakari as a hypervisor hostname

- i am running both hostmonitor, processmonitor and instancemonitor



any ideas ?

see details below,

Greg.



2017-12-13 13:50:34.353 7637 INFO masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event: type=VM, hostname=devstack-masakari, uuid=6ae0b09b-3e93-4f0c-b81b-fa140636f267, time=2017-12-13 13:50:34.353037, event_id=LIFECYCLE, detail=STOPPED_DESTROYED)

2017-12-13 13:50:34.353 7637 INFO masakarimonitors.ha.masakari [-] Send a notification. {'notification': {'hostname': 'devstack-masakari', 'type': 'VM', 'payload': {'instance_uuid': '6ae0b09b-3e93-4f0c-b81b-fa140636f267', 'vir_domain_event': 'STOPPED_DESTROYED', 'event': 'LIFECYCLE'}, 'generated_time': datetime.datetime(2017, 12, 13, 13, 50, 34, 353037)}}

2017-12-13 13:50:34.461 7637 WARNING masakarimonitors.ha.masakari [-] Retry sending a notification. (HttpException: Bad Request (HTTP 400) (Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari could not be found.): HttpException: HttpException: Bad Request (HTTP 400) (Request-ID: req-556c6c4d-0e12-414e-ac8c-8ef3a61d4864), Host with name devstack-masakari could not be found.

^C2017-12-13 13:50:42.462 7637 INFO oslo_service.service [-] Caught SIGINT signal, instantaneous exiting

root@devstack-masakari:~#

root@devstack-masakari:~# ping devstack-masakari

PING localhost (127.0.0.1) 56(84) bytes of data.

64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.049 ms

64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.022 ms

^C

--- localhost ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 999ms

rtt min/avg/max/mdev = 0.022/0.035/0.049/0.014 ms

root@devstack-masakari:~#



stack@devstack-masakari:~$ nova hypervisor-list

+--------------------------------------+---------------------+-------+---------+

| ID                                   | Hypervisor hostname | State | Status  |

+--------------------------------------+---------------------+-------+---------+

| 5fb1b09b-e5f5-465a-828a-2101135ff700 | devstack-masakari   | up    | enabled |

+--------------------------------------+---------------------+-------+---------+

stack@devstack-masakari:~$













From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Wednesday, December 13, 2017 at 8:17 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master



So i believe the error is:

HttpException: HttpException: Expecting to find domain in project.



I have attached the /etc/masakarimonitors/masakarimonitors.conf file and the /etc/masakari/masakari.conf file .



See the domain related attributes in each file below.



Is the Default vs default causing this problem ?

Are there other domain related attributes that should be set to default ?



stack@devstack-masakari:~$ fgrep -i domain /etc/masakari/masakari.conf

project_domain_name = Default

user_domain_name = Default



stack@devstack-masakari:~$ fgrep -i domain /etc/masakarimonitors/masakarimonitors.conf

#logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s

# Domain ID to scope to (string value)

#domain_id = <None>

# Domain name to scope to (string value)

#domain_name = <None>

# Domain ID containing project (string value)

#project_domain_id = <None>

# Domain name containing project (string value)

#project_domain_name = <None>

project_domain_name = default

# Optional domain ID to use with v3 and v2 parameters. It will be used for both

# the user and project domain in v3 and ignored in v2 authentication. (string

#default_domain_id = <None>

# Optional domain name to use with v3 API and v2 parameters. It will be used for

# both the user and project domain in v3 and ignored in v2 authentication.

#default_domain_name = <None>

# User's domain id (string value)

#user_domain_id = <None>

# User's domain name (string value)

#user_domain_name = <None>

# Indicate whether this resource may be shared with the domain received in the

stack@devstack-masakari:~$





Greg.





From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Tuesday, December 12, 2017 at 7:19 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master



Any thoughts on what i don’t have setup correctly wrt the masakari client from the errors below ?

Greg.



From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 8:05 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master



I started up a VM and then deleted the VM ... and got some errors that make me think i don’t have the masakari client setup correctly.

Any ideas ?

Greg.



i.e.

stack@devstack-masakari:~/devstack$ nova list





+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+

| ID                                   | Name        | Status | Task State | Power State | Networks                                                |

+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+

| de8922c6-450d-4e2e-954d-ee4bd05ab909 | vm-1-cirros | ACTIVE | -          | Running     | private=10.0.0.10, fd1a:8f71:1a96:0:f816:3eff:fe5e:f315 |

+--------------------------------------+-------------+--------+------------+-------------+---------------------------------------------------------+

stack@devstack-masakari:~/devstack$

stack@devstack-masakari:~/devstack$ nova delete vm-1-cirros

Request to delete server vm-1-cirros has been accepted.

stack@devstack-masakari:~/devstack$ 2017-12-11 12:58:28.319 25974 INFO masakarimonitors.instancemonitor.libvirt_handler.callback [-] Libvirt Event: type=VM, hostname=devstack-masakari, uuid=de8922c6-450d-4e2e-954d-ee4bd05ab909, time=2017-12-11 12:58:28.318781, event_id=LIFECYCLE, detail=STOPPED_DESTROYED)

2017-12-11 12:58:28.319 25974 INFO masakarimonitors.ha.masakari [-] Send a notification. {'notification': {'hostname': 'devstack-masakari', 'type': 'VM', 'payload': {'instance_uuid': 'de8922c6-450d-4e2e-954d-ee4bd05ab909', 'vir_domain_event': 'STOPPED_DESTROYED', 'event': 'LIFECYCLE'}, 'generated_time': datetime.datetime(2017, 12, 11, 12, 58, 28, 318781)}}

2017-12-11 12:58:28.351 25974 WARNING masakarimonitors.ha.masakari [-] Retry sending a notification. (HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.): HttpException: HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-feaa30f9-fbb2-4259-8851-488ff7ab82f9), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.

... several times ... and then eventually

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari Traceback (most recent call last):

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/ha/masakari.py", line 91, in send_notification

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     payload=event['notification']['payload'])

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/masakariclient/sdk/ha/v1/_proxy.py", line 65, in create_notification

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self._create(_notification.Notification, **attrs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/proxy2.py", line 194, in _create

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return res.create(self._session)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/resource2.py", line 588, in create

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     json=request.body, headers=request.headers)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.request(url, 'POST', **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 64, in map_exceptions_wrapper

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return func(*args, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 352, in request

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return super(Session, self).request(*args, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 573, in request

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     auth_headers = self.get_auth_headers(auth)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 900, in get_auth_headers

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return auth.get_headers(self, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     token = self.get_token(session)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.get_access(session).auth_token

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     self.auth_ref = self.get_auth_ref(session)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self._plugin.get_auth_ref(session, **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/v3/base.py", line 165, in get_auth_ref

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     authenticated=False, log=False, **rkwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 848, in post

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     return self.request(url, 'POST', **kwargs)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari   File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 66, in map_exceptions_wrapper

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari     raise exceptions.from_exception(e)

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari HttpException: HttpException: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-3640dd4f-b9d9-4a10-ae37-43a0e202393f), Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.

2017-12-11 13:00:28.536 25974 ERROR masakarimonitors.ha.masakari

stack@devstack-masakari:~/devstack$





Greg.







From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:52 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master



Went back and started up corosync and pacemaker and now masakari-instancemonitor starts successfully.



stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &

[1] 25973

stack@devstack-masakari:~/masakari-monitors$ 2017-12-11 12:47:49.483 25974 INFO masakarimonitors.service [-] Starting masakarimonitors-instancemonitor



stack@devstack-masakari:~/masakari-monitors$



I don’t see any log file in /var/log for masakari-instancemonitor or masakari-engine ?



All processes seem to be up and running fine:

stack@devstack-masakari:~/masakari-monitors$ ps -ef | fgrep masakari

stack    11625     1  0 12:29 ?        00:00:10 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

stack    11778     1  0 12:29 ?        00:00:02 /usr/bin/python /usr/local/bin/masakari-engine --config-file=/etc/masakari/masakari.conf --debug

stack    12005 11625  0 12:29 ?        00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

stack    12006 11625  0 12:29 ?        00:00:00 /usr/bin/python /usr/local/bin/masakari-api --config-file=/etc/masakari/masakari.conf --debug

root     19336     1  0 12:19 ?        00:00:05 /opt/stack/bin/etcd --name devstack-masakari --data-dir /opt/stack/data/etcd --initial-cluster-state new --initial-cluster-token etcd-cluster-01 --initial-cluster devstack-masakari=http://10.10.10.7:2380 --initial-advertise-peer-urls http://10.10.10.7:2380 --advertise-client-urls http://10.10.10.7:2379 --listen-peer-urls http://0.0.0.0:2380 --listen-client-urls http://10.10.10.7:2379

root     25973 25760  0 12:47 pts/0    00:00:00 sudo masakari-instancemonitor

root     25974 25973  0 12:47 pts/0    00:00:00 /usr/bin/python /usr/local/bin/masakari-instancemonitor

stack    26123 25760  0 12:50 pts/0    00:00:00 grep -F --color=auto masakari

rabbitmq 26477 26348  0 12:06 ?        00:00:15 /usr/lib/erlang/erts-7.3/bin/beam.smp -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@devstack-masakari -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/[hidden email]"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/[hidden email]"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@devstack-masakari-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@devstack-masakari" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672

stack@devstack-masakari:~/masakari-monitors$





Greg.





From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:38 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master



Actually, i’m guessing this is wrong because masakari-instancemonitor fails to start when i use these instructions.



stack@devstack-masakari:~/masakari-monitors$ sudo masakari-instancemonitor &

[1] 22959

stack@devstack-masakari:~/masakari-monitors$ Traceback (most recent call last):

  File "/usr/local/bin/masakari-instancemonitor", line 10, in <module>

    sys.exit(main())

  File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/cmd/instancemonitor.py", line 31, in main

    config.parse_args(sys.argv)

  File "/usr/local/lib/python2.7/dist-packages/masakarimonitors/config.py", line 32, in parse_args

    default_config_files=default_config_files)

  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2495, in __call__

    self._check_required_opts()

  File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 3134, in _check_required_opts

    raise RequiredOptError(opt.name, group)

oslo_config.cfg.RequiredOptError: value required for option auth-url in group [api]



[1]+  Exit 1                  sudo masakari-instancemonitor



Greg.







From: Greg Waines <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Monday, December 11, 2017 at 7:17 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master



thanks Honjo,



I’ll try this out now.



I’m assuming that it is ok to start ONLY the instance-monitor if that’s all i want to test.

Is that correct ?



In that case, following the instructions you pointed me at,

I would do the following:

·         setup devstack

o   i.e. with ‘enable_plugin masakari git://git.openstack.org/openstack/masakari’

·

·         ( don’t need corosync and pacemaker )

·

·         install & startup client

o   cd

o   git clone https://github.com/openstack/python-masakariclient.git

o   cd python-masakariclient

o   sudo python setup.py build

o   sudo python setup.py install

·

·         install & startup instance monitor

o   cd

o   git clone https://github.com/openstack/masakari-monitors.git

o   sudo mkdir /etc/masakarimonitors

o   cd masakari-monitors

o   sudo python setup.py build

o   sudo python setup.py install

o

o   sudo masakari-instancemonitor &



Is this correct ?



Greg.









From: Rikimaru Honjo <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, December 7, 2017 at 12:24 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: [openstack-dev] [masakari] problems starting up masakari instance monitoring in devstack @ master



Hello Greg,



I forgot to tell you.



Please use process_list.yaml instead of proc.list.sample.



On 2017/12/07 14:03, Rikimaru Honjo wrote:

Hello Greg,

Please use masakarimonitors.conf instead of hostmonitor.conf and processmonitor.conf.

You can generate it by "tox -egenconfig".

hostmonitor.conf and processmonitor.conf are used for monitors implemented by shell script.

masakarimonitors.conf is a configuration file for monitors implemented by python that you installed.

And, we are preparing setting guides.

Please see it if you are good.

masakari:

https://review.openstack.org/#/c/489570/

masakari-monitors:

https://review.openstack.org/#/c/489095/

Best regards,

On 2017/12/06 22:48, Waines, Greg wrote:

I am just getting started working with masakari.

I am working on master.



I have setup Masakari in Devstack (see details at end of email) ... which starts up masakari-engine and masakari-api processes.



I have git cloned the masakari-monitors and started them up (roughly) following the instructions at https://github.com/openstack/masakari-monitors .

Specifically:

# install & startup monitors

cd

git clone https://github.com/openstack/masakari-monitors.git

cd masakari-monitors

sudo python setup.py install

cd

sudo mkdir /etc/masakarimonitors

sudo cp ~/masakari-monitors/etc/masakarimonitors/hostmonitor.conf.sample /etc/masakarimonitors/hostmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/processmonitor.conf.sample /etc/masakarimonitors/processmonitor.conf

sudo cp ~/masakari-monitors/etc/masakarimonitors/proc.list.sample /etc/masakarimonitors/proc.list

cd ~/masakari-monitors/masakarimonitors/cmd

sudo masakari-processmonitor.sh /etc/masakarimonitors/processmonitor.conf /etc/masakarimonitors/proc.list &

sudo masakari-hostmonitor.sh /etc/masakarimonitors/hostmonitor.conf &

sudo /usr/bin/python ./instancemonitor.py &



However the instancemonitor.py starts and exits ... and does not appear to start any process(es) ... with no error messages and no log file.



Is this the correct way to startup masakari instance monitoring ?



Greg.









My Masakari setup in Devstack



sudo useradd -s /bin/bash -d /opt/stack -m stack

echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack

sudo su - stack



git clone https://github.com/openstack-dev/devstack

cd devstack



                                                 local.conf file:

[[local|localrc]]

ADMIN_PASSWORD=admin

DATABASE_PASSWORD=admin

RABBIT_PASSWORD=admin

SERVICE_PASSWORD=admin

# setup Neutron services

disable_service n-net

enable_service q-svc

enable_service q-agt

enable_service q-dhcp

enable_service q-l3

enable_service q-meta

# ceilometer

enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer

enable_plugin aodh https://git.openstack.org/openstack/aodh

# heat

enable_plugin heat https://git.openstack.org/openstack/heat

# vitrage

enable_plugin vitrage https://git.openstack.org/openstack/vitrage

enable_plugin vitrage-dashboard https://git.openstack.org/openstack/vitrage-dashboard

# masakari

enable_plugin masakari git://git.openstack.org/openstack/masakari





./stack.sh











__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]<mailto:[hidden email]>?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





--

_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

Rikimaru Honjo

E-mail:[hidden email]<mailto:[hidden email]>





__________________________________________________________________________

OpenStack Development Mailing List (not for usage questions)

Unsubscribe: [hidden email]<mailto:[hidden email]>?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [hidden email]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev