Nova hypervisor uuid

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Nova hypervisor uuid

Ignazio Cassano
Hi  All,
Please anyone know where hypervisor uuid is retrived?
Sometime updating kmv nodes with yum update it changes and in nova database 2 uuids are assigned to the same node.
regards
Ignazio



_______________________________________________
OpenStack-operators mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Reply | Threaded
Open this post in threaded view
|

Re: Nova hypervisor uuid

Matt Riedemann-3
On 11/27/2018 11:32 AM, Ignazio Cassano wrote:

> Hi  All,
> Please anyone know where hypervisor uuid is retrived?
> Sometime updating kmv nodes with yum update it changes and in nova
> database 2 uuids are assigned to the same node.
> regards
> Ignazio
>
>
>
>
> _______________________________________________
> OpenStack-operators mailing list
> [hidden email]
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

To be clear, do you mean the computes_nodes.uuid column value in the
cell database? Which is also used for the GET /os-hypervisors response
'id' value if using microversion >= 2.53. If so, that is generated
randomly* when the compute_nodes table record is created:

https://github.com/openstack/nova/blob/8545ba2af7476e0884b5e7fb90965bef92d605bc/nova/compute/resource_tracker.py#L588

https://github.com/openstack/nova/blob/8545ba2af7476e0884b5e7fb90965bef92d605bc/nova/objects/compute_node.py#L312

When you hit this problem, are you sure the hostname on the compute host
is not changing? Because when nova-compute starts up, it should look for
the existing compute node record by host name and node name, which for
the libvirt driver should be the same. That lookup code is here:

https://github.com/openstack/nova/blob/8545ba2af7476e0884b5e7fb90965bef92d605bc/nova/compute/resource_tracker.py#L815

So the only way nova-compute should create a new compute_nodes table
record for the same host is if the host/node name changes during the
upgrade. Is the deleted value in the database the same (0) for both of
those records?

* The exception to this is for the ironic driver which re-uses the
ironic node uuid as of this change: https://review.openstack.org/#/c/571535/

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Nova hypervisor uuid

Ignazio Cassano
Hi Matt, sorry but I lost your answer and Gianpiero forwarded it to me.
I am sure kvm nodes names are note changed.
Tables where uuid are duplicated are:
dataresource_providers in nova_api db
compute_nodes in nova db
Regards
Ignazio

Il 28/Nov/2018 11:09 AM, "Gianpiero Ardissono" <[hidden email]> ha scritto:

---------- Forwarded message ---------
From: Matt Riedemann <[hidden email]>
Date: mar 27 nov 2018, 19:03
Subject: Re: [Openstack-operators] Nova hypervisor uuid
To: <[hidden email]>


On 11/27/2018 11:32 AM, Ignazio Cassano wrote:
> Hi  All,
> Please anyone know where hypervisor uuid is retrived?
> Sometime updating kmv nodes with yum update it changes and in nova
> database 2 uuids are assigned to the same node.
> regards
> Ignazio
>
>
>
>
> _______________________________________________
> OpenStack-operators mailing list
> [hidden email]
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

To be clear, do you mean the computes_nodes.uuid column value in the
cell database? Which is also used for the GET /os-hypervisors response
'id' value if using microversion >= 2.53. If so, that is generated
randomly* when the compute_nodes table record is created:

https://github.com/openstack/nova/blob/8545ba2af7476e0884b5e7fb90965bef92d605bc/nova/compute/resource_tracker.py#L588

https://github.com/openstack/nova/blob/8545ba2af7476e0884b5e7fb90965bef92d605bc/nova/objects/compute_node.py#L312

When you hit this problem, are you sure the hostname on the compute host
is not changing? Because when nova-compute starts up, it should look for
the existing compute node record by host name and node name, which for
the libvirt driver should be the same. That lookup code is here:

https://github.com/openstack/nova/blob/8545ba2af7476e0884b5e7fb90965bef92d605bc/nova/compute/resource_tracker.py#L815

So the only way nova-compute should create a new compute_nodes table
record for the same host is if the host/node name changes during the
upgrade. Is the deleted value in the database the same (0) for both of
those records?

* The exception to this is for the ironic driver which re-uses the
ironic node uuid as of this change: https://review.openstack.org/#/c/571535/

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

_______________________________________________
OpenStack-operators mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Nova hypervisor uuid

Matt Riedemann-3
On 11/28/2018 4:19 AM, Ignazio Cassano wrote:
> Hi Matt, sorry but I lost your answer and Gianpiero forwarded it to me.
> I am sure kvm nodes names are note changed.
> Tables where uuid are duplicated are:
> dataresource_providers in nova_api db
> compute_nodes in nova db
> Regards
> Ignazio

It would be easier if you simply dumped the result of a select query on
the compute_nodes table where the duplicate nodes exist (you said
duplicate UUIDs but I think you mean duplicate host/node names with
different UUIDs, correct?).

There is a unique constraint on host/hypervisor_hostname (nodename)/deleted:

         schema.UniqueConstraint(
             'host', 'hypervisor_hostname', 'deleted',
             name="uniq_compute_nodes0host0hypervisor_hostname0deleted"),

So I'm wondering if the deleted field is not 0 on one of those because
if one is marked as deleted, then the compute service will create a new
compute_nodes table record on startup (and associated resource provider).

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Nova hypervisor uuid

Ignazio Cassano
Hello Mattm
Yes I mean sometimes I have same host/node names with different uuid in compute_nodes table in nova database
I must delete nodes with uuid those not match with nova-hypervisor list command.
At this time I have the following:
MariaDB [nova]> select hypervisor_hostname,uuid,deleted from compute_nodes;
+---------------------+--------------------------------------+---------+
| hypervisor_hostname | uuid                                 | deleted |
+---------------------+--------------------------------------+---------+
| tst2-kvm02          | 802b21c2-11fb-4426-86b9-bf25c8a5ae1d |       0 |
| tst2-kvm01          | ce27803b-06cd-44a7-b927-1fa42c813b0f |       0 |
+---------------------+--------------------------------------+---------+
2 rows in set (0,00 sec)


But sometimes old uuid are inserted in the table .
I deleted again them.
I restarted kvm nodes and now the table is ok.
I also restarded each controller and the tables is ok.
I do not know because 3 days ago I had same compute nodes names with different uuids.

Thanks and Regards
Ignazio

Il giorno mer 28 nov 2018 alle ore 17:54 Matt Riedemann <[hidden email]> ha scritto:
On 11/28/2018 4:19 AM, Ignazio Cassano wrote:
> Hi Matt, sorry but I lost your answer and Gianpiero forwarded it to me.
> I am sure kvm nodes names are note changed.
> Tables where uuid are duplicated are:
> dataresource_providers in nova_api db
> compute_nodes in nova db
> Regards
> Ignazio

It would be easier if you simply dumped the result of a select query on
the compute_nodes table where the duplicate nodes exist (you said
duplicate UUIDs but I think you mean duplicate host/node names with
different UUIDs, correct?).

There is a unique constraint on host/hypervisor_hostname (nodename)/deleted:

         schema.UniqueConstraint(
             'host', 'hypervisor_hostname', 'deleted',
             name="uniq_compute_nodes0host0hypervisor_hostname0deleted"),

So I'm wondering if the deleted field is not 0 on one of those because
if one is marked as deleted, then the compute service will create a new
compute_nodes table record on startup (and associated resource provider).

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Nova hypervisor uuid

Matt Riedemann-3
On 11/29/2018 12:49 AM, Ignazio Cassano wrote:

> Hello Mattm
> Yes I mean sometimes I have same host/node names with different uuid in
> compute_nodes table in nova database
> I must delete nodes with uuid those not match with nova-hypervisor list
> command.
> At this time I have the following:
> MariaDB [nova]> select hypervisor_hostname,uuid,deleted from compute_nodes;
> +---------------------+--------------------------------------+---------+
> | hypervisor_hostname | uuid                                 | deleted |
> +---------------------+--------------------------------------+---------+
> | tst2-kvm02          | 802b21c2-11fb-4426-86b9-bf25c8a5ae1d |       0 |
> | tst2-kvm01          | ce27803b-06cd-44a7-b927-1fa42c813b0f |       0 |
> +---------------------+--------------------------------------+---------+
> 2 rows in set (0,00 sec)
>
>
> But sometimes old uuid are inserted in the table .
> I deleted again them.
> I restarted kvm nodes and now the table is ok.
> I also restarded each controller and the tables is ok.
> I do not know because 3 days ago I had same compute nodes names with
> different uuids.
>
> Thanks and Regards
> Ignazio

OK I guess if it happens again, please get the
host/hypervisor_hostname/uuid/deleted values from the compute_nodes
table before you cleanup any entries.

Also, when you're deleting the resources from the DB, are you doing it
in the DB directly or via the DELETE /os-services/{service_id} API?
Because the latter cleans up other related resources to the nova-compute
service (the services table record, the compute_nodes table record, the
related resource_providers table record in placement, and the
host_mappings table record in the nova API DB). The resource
provider/host mappings cleanup when deleting a compute service is a more
recent bug fix though which depending on your release you might not have:

https://review.openstack.org/#/q/I7b8622b178d5043ed1556d7bdceaf60f47e5ac80

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Nova hypervisor uuid

Ignazio Cassano
Hi Matt,
I did in the DB directly.
I am using queens now.
Any python client command to delete hold records or I must use api ?

Thanks & Regards
Ignazio


Il giorno gio 29 nov 2018 alle ore 16:28 Matt Riedemann <[hidden email]> ha scritto:
On 11/29/2018 12:49 AM, Ignazio Cassano wrote:
> Hello Mattm
> Yes I mean sometimes I have same host/node names with different uuid in
> compute_nodes table in nova database
> I must delete nodes with uuid those not match with nova-hypervisor list
> command.
> At this time I have the following:
> MariaDB [nova]> select hypervisor_hostname,uuid,deleted from compute_nodes;
> +---------------------+--------------------------------------+---------+
> | hypervisor_hostname | uuid                                 | deleted |
> +---------------------+--------------------------------------+---------+
> | tst2-kvm02          | 802b21c2-11fb-4426-86b9-bf25c8a5ae1d |       0 |
> | tst2-kvm01          | ce27803b-06cd-44a7-b927-1fa42c813b0f |       0 |
> +---------------------+--------------------------------------+---------+
> 2 rows in set (0,00 sec)
>
>
> But sometimes old uuid are inserted in the table .
> I deleted again them.
> I restarted kvm nodes and now the table is ok.
> I also restarded each controller and the tables is ok.
> I do not know because 3 days ago I had same compute nodes names with
> different uuids.
>
> Thanks and Regards
> Ignazio

OK I guess if it happens again, please get the
host/hypervisor_hostname/uuid/deleted values from the compute_nodes
table before you cleanup any entries.

Also, when you're deleting the resources from the DB, are you doing it
in the DB directly or via the DELETE /os-services/{service_id} API?
Because the latter cleans up other related resources to the nova-compute
service (the services table record, the compute_nodes table record, the
related resource_providers table record in placement, and the
host_mappings table record in the nova API DB). The resource
provider/host mappings cleanup when deleting a compute service is a more
recent bug fix though which depending on your release you might not have:

https://review.openstack.org/#/q/I7b8622b178d5043ed1556d7bdceaf60f47e5ac80

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Nova hypervisor uuid

Matt Riedemann-3
On 11/29/2018 10:27 AM, Ignazio Cassano wrote:
> I did in the DB directly.
> I am using queens now.
> Any python client command to delete hold records or I must use api ?

You can use the CLI:

https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-service-delete

https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/compute-service.html#compute-service-delete

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Nova hypervisor uuid

Ignazio Cassano
Many thks Matt.
If the issue will happen again 
I hope openstack commands will show duplicate entries and let me clean them.
When happened nova-hypervisor list command did not show duplcated entries but I saw them only in the database.
Regards
Ignazio 

Il giorno Gio 29 Nov 2018 17:28 Matt Riedemann <[hidden email]> ha scritto:
On 11/29/2018 10:27 AM, Ignazio Cassano wrote:
> I did in the DB directly.
> I am using queens now.
> Any python client command to delete hold records or I must use api ?

You can use the CLI:

https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-service-delete

https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/compute-service.html#compute-service-delete

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
[hidden email]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators