Running OpenStack Newton, fresh install, am getting this on the compute node (in nova-compute.log) when trying to launch an instance:
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [req-af37e2ee-0ef9-4d4e-b3ce-d7a1bf27a780 - - - - -] [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] An error occurred while refreshing the network cache.
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] Traceback (most recent call last):
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5766, in _heal_instance_info_cache
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] self.network_api.get_instance_nw_info(context, instance)
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 369, in get_instance_nw_info
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] **kwargs)
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] File "/usr/lib/python2.7/dist-packages/nova/network/base_api.py", line 249, in get_instance_nw_info
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] result = self._get_instance_nw_info(context, instance, **kwargs)
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 378, in _get_instance_nw_info
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] nw_info = self.network_rpcapi.get_instance_nw_info(context, **args)
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 211, in get_instance_nw_info
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] host=host, project_id=project_id)
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] retry=self.retry)
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in _send
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] timeout=timeout, retry=retry)
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] retry=retry)
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 453, in _send
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] result = self._waiter.wait(msg_id, timeout)
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in wait
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] message = self.waiters.get(msg_id, timeout=timeout)
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 238, in get
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] 'to message ID %s' % msg_id)
2017-04-04 19:28:47.546 31726 ERROR nova.compute.manager [instance: 6ecaf72c-88bc-4f26-8907-dc19d7924327] MessagingTimeout: Timed out waiting for a reply to message ID bb7d1a5d89c8469aa1243f9102656d3
This only happens for messages in exchange 'nova' topic 'network':
Apr 4 19:26:47 ip-192-168-99-11 nova-compute[31726]: 2017-04-04 19:26:47.544 31726 DEBUG oslo_messaging._drivers.amqpdriver [req-af37e2ee-0ef9-4d4e-b3ce-d7a1bf27a780 - - - - -] CALL msg_id: bb7d1a5d89c8469aa1243f9102656d3f exchange 'nova' topic 'network' _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
(e.g., messages on topic 'conductor' go through fine).
I notice that in RabbitMQ there is a conductor queue (with conductor routing key) but no network queue (which corresponds to https://ilearnstack.com/2013/04/24/messaging-in-openstack-using-rabbitmq/)
The connectivity between the compute node and controller (where Rabbit runs) is fine.
Tried turning on the Rabbit tracing (http://www.rabbitmq.com/firehose.html), and I see all messages BUT the ones in question.
Any pointers?