Monday, May 19, 2014

OpenStack - OpenVSwitch Agent won't register

I hit the wall today when I tried to add an another compute node to my OpenStack. Everything seemed to install fine. However, the Openvswitch Agent did not register itself. Of course, I am using the Oracle Enterprise Linux which is not official supported yet by OpenStack and it made me uneasy. During the installation process, I hit a few problems of loading the wrong yum packages and I correct them manaully.

First I checked out neutron-openvswitch-agent.log on the compute node. There is no trace and error. I checked out the neutron-server.log, no trace and error neither.  I understand that the registration mechanism is by sending a report_status message from the agent to neutron controller.

I used rabbitmqctl to check for the existence of the queue and it did. It really buffled me. So I compared the message in the neutron-server.log.  With the surprise, the clock on the new compute node was not set correctly and the '_context_timestamp' is off in the debug messager.

2014-05-19 15:13:33.923 21639 DEBUG neutron.db.agents_db [req-915d3718-8489-4bd1-905f-3a02959b2746 None] Message with invalid timestamp received report_state /opt/stack/neutron/neutron/db/agents_db.py:214

So the time is the problem.  I adjusted the clock. Magically, the agent is registered. I hope there is much more clear message to idenify the problem.

Friday, May 9, 2014

Adding your own configuration to OpenStack Neutron using oslo.config

If you are doing OpenStack development,  eventually you will want to add your own configuration in OpenStack for you own plugin/mechanism driver. The steps required is pretty simple.

Let say my plugin/mechanism driver need to talk to a web service running to perform some operation and I do not want to hard code the web service configuration.

OpenStack uses oslo.config to do the configuration parsing. 

First, you need to tell oslo.config what configuration options are needed.

Here is the sample code fragment to create my configuration parameter list. 

ml2_my_opts = [
    cfg.StrOpt('hostname', default=None,
               help=_("The hostname of the my service")),
    cfg.IntOpt('port', default=8443,
               help=_("The port number of my service")),
    cfg.StrOpt('username', default=None,
               help=_("The username of my service")),
    cfg.StrOpt('password', default=None,
               help=_("The password of my service"))]

I specify the hostname, port, username and password. I set the port default value as 8443.

Then I register it by doing the following.

cfg.CONF.register_opts(my_service_opts, "my_service")

"my_service" is optional. If specified, it is the namespace I am going to use. 

To retrieve the configuration,

hostname = cfg.CONF.my_service_opts.hostname
port = cfg.CONF.my_service_opts.port
username = cfg.CONF.my_service_opts.username
password = cfg.CONF.my_service_opts.password

Now the coding is done. 

Let put the parameters on the configuration file. The configuration file is specified by the --config-file in the argument of python. For example, 

stack    17357  0.1  0.0 284908 52156 pts/9    S+   08:57   0:05 python /usr/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini

For the neutron-server, two configuration files are specified. One is /etc/neutron/neutron.conf and the other is /etc/neutron/plugins/ml2/ml2_conf.ini.

For the above example, I will put the following configuration in one of the configuration file listed above.

[my_service]
hostname=my_host
port=8443
username=guest
password=guest

With oslo.config, adding your own additional configuration is extremely simple.

Wednesday, April 9, 2014

Listen to OpenStack Neutron Messages from RabbitMQ using Kombu messaging library

As I continue to investigate on how to write the plugin or more precise the mechanism driver for the neutron ml2 plugin. I would like to look at the interaction among nova, neutron and its agents. I found out that some of the communication is using the RabbitMQ messaging. I understand that neutron uses the python kombu message library.  So I trying to write a few lines of code to listen to the messages.

I've modified the sample code of worker.py from here to suit my need. Here is my initial code.

from kombu.mixins import ConsumerMixin
from kombu.log import get_logger
from kombu import Queue, Exchange

logger = get_logger(__name__)


class Worker(ConsumerMixin):
    task_queue = Queue('notifications.info', Exchange('neutron', 'topic'))

    def __init__(self, connection):
        self.connection = connection

    def get_consumers(self, Consumer, channel):
        return [Consumer(queues=[self.task_queue],
                         accept=['json'],
                         callbacks=[self.process_task])]

    def process_task(self, body, message):
        print("RECEIVED MESSAGE: %r" % (body, ))
        message.ack()

if __name__ == '__main__':
    from kombu import Connection
    from kombu.utils.debug import setup_logging
    # setup root logger
    setup_logging(loglevel='DEBUG', loggers=[''])

    with Connection('amqp://guest:supersecrete@localhost:5672//') as conn:
        try:
            print(conn)
            worker = Worker(conn)
            worker.run()
        except KeyboardInterrupt:
            print('bye bye')


The above highlighted codes are the changes. I make sure I use the correct queue name and exchange, and the 'guess' password you set in your setup. 

To find out which queue name and topic available, I use

sudo rabbitmqctl list_exchanges

and

sudo rabbitmqctl list_queues

To find out more info about rabbitmqctl,  read the man page here.

I pick the 'notifications.info' queue because I am interesting to look into the 'port.create.start' and 'port.create.end' events which are useful to my current work.

Then I ran the above program,
<Connection: amqp://guest@localhost:5672// at 0x2396050>

Everything seemed fine but I did not receive any events when the port creation was triggered by instance creation.

So what did it go wrong?

After poking a few places, I saw the message from the rabbitmq log. The log is located at /var/log/rabbitmq.

=ERROR REPORT==== 8-Apr-2014::17:47:06 ===
connection <0.25614.29>, channel 1 - soft error:
{amqp_error,precondition_failed,
            "cannot redeclare exchange 'neutron' in vhost '/' with different type, durable, internal or autodelete value",
            'exchange.declare'}

So the default settings of the kombu topic is different from the Neutron ml2 plugin of the OpenStack. RabbitMQ thought I tried to redeclare some of the attributes of the topic. Since sample code uses the default settings of the Exchange class, I check the default settings from the API doc and the settings of the topic using 'rabbitmqctl list_exchanges'.

Note: The default output of 'sudo rabbitmqctl list_exchanges' only shows name and type attributes. To list the addition attributes, you need to specify them as arguments. For example, 'sudo rabbitmqctl list_exchanges name type autodelete' lists the name, type and autodelete attributes. Please see the man page for details.

I found that the autodelete attributes for both topic and queue needs to be set to False. Here is the source code with the highlighted changes.

from kombu.mixins import ConsumerMixin
from kombu.log import get_logger
from kombu import Queue, Exchange

logger = get_logger(__name__)


class Worker(ConsumerMixin):
    task_queue = Queue('notifications.info', Exchange('neutron', 'topic', durable=False), durable=False)

    def __init__(self, connection):
        self.connection = connection

    def get_consumers(self, Consumer, channel):
        return [Consumer(queues=[self.task_queue],
                         accept=['json'],
                         callbacks=[self.process_task])]

    def process_task(self, body, message):
        print("RECEIVED MESSAGE: %r" % (body, ))
        message.ack()

if __name__ == '__main__':
    from kombu import Connection
    from kombu.utils.debug import setup_logging
    # setup root logger
    setup_logging(loglevel='DEBUG', loggers=[''])

    with Connection('amqp://guest:supersecrete@localhost:5672//') as conn:
        try:
            print(conn)
            worker = Worker(conn)
            worker.run()
        except KeyboardInterrupt:
            print('bye bye')

After making the changes, now I can receive the messages. 

Saturday, April 5, 2014

Devstack with Oracle Enterprise Linux

Currently I am working on a development of OpenStack Neutron plugin for a network switch. One of the OS I need to deploy on is Oracle Enterprise Linux. Oracle announced the support of OpenStack in last December, the annocument is here. However, if you try to use devstack to setup your enviroment, it still complains that it is not a supported platform.

If you make the two line changes on the funtions-common in devstack show below.

--- a/functions-common
+++ b/functions-common
@@ -364,8 +364,6 @@ function GetOSVersion {
             fi
         elif [[ $os_VENDOR == "openSUSE project" ]]; then
             os_VENDOR="openSUSE"
-        elif [[ $os_VENDOR == "OracleServer" ]]; then
-            os_VENDOR="Red Hat"
         elif [[ $os_VENDOR =~ Red.*Hat ]]; then
             os_VENDOR="Red Hat"

         fi

Run stack.sh again, you can deploy it without any issue.

In next article, I will talk about using vagrant, pycharm and virtual box to set up your development environment.