Adding your own configuration to OpenStack Neutron using oslo.config

If you are doing OpenStack development,  eventually you will want to add your own configuration in OpenStack for you own plugin/mechanism driver. The steps required is pretty simple.

Let say my plugin/mechanism driver need to talk to a web service running to perform some operation and I do not want to hard code the web service configuration.

OpenStack uses oslo.config to do the configuration parsing. 

First, you need to tell oslo.config what configuration options are needed.

Here is the sample code fragment to create my configuration parameter list. 

ml2_my_opts = [

    cfg.StrOpt('hostname', default=None,

               help=_("The hostname of the my service")),

    cfg.IntOpt('port', default=8443,

               help=_("The port number of my service")),

    cfg.StrOpt('username', default=None,

               help=_("The username of my service")),

    cfg.StrOpt('password', default=None,

               help=_("The password of my service"))]

I specify the hostname, port, username and password. I set the port default value as 8443.

Then I register it by doing the following.

cfg.CONF.register_opts(my_service_opts, "my_service")

"my_service" is optional. If specified, it is the namespace I am going to use. 

To retrieve the configuration,

hostname = cfg.CONF.my_service_opts.hostname

port = cfg.CONF.my_service_opts.port

username = cfg.CONF.my_service_opts.username

password = cfg.CONF.my_service_opts.password

Now the coding is done. 

Let put the parameters on the configuration file. The configuration file is specified by the --config-file in the argument of python. For example, 

stack    17357  0.1  0.0 284908 52156 pts/9    S+   08:57   0:05 python /usr/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini

For the neutron-server, two configuration files are specified. One is /etc/neutron/neutron.conf and the other is /etc/neutron/plugins/ml2/ml2_conf.ini.

For the above example, I will put the following configuration in one of the configuration file listed above.

[my_service]

hostname=my_host

port=8443

username=guest

password=guest

With oslo.config, adding your own additional configuration is extremely simple.

Listen to OpenStack Neutron Messages from RabbitMQ using Kombu messaging library

As I continue to investigate on how to write the plugin or more precise the mechanism driver for the neutron ml2 plugin. I would like to look at the interaction among nova, neutron and its agents. I found out that some of the communication is using the RabbitMQ messaging. I understand that neutron uses the python kombu message library.  So I trying to write a few lines of code to listen to the messages.

I've modified the sample code of worker.py from here to suit my need. Here is my initial code.

from kombu.mixins import ConsumerMixin
from kombu.log import get_logger
from kombu import Queue, Exchange

logger = get_logger(__name__)


class Worker(ConsumerMixin):
    task_queue = Queue('notifications.info', Exchange('neutron', 'topic'))

    def __init__(self, connection):
        self.connection = connection

    def get_consumers(self, Consumer, channel):
        return [Consumer(queues=[self.task_queue],
                         accept=['json'],
                         callbacks=[self.process_task])]

    def process_task(self, body, message):
        print("RECEIVED MESSAGE: %r" % (body, ))
        message.ack()

if __name__ == '__main__':
    from kombu import Connection
    from kombu.utils.debug import setup_logging
    # setup root logger
    setup_logging(loglevel='DEBUG', loggers=[''])

    with Connection('amqp://guest:supersecrete@localhost:5672//') as conn:
        try:
            print(conn)
            worker = Worker(conn)
            worker.run()
        except KeyboardInterrupt:
            print('bye bye')

The above highlighted codes are the changes. I make sure I use the correct queue name and exchange, and the 'guess' password you set in your setup. 

To find out which queue name and topic available, I use

sudo rabbitmqctl list_exchanges

and

sudo rabbitmqctl list_queues

To find out more info about rabbitmqctl,  read the man page here.

I pick the 'notifications.info' queue because I am interesting to look into the 'port.create.start' and 'port.create.end' events which are useful to my current work.

Then I ran the above program,

<Connection: amqp://guest@localhost:5672// at 0x2396050>

Everything seemed fine but I did not receive any events when the port creation was triggered by instance creation.

So what did it go wrong?

After poking a few places, I saw the message from the rabbitmq log. The log is located at /var/log/rabbitmq.

=ERROR REPORT==== 8-Apr-2014::17:47:06 ===

connection <0.25614.29>, channel 1 - soft error:

{amqp_error,precondition_failed,

            "cannot redeclare exchange 'neutron' in vhost '/' with different type, durable, internal or autodelete value",

            'exchange.declare'}

So the default settings of the kombu topic is different from the Neutron ml2 plugin of the OpenStack. RabbitMQ thought I tried to redeclare some of the attributes of the topic. Since sample code uses the default settings of the Exchange class, I check the default settings from the API doc and the settings of the topic using 'rabbitmqctl list_exchanges'.

Note: The default output of 'sudo rabbitmqctl list_exchanges' only shows name and type attributes. To list the addition attributes, you need to specify them as arguments. For example, 'sudo rabbitmqctl list_exchanges name type autodelete' lists the name, type and autodelete attributes. Please see the man page for details.

I found that the autodelete attributes for both topic and queue needs to be set to False. Here is the source code with the highlighted changes.

from kombu.mixins import ConsumerMixin

from kombu.log import get_logger

from kombu import Queue, Exchange

logger = get_logger(__name__)

class Worker(ConsumerMixin):

    task_queue = Queue('notifications.info', Exchange('neutron', 'topic', durable=False), durable=False)

    def __init__(self, connection):

        self.connection = connection

    def get_consumers(self, Consumer, channel):

        return [Consumer(queues=[self.task_queue],

                         accept=['json'],

                         callbacks=[self.process_task])]

    def process_task(self, body, message):

        print("RECEIVED MESSAGE: %r" % (body, ))

        message.ack()

if __name__ == '__main__':

    from kombu import Connection

    from kombu.utils.debug import setup_logging

    # setup root logger

    setup_logging(loglevel='DEBUG', loggers=[''])

    with Connection('amqp://guest:supersecrete@localhost:5672//') as conn:

        try:

            print(conn)

            worker = Worker(conn)

            worker.run()

        except KeyboardInterrupt:

            print('bye bye')

After making the changes, now I can receive the messages. 

Exciting Python Machine Learning Package (scikit-learn)

A while back, I blogged about using rpy2 to leverage the power of plotting and aplenty model selection of R in python. It's usable but still a bit cumbersome. Turns out there is even an easier way to do machine learning in python: use the scikit-learn.

Scikit-learn is another project born out of Google's summer of code. It's currently only 0.11, but has been around for 2+ years and supports many models in supervised and unsupervised learning. Its BSD license may be more attractive to people who is considering embedding a machine learning library in their own products. Overall it seems to be a very exciting new module to be added to python's machine learning toolkit. 

Their web site is full of useful info (docs, tutorials, and demo videos), so go check it out: scikit-learn.org

P.S. if you encounter problems installing scikit-learn on your mac, here is a very useful page on installing all the required packages: http://kral4u.blogspot.com/2012/07/installing-numpy-scipy-matplotlib.html. Also highly recommend upgrading easy_install to pip.

How to Install rpy2 on Mac OS X 10.7 (Lion)

Python and R are powerful tools for machine learning and data analysis. Like super heroes in movies, their power can be unmatched when combined. Python provides a richer set of scientific and data processing modules, while R provides easier plotting and analytic modeling capabilities. 

To access R from python, you will need to install the rpy2* package. Usually, it's just as easy as running the python "easy_install":

easy_install rpy2

However, I found I had to jump through a few hoops to get the rpy2 package compiled and installed on my mac. The time I spent/wasted convinced me the info is worth sharing. 

If you encounter errors while running easy_install on your mac os 10.7, try the following steps: 

1. Install Xcode on your mac.

You will need the gcc compiler to build the rpy2. If Xcode is not installed, download and install it from the mac app store. (It's free.) Then install the command line tools from the xcode (go to preferences -> Downloads tab and click the "Install" button next to the Command Line Tools). This is what the preferences pop-up looks like after installation.

    

    Note if you upgraded your mac os to 10.7 (lion) from 1.6 (snow leopard) and had xcode installed before the upgrade, you still have to do this since the old xcode tools were moved from /usr/bin/ to /Developer/usr/bin (it was a surprise to me) and the old binaries may not work properly.

2. Make sure your R installation is shared library enabled. If not, build it yourself. 

You will need the header files from R to build rpy2. If your R is installed from a binary only release (i.e installed from the one click mac os R package), you need to download the R source code and build it yourself. Here is the instruction from CRAN on how to build R from the source: http://cran.r-project.org/doc/manuals/R-admin.html#Installing-R-under-_0028Mac_0029-OS-X

You may have to install the gfortran to build R. Unfortunately the gfortran link provided from CRAN site does not work for osx 10.7. Make sure you get the right version. You can find a good reference here:

 http://web.mit.edu/mfloyd/www/computing/mac/gfortran/

3. Download and build rpy2. 

The rpy2 page on source forge (http://rpy.sourceforge.net/rpy2/doc-2.2/html/overview.html#installation) provides pretty good instructions on how to build and install rpy2.  

Notice the default python installation (/usr/bin/python) on Lion is python 2.7. If you encounter version compatibility issue, you can still build it using python 2.6:

export ARCHFLAGS='-arch i386 -arch x86_64'

/usr/bin/python2.6 setup.py build  # specify --r-home if R not in default location

4. Install and test.

After successfully building it, you can install the python package (to the same version you used to build the package):

python setup.py install

and verify your installation with the following:

import rpy2.robjects as robjects

If you don't see any error, congratulations, your rpy2 is ready to go.

*rpy2 is the redesign of the rpy module. It's interface to R is better designed and is recommended over the rpy module.