通过neutron-lbaas实现对https的代理，引用官方的解释https://docs.openstack.org/mitaka/networking-guide/config-lbaas.html，neutron-lbaas是OpenStack负载均衡服务的实现，有lbaas v1和lbaas v2两种实现，其中v1是在Juno版本引入，在Liberty版本被弃用；v2是在Kilo版本引入；v1和v2无法兼容，两种实现都使用agent。agent处理HAProxy配置并管理HAProxy守护进程。另一种基于LBaaS v2的实现是Octavia项目，具有独立的API和独立的工作进程，在nova创建的虚拟机中创建负载均衡器。对于Octavia，不再需要agent。

测试环境

CentOS 7.1 （OpenStack Kilo）

Neutron lbaas v2（Kilo）

Barbican（Kilo）

概念介绍

lbaas有一些名字概念，引用官方的解释，它们分别是：

• Load balancer负载均衡器会占用一个子网的端口和ip。
• Listener负载平衡器可以监听多个端口上的请求。每一个端口都由Listener来指定。
• pool池包含通过负载均衡器提供服务的成员列表。
• member

成员是在负载均衡器后端真正提供服务的server。每个成员由提供服务的IP地址和端口来指定。

• Health monitor成员可能会不时地掉线，健康监视器会将服务没有正常响应的成员转移出去。
• 健康监视器与池相关联。
• LBaaS v2通过不同的服务插件有多个实现。两个最常见的实现使用代理或Octavia服务。这两个实现都使用LBaaS v2 API。
• LBaaS v2将listener的概念添加到负载均衡器，LBaaS v2允许在单个负载平衡器IP地址上配置多个listener。

用官方的一张图来看它们之间的关系：

部署lbaas

这里只介绍lbaas v2后端为haproxy，代理https的方式。

安装lbaas

[root@con01 ~(keystone_admin)]# yum install -y openstack-neutron-lbaashaproxy

配置lbaas v2

[root@con01 ~(keystone_admin)]# egrep -v "^$|#" /etc/neutron/lbaas_agent.ini

[DEFAULT]
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
device_driver = neutron_lbaas.drivers.haproxy.namespace_driver.HaproxyNSDriver

device_driver =neutron_lbaas.services.loadbalancer.drivers.haproxy.namespace_driver.HaproxyNSDriver# 默认值是这个，误导性太大

[haproxy]
user_group = haproxy

[root@con01 ~(keystone_admin)]# egrep -v "^$|#" /etc/neutron/neutron_lbaas.conf
[DEFAULT]
[quotas]
[service_providers]
service_provider=LOADBALANCERV2:Haproxy:neutron_lbaas.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver:default
[certificates]

追加lbaas v2 plugin

[root@con01 ~(keystone_admin)]# egrep -v "^$|#" /etc/neutron/neutron.conf
[DEFAULT]
service_plugins =router,neutron_lbaas.services.loadbalancer.plugin.LoadBalancerPluginv2
[keystone_authtoken]
auth_uri = http://10.125.224.21:35357/v2.0
identity_uri = http://10.125.224.52:35357/
admin_tenant_name = services
admin_user = neutron
admin_password = xxxxxx
auth_version = v2

初始化lbaas数据库表

[root@con01 ~(keystone_admin)]# neutron-db-manage --service lbaasupgrade head

创建lbaas v2启动服务脚本

[root@con01 ~(keystone_admin)]# egrep -v "^$|#" /usr/lib/systemd/system/neutron-lbaasv2-agent.service

[Unit]
Description=OpenStack Neutron Load Balancing V2 as a Service Agent
After=syslog.target network.target
[Service]
Type=simple
User=root
ExecStart=/usr/bin/neutron-lbaasv2-agent --config-file/etc/neutron/neutron.conf --config-file /etc/neutron/neutron_lbaas.conf--config-file /etc/neutron/lbaas_agent.ini --log-file/var/log/neutron/lbaasv2-agent.log
PrivateTmp=false
KillMode=process
[Install]
WantedBy=multi-user.target

启动服务

[root@con01 ~(keystone_admin)]# systemctl daemon-reload

[root@con01 ~(keystone_admin)]# systemctl start neutron-lbaasv2-agent.service
[root@con01 ~(keystone_admin)]# systemctl restart neutron-server.service

lbaas https方式

lbaas v2支持TERMINATED_HTTPS和HTTPS，后端driver为haproxy的话，分别对应haproxy代理ssl的这两种模式：

1、haproxy本身提供ssl证书，以http访问后端realserver（lbaasTERMINATED_HTTPS）

这种方式，haproxy需要支持ssl(从haproxy 1.5版本开始支持ssl)，对应的haproy配置如下：

frontendhttps_frontend
bind *:443 ssl crt /etc/ssl/certs/xxxx.pem
mode http
option httpclose
option forwardfor
reqadd X-Forwarded-Proto:\ https
default_backend real_server

backend real_server
mode http
balance roundrobin
server s1 192.168.1.5:80
server s2 192.168.1.6:80

上述方式还需要pem文件，pem是private key和certificate的合体；好在lbaas key管理后端backend支持barbican和local，这里只介绍barbican方式。

2、haproxy本身只提供代理，以https访问后端realserver，简称ssl透传（lbaas HTTPS）

这种方式，haproxy支不支持ssl都没关系; 因为是tcp方式，所以后端realserver就获取不到报头X-Forwarded-*信息。对应的haproxy配置如下：

frontendhttps_frontend
bind *:443
mode tcp
default_backend real_server

backend real_server
mode tcp
balance roundrobin
server node01 192.168.1.5:443
server node02 192.168.1.6:443

部署barbican

barbican安装

参考这篇文章 https://github.com/cloudkeep/barbican/wiki/Barbican-Quick-Start-Guide,推荐使用virtualenv虚拟环境来安装barbican

yum installpython-devel libffi-devel openssl-devel openldap-devel gcc -y # 安装相关依赖包

pip install virtualenv
virtualenv barbican27
source barbican27/bin/activate # 进入虚拟环境；推出虚拟环境是deactivate

pip install MySQL-python
git clone https://github.com/openstack/barbican.git／⌥
git checkout kilo-eol
cd barbican
bin/barbican.sh install

与keystone集成、配置详情见devstack kilo barbican过程https://github.com/openstack/barbican/blob/kilo-eol/contrib/devstack/lib/barbican

barbican集成keystone

keystone user-create --name=barbican --pass=barbican--tenant-id <services租户ID> --email=barbican@example.com

keystone user-role-add --tenant-id <services租户ID> --user-id <barbican用户ID> --role-id <admin角色ID>
keystone service-create --name=barbican --type='key-manager'--description="Barbican Service"
keystone endpoint-create --region NEW_Region
--service-id <barbican服务ID>
--publicurl "http://10.125.224.21:9311"
--adminurl "http://10.125.224.21:9312"
--internalurl "http://10.125.224.21:9311"

创建barbican数据库

create database barbican character set utf8; # 进入mariadb，创建barbican数据库

grant all privileges on barbican.* to barbican@'localhost' identified by'barbican';
grant all privileges on barbican.* to barbican@'%' identified by 'barbican';

修改barbican配置

export BARBICAN_CONF_DIR=/etc/barbican

export BARBICAN_DIR=/opt/barbican

mkdir -p $BARBICAN_CONF_DIR
cp $BARBICAN_DIR/etc/barbican/barbican-api.conf $BARBICAN_CONF_DIR
cp $BARBICAN_DIR/etc/barbican/barbican-api-paste.ini $BARBICAN_CONF_DIR
cp $BARBICAN_DIR/etc/barbican/barbican-admin-paste.ini $BARBICAN_CONF_DIR
cp -R $BARBICAN_DIR/etc/barbican/vassals $BARBICAN_CONF_DIR/
cp $BARBICAN_DIR/etc/barbican/barbican-functional.conf $BARBICAN_CONF_DIR
cp $BARBICAN_DIR/etc/barbican/policy.json $BARBICAN_CONF_DIR

touch /var/log/barbican/api.log
mkdir -p /var/lib/barbican/cache

[root@con01 ~(keystone_admin)]# egrep -v "^$|#" /etc/barbican/barbican-api.conf
[DEFAULT]
verbose = True
debug = False
bind_host = 0.0.0.0
bind_port = 9311
host_href = http://10.125.224.21:9311 # 根据节点ip修改
log_file = /var/log/barbican/api.log
backlog = 4096
max_allowed_secret_in_bytes = 10000
max_allowed_request_size_in_bytes = 1000000
sql_connection = mysql://barbican:barbican@con01/barbican?charset=utf8
sql_idle_timeout = 3600
default_limit_paging = 10
max_limit_paging = 100
workers = 1
delayed_delete = False
scrub_time = 43200
scrubber_datadir = /var/lib/barbican/scrubber
policy_file=/etc/barbican/policy.json
policy_default_rule=default
ampq_durable_queues = True
rabbit_userid=guest
rabbit_password=guest
rabbit_ha_queues = True
rabbit_port=5672
rabbit_hosts=con01:5672 # 根据节点增加
[queue]
enable = False
namespace = 'barbican'
topic = 'barbican.workers'
version = '1.1'
server_name = 'barbican.queue'
[retry_scheduler]
initial_delay_seconds = 10.0
periodic_interval_max_seconds = 10.0
[keystone_notifications]
enable = False
control_exchange = 'openstack'
topic = 'notifications'
allow_requeue = False
version = '1.0'
thread_pool_size = 10
[secrets]
broker = rabbit://guest:guest@con01 # 视情况修改
[secretstore]
namespace = barbican.secretstore.plugin
enabled_secretstore_plugins = store_crypto
[crypto]
namespace = barbican.crypto.plugin
enabled_crypto_plugins = simple_crypto
[simple_crypto_plugin]
kek = 'YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY='
[dogtag_plugin]
pem_path = '/etc/barbican/kra_admin_cert.pem'
dogtag_host = localhost
dogtag_port = 8443
nss_db_path = '/etc/barbican/alias'
nss_db_path_ca = '/etc/barbican/alias-ca'
nss_password = 'password123'
simple_cmc_profile = 'caOtherCert'
[p11_crypto_plugin]
library_path = '/usr/lib/libCryptoki2_64.so'
login = 'mypassword'
mkek_label = 'an_mkek'
mkek_length = 32
hmac_label = 'my_hmac_label'
[kmip_plugin]
username = 'admin'
password = 'password'
host = localhost
port = 5696
keyfile = '/path/to/certs/cert.key'
certfile = '/path/to/certs/cert.crt'
ca_certs = '/path/to/certs/LocalCA.crt'
[certificate]
namespace = barbican.certificate.plugin
enabled_certificate_plugins = simple_certificate
[certificate_event]
namespace = barbican.certificate.event.plugin
enabled_certificate_event_plugins = simple_certificate

[root@con01 ~(keystone_admin)]# vim $BARBICAN_CONF_DIR/vassals/barbican-api.ini
[uwsgi]
buffer-size = 65535 # 增加这个配置

[root@con01 ~(keystone_admin)]# vim /etc/barbican/barbican-api-paste.ini # 修改为如下配置
[pipeline:barbican_api]
pipeline = keystone_authtoken context apiapp
[filter:keystone_authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory
auth_protocol = http
auth_host = 10.125.224.21
auth_port = 35357
auth_uri = http://10.125.224.21:5000
identity_uri = http://10.125.224.21:35357
admin_tenant_name = services
admin_user = barbican
admin_password = barbican
auth_version = v2
signing_dir = /var/lib/barbican/cache

[root@con01 ~(keystone_admin)]# vim /etc/barbican/policy.json # 修改policy权限,增加rule:admin
"secret:decrypt": "rule:secret_decrypt_non_private_read orrule:secret_creator_user or rule:secret_acl_read or rule:admin",
"secret:get": "rule:secret_non_private_read orrule:secret_creator_user or rule:secret_acl_read or rule:admin",
"secrets:get": "rule:all_but_audit or rule:admin",
"container:get": "rule:container_non_private_read orrule:container_creator_user or rule:container_acl_read or rule:admin",

启动barbican api

[root@con01 ~(keystone_admin)]# sourcebarbican27/bin/activate

(barbican27) [root@con01 ~(keystone_admin)]# uwsgi --master --emperor/etc/barbican/vassals/

更新barbicanclient

git clone https://github.com/openstack/python-barbicanclient/opt/

git checkout kilo-eol
mv /usr/lib/python2.7/site-packages/{barbicanclient,barbicanclient.bak}
cp -r /opt/python-barbicanclient/barbicanclient/usr/lib/python2.7/site-packages/

测试

创建https的负载均衡器来验证，参考官方wiki的这篇教程，很详细。https://wiki.openstack.org/wiki/Network/LBaaS/docs/how-to-create-tls-loadbalancer#Update_neutron_config

创建认证串和key

openssl genrsa -des3 -out ca.key 1024

openssl req -new -x509 -days 3650 -key ca.key -out ca.crt
openssl x509 -in ca.crt -out ca.pem
openssl genrsa -des3 -out ca-int_encrypted.key 1024
openssl rsa -in ca-int_encrypted.key -out ca-int.key
openssl req -new -key ca-int.key -out ca-int.csr -subj "/CN=ca-int@acme.com"
openssl x509 -req -days 3650 -in ca-int.csr -CA ca.crt -CAkey ca.key-set_serial 01 -out ca-int.crt
openssl genrsa -des3 -out server_encrypted.key 1024
openssl rsa -in server_encrypted.key -out server.key
openssl req -new -key server.key -out server.csr -subj "/CN=server@acme.com"
openssl x509 -req -days 3650 -in server.csr -CA ca-int.crt -CAkeyca-int.key -set_serial 01 -out server.crt

创建Barbican secrets和containers

barbican secret store --payload-content-type='text/plain'\

                 --name='certificate' \
                 --payload="$(cat server.crt)"

barbican secret store --payload-content-type='text/plain'
--name='private_key'
--payload="$(cat server.key)"

barbican container create --name='tls_container'
--type='certificate' --secret="certificate=$(barbican secret list | awk '/certificate / {print $2}')"
--secret="private_key=$(barbican secret list | awk '/ private_key / {print$2}')"

创建实例和网络

• 创建两个实例，ip分别为10.10.10.20、10.10.10.21
• 创建一个私网的子网subet-net01

创建loadbalancer

neutron lbaas-loadbalancer-create --name lb1subet-net01

创建listener

neutron lbaas-listener-create --loadbalancer lb1 \

      --protocol-port 443 \
      --protocolTERMINATED\_HTTPS \
      --name listener1 \
      --default-tls-container-id=$(barbicancontainer list | awk '/ tls\_container / {print $2}')

创建pool

neutron lbaas-pool-create --name pool1 --protocol HTTP--listener listener1 --lb-algorithm ROUND_ROBIN

创建members

neutron lbaas-member-create --subnet subet-net01 \

  --address 10.10.10.20 \
  --protocol-port 80 pool1

neutron lbaas-member-create --subnet subet-net01
--address 10.10.10.21
--protocol-port 80 pool1

创建healthmonitor

neutron lbaas-healthmonitor-create --type HTTP \

    --delay 3 \
    --max-retries 3 \
    --timeout 3 \
    --pool <pool-id>

.pem文件默认会在形如这样的目录下：/var/lib/neutron/lbaas/v2/b2fa71b0-67ba-4e72-b206-fbc306c6d5dc/b6c834e0-5d37-457e-a669-17b683efb176/

设置lbaas v2配额

neutron quota-update --tenant-id TENANT_UUID--loadbalancer 25

neutron quota-update --tenant-id TENANT_UUID --pool 50

代码调试

部分lbaas相关关键代码：

neutron api入口：
neutron_lbaas.services.loadbalancer.plugin.LoadBalancerPluginv2

lbaas v2 agent入口:
neutron_lbaas.drivers.haproxy.namespace_driver.HaproxyNSDriver

/usr/lib/python2.7/site-packages/neutron_lbaas/drivers/haproxy/namespace_driver.py(203)create()
199 def create(self,loadbalancer):
200 namespace= get_ns_name(loadbalancer.id)
201
202 self._plug(namespace,loadbalancer.vip_port) # 创建lbaas namespace
203 -> self._spawn(loadbalancer) # 根据jinja模版生成haporxy配置文件，并在对应namespace中启动haproxy进程

参考链接

• http://www.cnblogs.com/pmyewei/p/7376921.html
• https://www.cnblogs.com/zhanmeiliang/p/6232245.html
• https://chapter60.wordpress.com/2015/04/14/sample-scripts-to-automatically-set-up-lbaas-v2-loadbalancers-in-devstack/

文章来源于腾讯云开发者社区，点击查看原文