Setting up the registry build environment with Docker

Setup the initial environment

We're using a centos 7 minimal VM (Started with Stein builds and moved to Train)

setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux

yum -y install git vim screen
yum -y groupinstall 'development tools'
yum -y install python-devel libffi-devel gcc openssl-devel libselinux-python
yum -y install python-virtualenv

virtualenv /root/virtualenv
source virtualenv/bin/activate

pip install -U pip
pip install 'ansible<2.10'
pip install kolla-ansible
pip install tox

mkdir scratch
cd scratch

git clone https://opendev.org/openstack/kolla

cd kolla/
git checkout stable/stein
cd ..
pip install -r kolla/requirements.txt
pip install kolla/
cd kolla/
cp -rvf etc/kolla /etc/
tox -e genconfig

Setup a local registry

Using docker for this

# setup a local registry
yum install -y yum-utils
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum -y install docker-ce docker-ce-cli containerd.io

# setup insecure registry
vi /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd --insecure-registry 192.168.17.21:5000

systemctl daemon-reload
systemctl start docker
systemctl enable docker
docker run -d -p 5000:5000 --restart=always --name registry registry:2

Make sure the MTU for the registry is set correctly (When running in a VM only)

# warning - when building on a VM environment with an MTU of 1450, you'll see a lot of network timeouts and issues. docker does not check the MTU 1500 / should be 1450.
(virtualenv) [root@registry-dt kolla]# docker network inspect bridge | grep mtu
            "com.docker.network.driver.mtu": "1500"

# update the daemon to run with an mtu of 1450
(virtualenv) [root@registry-dt kolla]# cat /etc/docker/daemon.json
{
  "mtu": 1450
}

# restart daemon
systemctl restart docker

(virtualenv) [root@registry-dt kolla]# docker network inspect bridge | grep mtu
            "com.docker.network.driver.mtu": "1450"

# ok we should be ready to build now

Setup the registry with SSL certs

# registry with SSL
(virtualenv) [root@registry-dt ~]# mkdir -p docker_reg_certs
(virtualenv) [root@registry-dt ~]# openssl req  -newkey rsa:4096 -nodes -sha256 -keyout docker_reg_certs/domain.key -x509 -days 365 -out docker_reg_certs/domain.crt
Generating a 4096 bit RSA private key
..................................................++
...........................................................................................................................++
writing new private key to 'docker_reg_certs/domain.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:Uk
string is too long, it needs to be less than  2 bytes long
Country Name (2 letter code) [XX]:uk
State or Province Name (full name) []:london
Locality Name (eg, city) [Default City]:london
Organization Name (eg, company) [Default Company Ltd]:define tech
Organizational Unit Name (eg, section) []:cloud
Common Name (eg, your name or your server's hostname) []:registry.define-technology.com
Email Address []:info@define-technology.com

mkdir -p /etc/docker/certs.d/registry.define-technology.com:5000
cp docker_reg_certs/domain.crt /etc/docker/certs.d/registry.define-technology.com:5000/ca.crt
cp docker_reg_certs/domain.key /etc/docker/certs.d/registry.define-technology.com:5000/ca.key

docker stop registry
docker rm registry
docker run -d -p 5000:5000 --restart=always --name registry -v $PWD/docker_reg_certs:/certs -v $PWD/docker_reg_auth:/auth -v /reg:/var/lib/registry -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key registry:2

cd /etc/docker/certs.d/registry.define-technology.com\:5000/
openssl genrsa -out client.key 4096
openssl req -new -x509 -text -key client.key -out domain.crt
ln -s ca.crt ca.cert
ln -s domain.crt client.cert
systemctl restart docker

Undo the secure registry setup

# un-setup insecure registry
vi /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd --insecure-registry 192.168.17.21:5000

curl -k  https://registry.define-technology.com:5000/v2/_catalog

Using Letsencrypt for the certs

## Attempt 3 - using letsencrypt for the certs
yum -y install epel-release
yum install certbot
certbot certonly --keep-until-expiring --standalone -d registry.define-technology.com --email info@define-technology.com

autorenew="30 2 * * 1 certbot renew >> /var/log/letsencrypt-renew.log"
(crontab -u root -l; echo "$autorenew" ) | crontab -u root -

cd /etc/letsencrypt/live/registry.define-technology.com/
cp privkey.pem domain.key
cat cert.pem chain.pem > domain.crt
chmod 777 domain.crt
chmod 777 domain.key

# docker run --entrypoint htpasswd registry:2 -Bbn definetech dtpass > auth/htpasswd

# Generate Password for Basic Auth
mkdir auth
# doesnt seem to work
#docker run \
#  --entrypoint htpasswd \
#  registry:2 -Bbn testuser testpassword > auth/htpasswd

[centos@ip-172-31-20-154 ~]$ htpasswd -Bbn definetech dtpass
definetech:$2y$05$eNMK4JvQTgvDJI5EUHg2gOLwye5zfOqFPV6ltHux2Wh.wbuEYkYJ2

echo 'definetech:$2y$05$eNMK4JvQTgvDJI5EUHg2gOLwye5zfOqFPV6ltHux2Wh.wbuEYkYJ2' >> auth/htpasswd

# launch command
docker run -d -p 5000:5000 --restart=always --name registry \
  -v /etc/letsencrypt/live/registry.define-technology.com:/certs \
  -v /etc/letsencrypt/live/registry.define-technology.com//auth:/auth \
  -e "REGISTRY_AUTH=htpasswd" \
  -e "REGISTRY_AUTH_HTPASSWD_REALM=DefineTech Docker Registry" \
  -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
  registry:2

# login
docker login registry.define-technology.com:5000
Username: definetech
Password: