Difference between revisions of "Sregistry Install"

From Define Wiki
Jump to navigation Jump to search
 
(15 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
==Singularity Registry Server==
 +
Singularity Registry Server is a Dockerized web application that an institution or individual can deploy to organize and manage Singularity images. After you install and setup your registry, you are welcomed with the home screen. In this case, our institution is called “Boston labs”:
 +
 
==VM machine Setup==
 
==VM machine Setup==
−
Create a Virtual machine with 4core / 8 GB and 40GB  
+
 
 +
Create a Virtual machine with 4 core / 16 GB and 40GB,
 +
VM with 4GB to 8 GB ram will have error starting up the docker application.
  
 
==Host Installation==
 
==Host Installation==
  
−
Starting form scratch we would need to install docker and docker compose.
+
Starting from scratch we would need to install docker and docker-compose.
  
 
<pre>
 
<pre>
Line 107: Line 112:
 
</pre>
 
</pre>
  
−
Inside secrets file you want to add a SECRET_KEY.
+
Inside secrets file, you want to add a SECRET_KEY.
  
 
You can use the secret key generator to make a new secret key, and call it SECRET_KEY in your secrets.py file, like this:
 
You can use the secret key generator to make a new secret key, and call it SECRET_KEY in your secrets.py file, like this:
Line 113: Line 118:
 
SECRET_KEY="5h1tc_x#3r@5ws7ce-@ci(6b1d-a54xw)!#gkc3ieekswg_@w5"
 
SECRET_KEY="5h1tc_x#3r@5ws7ce-@ci(6b1d-a54xw)!#gkc3ieekswg_@w5"
 
</pre>
 
</pre>
 +
 +
Change it to a unique number
  
 
=== Select Authentication type ===
 
=== Select Authentication type ===
Line 134: Line 141:
  
 
==== Create a github account and register application ====  
 
==== Create a github account and register application ====  
−
<todo>
+
If you dont have github account , create a new one and then follow the steps
−
==== Paste the secrets in the secrates file ====
+
 
−
<todo>
+
- Register the application here https://github.com/settings/applications/new
 +
[[File: Screenshot 2020-05-21 at 6.46.46 PM.png]]
 +
[[File: Screenshot 2020-05-21 at 6.47.15 PM.png]]
 +
 
 +
=== Paste the secrets in the secrats file ===
 +
 
 +
Uncomment the Github Auth section in the secrets file and edit as follows. Paste the Github key and GITHUB_SECRET from the newly created application in Github.
 +
 
 +
<pre>
 +
[root@sregistry sregistry]# vi shub/settings/secrets.py
 +
# -----------------------------------------------------------------------------
 +
# GitHub OAuth
 +
# Only required if ENABLE_GITHUB_AUTH=TRUE in config.py
 +
# http://psa.matiasaguirre.net/docs/backends/github.html?highlight=github
 +
 
 +
SOCIAL_AUTH_GITHUB_KEY="a93cd3f909e95e134a35"
 +
SOCIAL_AUTH_GITHUB_SECRET="690267057e4e2a5771d2a7cf4df987d200149b4f"
 +
 
 +
# You shouldn't actually need this if we aren't using repos
 +
SOCIAL_AUTH_GITHUB_SCOPE = ["repo","user"]
 +
 
 +
# -----------------------------------------------------------------------------
 +
 
 +
</pre>
  
 
== Further edit config ==
 
== Further edit config ==
  
 
We have to provide the DOMAIN_NAME and DOMAIN_NAME_HTTP as Public IP or the IP of the server.  
 
We have to provide the DOMAIN_NAME and DOMAIN_NAME_HTTP as Public IP or the IP of the server.  
−
We can assign private only repostories as well but assgining
+
We can assign private only repositories as well but assigning
  
 
<pre>
 
<pre>
Line 159: Line 189:
 
REGISTRY_URI = "boston"
 
REGISTRY_URI = "boston"
 
GOOGLE_ANALYTICS = None  # "UA-XXXXXXXXX"
 
GOOGLE_ANALYTICS = None  # "UA-XXXXXXXXX"
 +
</pre>
  
 +
By default, any authenticated user in your Registry can create collections, and decide to make them public or private.
 +
 +
<pre>
 
USER_COLLECTIONS = True
 
USER_COLLECTIONS = True
  
Line 173: Line 207:
  
 
We can leave rest as defaults.
 
We can leave rest as defaults.
 +
For more info on the setting
 +
Refer: https://singularityhub.github.io/sregistry/docs/install/settings
 +
 +
=== Launch the application with docker-compose ===
 +
 +
We can launch the application in docker with the docker-compose command. This will by default use the Docker containers available in quay.io. Look at the docker docker-compose.yml to check the setting.
 +
 +
<pre>
 +
docker-compose up -d
 +
</pre>
 +
 +
If there are error remove -d to check the progress of the build. The compose command will bring up the application (and download containers provided on Quay.io, previously on Docker Hub
 +
 +
====LTDR Docker compose====
 +
Perform all these operations in the registry file (which had the docker-compose.yml)
 +
 +
<pre>
 +
# To restart the commands
 +
docker-compose restart uwsgi worker nginx
 +
 +
# To start-stop-restart the containers 
 +
docker-compose stop
 +
docker-compose restart
 +
 +
# Remove old containers
 +
docker-compose rm
 +
 +
# To pull again the images
 +
 +
docker rmi quay.io/vanessa/sregistry
 +
docker rmi quay.io/vanessa/sregistry_nginx
 +
 +
# To inspect the container with errors
 +
 +
docker-compose logs uwsgi
 +
 +
# Will help debug the startup problem and Ctrl-C to kill and restart with -d option
 +
docker-compose up
 +
 +
- List all running containers:
 +
    docker-compose ps
 +
 +
  - Create and start all containers in the background using a docker-compose.yml file from the current directory:
 +
    docker-compose up -d
 +
 +
  - Start all containers, rebuild if necessary:
 +
    docker-compose up --build
 +
 +
  - Start all containers using an alternate compose file:
 +
    docker-compose --file path/to/file up
 +
 +
  - Stop all running containers:
 +
    docker-compose stop
 +
 +
  - Stop and remove all containers, networks, images, and volumes:
 +
    docker-compose down --rmi all --volumes
 +
 +
  - Follow logs for all containers:
 +
    docker-compose logs --follow
 +
 +
 +
 
</pre>
 
</pre>
 +
 +
=== Image storage and the webpage storage ===
 +
 +
By default, the images are stored in the containers which are linked to the directories in the host machine. We can change the location of the image by creating the new folder and changing the location in the docker-compuse.yml file.
 +
 +
<pre>
 +
- ./static:/var/www/static
 +
- ./images:/var/www/images
 +
</pre>
 +
 +
==Upload image in the registry ==
 +
<todo>
 +
 +
== Future integration ==
 +
# Use the google builder with the build the docker images.
 +
# Use different storages and object storage.
 +
 +
== Resources ==
 +
Hardware Powered by: https://demo.vscaler.com
 +
 +
Further reading
 +
#https://docs.docker.com/engine/install/centos/
 +
#https://singularity.lbl.gov/docs-docker
 +
#https://singularityhub.github.io/sregistry/

Latest revision as of 18:02, 21 May 2020

Singularity Registry Server

Singularity Registry Server is a Dockerized web application that an institution or individual can deploy to organize and manage Singularity images. After you install and setup your registry, you are welcomed with the home screen. In this case, our institution is called “Boston labs”:

VM machine Setup

Create a Virtual machine with 4 core / 16 GB and 40GB, VM with 4GB to 8 GB ram will have error starting up the docker application.

Host Installation

Starting from scratch we would need to install docker and docker-compose. 

MacBook-Air-2:~ office$ ssh -i shanaldemo.pem centos@185.93.31.53
[centos@sregistry ~]$ sudo su -
[root@sregistry ~]$ yum update ; reboot
[root@sregistry ~]$ yum groupinstall "Development Tools"
[root@sregistry ~]$ yum install wget
[root@sregistry ~]$ yum install git nginx 
[root@sregistry ~] yum install epel-release

Install Python and PIP

[root@sregistry ~]$ wget https://bootstrap.pypa.io/get-pip.py
[root@sregistry ~]$ sudo /usr/bin/python get-pip.py
[root@sregistry ~]$ sudo pip install ipaddress
[root@sregistry ~]$ sudo pip install oauth2client

Add Docker and Docker-compose

[root@sregistry ~]$ yum install -y yum-utils
[root@sregistry ~]$ yum-config-manager     --add-repo     https://download.docker.com/linux/centos/docker-ce.repo
[root@sregistry ~]$ yum install docker-ce docker-ce-cli containerd.io
[root@sregistry ~]$ usermod -aG docker docker 
[root@sregistry ~]$ yum install docker-compose     
[root@sregistry ~]$ service docker start
[root@sregistry ~]# service docker status
Redirecting to /bin/systemctl status docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2020-05-20 12:43:08 UTC; 1 day 3h ago
     Docs: https://docs.docker.com
 Main PID: 8723 (dockerd)
    Tasks: 29
   Memory: 162.3M
   CGroup: /system.slice/docker.service
           ├─ 8723 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/con...
           └─13050 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-po...

May 20 14:57:15 sregistry.novalocal dockerd[8723]: time="2020-05-20T14:57:15....
May 20 14:57:15 sregistry.novalocal dockerd[8723]: time="2020-05-20T14:57:15....
May 20 14:57:36 sregistry.novalocal dockerd[8723]: time="2020-05-20T14:57:36....
May 20 14:57:36 sregistry.novalocal dockerd[8723]: time="2020-05-20T14:57:36....
May 20 14:57:36 sregistry.novalocal dockerd[8723]: time="2020-05-20T14:57:36....
May 20 14:57:47 sregistry.novalocal dockerd[8723]: time="2020-05-20T14:57:47....
May 20 14:57:47 sregistry.novalocal dockerd[8723]: time="2020-05-20T14:57:47....
May 20 14:57:47 sregistry.novalocal dockerd[8723]: time="2020-05-20T14:57:47....
May 20 14:57:48 sregistry.novalocal dockerd[8723]: time="2020-05-20T14:57:48....
May 20 16:00:02 sregistry.novalocal dockerd[8723]: time="2020-05-20T16:00:02....
Hint: Some lines were ellipsized, use -l to show in full.

[root@sregistry ~]# docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Download Sregistry

[centos@sregistry ~]$ git clone https://github.com/singularityhub/sregistry
[centos@sregistry ~]$ cd 
[root@sregistry sregistry]# ls
AUTHORS.md          get-pip.py  nginx.conf        shub.wsgi
backup              https       paper             Singularity
CHANGELOG.md        images      README.md         static
contributors.svg    LICENSE     requirements.txt  uwsgi.ini
docker-compose.yml  Makefile    run_uwsgi.sh      uwsgi_params.par
Dockerfile          manage.py   scripts           VERSION
docs                nginx       shub

Configration for sregistry setting

There are some setting in the shub and ngnix which we can change to effect the installation. We will change them in these files before we start the application. There are actually only two files you need to change into, generating a settings/secrets.py from the template settings/dummy_secrets.py for application secrets, and settings/config.py to configure your database and registry information.

Secrets

Create a secrets.py file in the shub/setting/ from the template 

cp shub/settings/dummy_secrets.py shub/settings/secrets.py

Inside secrets file, you want to add a SECRET_KEY.

You can use the secret key generator to make a new secret key, and call it SECRET_KEY in your secrets.py file, like this: 

SECRET_KEY="5h1tc_x#3r@5ws7ce-@ci(6b1d-a54xw)!#gkc3ieekswg_@w5"

Change it to a unique number

Select Authentication type

Singularity Registry uses OAuth2 with a token–> refresh flow because it gives the user power to revoke permission at any point, and is a more modern strategy than storing a database of usernames and passwords. A lot of third party Auth option are supports and more can be added.

The preferred auth option is selected in 

[root@sregistry sregistry]# vi shub/settings/config.py

# Which social auths do you want to use?
ENABLE_GOOGLE_AUTH=False
ENABLE_TWITTER_AUTH=False
ENABLE_GITHUB_AUTH=True
ENABLE_GITLAB_AUTH=False
ENABLE_BITBUCKET_AUTH=False

we have selected Github auth option for the testing. Which is fast and simple

Create a github account and register application

If you dont have github account , create a new one and then follow the steps

- Register the application here https://github.com/settings/applications/new Screenshot 2020-05-21 at 6.46.46 PM.png Screenshot 2020-05-21 at 6.47.15 PM.png

Paste the secrets in the secrats file

Uncomment the Github Auth section in the secrets file and edit as follows. Paste the Github key and GITHUB_SECRET from the newly created application in Github. 

[root@sregistry sregistry]# vi shub/settings/secrets.py 
# -----------------------------------------------------------------------------
# GitHub OAuth
# Only required if ENABLE_GITHUB_AUTH=TRUE in config.py
# http://psa.matiasaguirre.net/docs/backends/github.html?highlight=github

SOCIAL_AUTH_GITHUB_KEY="a93cd3f909e95e134a35"
SOCIAL_AUTH_GITHUB_SECRET="690267057e4e2a5771d2a7cf4df987d200149b4f"

# You shouldn't actually need this if we aren't using repos
SOCIAL_AUTH_GITHUB_SCOPE = ["repo","user"]

# -----------------------------------------------------------------------------

Further edit config

We have to provide the DOMAIN_NAME and DOMAIN_NAME_HTTP as Public IP or the IP of the server. We can assign private only repositories as well but assigning 


[root@sregistry sregistry]# vi shub/settings/config.py

DOMAIN_NAME = "http://185.93.31.53"
DOMAIN_NAME_HTTP = "http://185.93.31.53"
DOMAIN_NAKED = DOMAIN_NAME_HTTP.replace("http://", "")

ADMINS = (("vsochat", "vsochat@gmail.com"),)
MANAGERS = ADMINS

HELP_CONTACT_EMAIL = "shanal.tiwari@boston.co.uk"
HELP_INSTITUTION_SITE = "Boston Labs"
REGISTRY_NAME = "Boston Labs"
REGISTRY_URI = "boston"
GOOGLE_ANALYTICS = None  # "UA-XXXXXXXXX"

By default, any authenticated user in your Registry can create collections, and decide to make them public or private. 

USER_COLLECTIONS = True

# Limit users to N collections (None is unlimited)
USER_COLLECTION_LIMIT = 2

# Should registries by default be private, with no option for public?
PRIVATE_ONLY = False

# Should the default for a new registry be private or public?
DEFAULT_PRIVATE = False

We can leave rest as defaults. For more info on the setting Refer: https://singularityhub.github.io/sregistry/docs/install/settings

Launch the application with docker-compose

We can launch the application in docker with the docker-compose command. This will by default use the Docker containers available in quay.io. Look at the docker docker-compose.yml to check the setting. 

docker-compose up -d

If there are error remove -d to check the progress of the build. The compose command will bring up the application (and download containers provided on Quay.io, previously on Docker Hub

LTDR Docker compose

Perform all these operations in the registry file (which had the docker-compose.yml) 

# To restart the commands 
docker-compose restart uwsgi worker nginx

# To start-stop-restart the containers  
docker-compose stop
docker-compose restart

# Remove old containers
docker-compose rm 

# To pull again the images

docker rmi quay.io/vanessa/sregistry
docker rmi quay.io/vanessa/sregistry_nginx

# To inspect the container with errors 

docker-compose logs uwsgi

# Will help debug the startup problem and Ctrl-C to kill and restart with -d option
docker-compose up

- List all running containers:
    docker-compose ps

  - Create and start all containers in the background using a docker-compose.yml file from the current directory:
    docker-compose up -d

  - Start all containers, rebuild if necessary:
    docker-compose up --build

  - Start all containers using an alternate compose file:
    docker-compose --file path/to/file up

  - Stop all running containers:
    docker-compose stop

  - Stop and remove all containers, networks, images, and volumes:
    docker-compose down --rmi all --volumes

  - Follow logs for all containers:
    docker-compose logs --follow

Image storage and the webpage storage

By default, the images are stored in the containers which are linked to the directories in the host machine. We can change the location of the image by creating the new folder and changing the location in the docker-compuse.yml file. 

- ./static:/var/www/static
- ./images:/var/www/images

Upload image in the registry

<todo>

Future integration

  1. Use the google builder with the build the docker images.
  2. Use different storages and object storage.

Resources

Hardware Powered by: https://demo.vscaler.com

Further reading

  1. https://docs.docker.com/engine/install/centos/
  2. https://singularity.lbl.gov/docs-docker
  3. https://singularityhub.github.io/sregistry/
Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=Sregistry_Install&oldid=30924"