OpenOnDemand

From Define Wiki
Revision as of 18:42, 7 November 2023 by Antony (talk | contribs) (add grafana hosting behind ood)
Jump to navigation Jump to search

Setting up Open OnDemand

What is it?

Open OnDemand is a portal to run web applications to help add ease of use functionality to any Compute Cluster.

the deplyonment is now pretty much handled by the stackhpc/osc playbooks BUT I'm adding some extra notes do document how to do stuff by hand to tweak an existing setup.

I'm getting 500 errors

This is frequently caused by routing issues. One that got me is that in CIX the login node cannot curl to it's external ip 

[root@login-1 config]# curl 217.74.56.217
^C

we solve this by adding this to /etc/hosts TODO lookup how to configure a response policy zone in freeipa

[rocky@login-1 ~]$ cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

1 localhost localhost.localdomain localhost6 localhost6.localdomain6

10.0.3.240 nuig-cluster.define-technology.com

I'm getting Host "whatever" not specified in allowlist or cluster configs using the shell app

this is fixed by editing /etc/ood/config/apps/shell/env and adding or appending the OOD_SSHHOST_ALLOWLIST variable which is fully documented here https://osc.github.io/ood-documentation/latest/customizations.html?highlight=ood_sshhost_allowlist#set-ssh-allowlist I have three classes of nodes test-00[1-4] small-00[1-4] and login-1 so this regex works for me 

[root@login-1 config]# cat /etc/ood/config/apps/shell/env
OOD_SHELL_ORIGIN_CHECK="https://nuig-cluster.define-technology.com"
OOD_SSHHOST_ALLOWLIST="test-\d+,small-\d+,login-\d+"

Switch OOD to use FreeIPA and ondemand-dex auth using ldaps

on the portal node (here login-1) first step install the ondemand-dex package 

dnf install -y ondemand-dex

before we change any settings we need to find out the cn and dn voodoo for the cluster we are in. Thankfully freeipa makes this a lot easier than it could be assuming you can remember it. we are going to need ldapsearch now don't panic is going to be ok because you can sort out all auth and bind details using kerberos (yes there had to be good reason for kerberos and this is it).

before we do antything lets make a new service account for dex to use to authenticate to freeipa. This is so we can connect in the first place, I think this is required for groupsearch stuff and if se we may need to mess with the permissions of the dex user later but it works ok so far without this. this is a normal user and we can modify the shell to nologin if we want so that IF someone was able to get this password damage would be much limited. 

kinit antony
ipa user add dex --first=dex --last="service account" --random
kdestroy -A
kinit dex
# it will ask you to change the password you NEED to do this for the account to work or it WILL FAIL

I repeat you will need to login or kinit once as the service account user (dex) to change the initial pw or everything will fail to work now we can install ldapsearch and run it with NO args 

dnf install -y openldap-clients &> /dev/null
kinit antony
ldapsearch

and this will dump the entire contents of the ldap visible to this user. Seriously that's it. if you wont to pick a specific server then add -H ldaps://freeipa-1.cluster.internal with the hostname of your ipa server

the following output is just the lines containing dex as this allows us to see what we need 

[rocky@login-1 ~]$ ldapsearch | grep dex
SASL/GSSAPI authentication started
SASL username: dex@CLUSTER.INTERNAL
SASL SSF: 256
SASL data security layer installed.
# dex, users, compat, cluster.internal
dn: uid=dex,cn=users,cn=compat,dc=cluster,dc=internal
gecos: dex service account
cn: dex service account
homeDirectory: /define/home/dex
uid: dex
# dex, groups, compat, cluster.internal
dn: cn=dex,cn=groups,cn=compat,dc=cluster,dc=internal
cn: dex
member: uid=dex,cn=users,cn=accounts,dc=cluster,dc=internal
# dex, users, accounts, cluster.internal
dn: uid=dex,cn=users,cn=accounts,dc=cluster,dc=internal
givenName: dex
uid: dex
cn: dex service account
displayName: dex service account
gecos: dex service account
krbPrincipalName: dex@CLUSTER.INTERNAL
homeDirectory: /define/home/dex
mail: dex@cluster.internal
krbCanonicalName: dex@CLUSTER.INTERNAL
# dex, groups, accounts, cluster.internal
dn: cn=dex,cn=groups,cn=accounts,dc=cluster,dc=internal
cn: dex
description: User private group for dex
mepManagedBy: uid=dex,cn=users,cn=accounts,dc=cluster,dc=internal

it's still a lot but it is sufficient to see that to bind as a specific user dex we need to use the path uid=dex,cn=users,cn=accounts,dc=cluster,dc=internal

we can see that all users are under cn=users,cn=accounts,dc=cluster,dc=internal

and all groups are under cn=groups,cn=accounts,dc=cluster,dc=internal

if yours are different then use yours!

I'm ignoring the cn=users,cn=compat,dc=cluster,dc=internal ones for now.

now we edit the /etc/ood/config/ood-portal.yml file

the current version of ondemand docs say to using this as a guide so here it is 

dex:
  connectors:
    - type: ldap
      id: ldap
      name: LDAP
      config:
        host: openldap.my_center.edu:636
        insecureSkipVerify: false
        bindDN: cn=admin,dc=example,dc=org
        bindPW: admin
        userSearch:
          baseDN: ou=People,dc=example,dc=org
          filter: "(objectClass=posixAccount)"
          username: uid
          idAttr: uid
          emailAttr: mail
          nameAttr: gecos
          preferredUsernameAttr: uid
        groupSearch:
          baseDN: ou=Groups,dc=example,dc=org
          filter: "(objectClass=posixGroup)"
          userMatchers:
            - userAttr: DN
              groupAttr: member
          nameAttr: cn

remove or comment all httpd-auth lines like these: 

#auth:
#- 'AuthType Basic'
#- 'AuthName "Open OnDemand"'
#- 'AuthBasicProvider PAM'
#- 'AuthPAMService ood'
#- 'Require valid-user'
#- 'ProxyPreserveHost On'

using the details we have from our freeipa search I have used this 

dex:
  connectors:
    - type: ldap
      id: ldap
      name: LDAP
      config:
        host: freeipa-1.cluster.internal:636
        insecureSkipVerify: false
        bindDN: uid=dex,cn=users,cn=accounts,dc=cluster,dc=internal
        bindPW: <redacted>
        userSearch:
          baseDN: cn=users,cn=accounts,dc=cluster,dc=internal
          filter: "(objectClass=person)"
          username: uid
          idAttr: uid
          emailAttr: mail
          #nameAttr: gecos
          preferredUsernameAttr: uid
        groupSearch:
          baseDN: cn=groups,dc=cluster,dc=internal
          filter: "(objectClass=group)"
          userMatchers:
            - userAttr: uid
              groupAttr: member
          nameAttr: name

Stop httpd from listening on port 80 and use certbot to create a trusted SSL cert that won't ever expire

first up make sure you ARE allowing port 80 through the firewall and security groups

next comment out all lines in /etc/httpd/conf.d/welcome.conf i should look like this. Do not delete it or it will come back from the dead when you update apache 

root@login-1 certs]# cat /etc/httpd/conf.d/welcome.conf
#
# This configuration file enables the default "Welcome" page if there
# is no default index page present for the root URL.  To disable the
# Welcome page, comment out all the lines below.
#
# NOTE: if this file is removed, it will be restored on upgrades.
#
#<LocationMatch "^/+$">
#    Options -Indexes
#    ErrorDocument 403 /.noindex.html
#</LocationMatch>
#
#<Directory /usr/share/httpd/noindex>
#    AllowOverride None
#    Require all granted
#</Directory>
#
#Alias /.noindex.html /usr/share/httpd/noindex/index.html
#Alias /poweredby.png /usr/share/httpd/icons/apache_pb3.png

comment out Listen 80 in /etc/httpd/conf/httpd.conf on mine its around line 45.

restart httpd

check it's not listening with systemctl and ss -t | grep -e ":http" -e ":80" 

[root@login-1 certs]# grep "Listen 80" /etc/httpd/conf/httpd.conf -n
45:#Listen 80
[root@login-1 tls]# systemctl restart httpd
[root@login-1 tls]# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/httpd.service.d
           └─ood-portal.conf, ood.conf
   Active: active (running) since Tue 2023-11-07 12:41:18 GMT; 8s ago
     Docs: man:httpd.service(8)
  Process: 98483 ExecStartPre=/opt/ood/ood-portal-generator/sbin/update_ood_portal --rpm (code=exited, status=0/SUCCESS)
 Main PID: 98499 (httpd)
   Status: "Started, listening on: port 443, port 81"
    Tasks: 213 (limit: 100627)
   Memory: 44.2M
   CGroup: /system.slice/httpd.service
           ├─98499 /usr/sbin/httpd -DFOREGROUND
           ├─98500 /usr/sbin/httpd -DFOREGROUND
           ├─98501 /usr/sbin/httpd -DFOREGROUND
           ├─98502 /usr/sbin/httpd -DFOREGROUND
           └─98503 /usr/sbin/httpd -DFOREGROUND

Nov 07 12:41:18 login-1.cluster.internal systemd[1]: Starting The Apache HTTP Server...
Nov 07 12:41:18 login-1.cluster.internal update_ood_portal[98483]: No change in Apache config.
Nov 07 12:41:18 login-1.cluster.internal update_ood_portal[98483]: No change in the Dex config.
Nov 07 12:41:18 login-1.cluster.internal httpd[98499]: [Tue Nov 07 12:41:18.946520 2023] [so:warn] [pid 98499:tid 140163328375104] AH01574: module status_module is already loaded, skipping
Nov 07 12:41:18 login-1.cluster.internal systemd[1]: Started The Apache HTTP Server.
Nov 07 12:41:18 login-1.cluster.internal httpd[98499]: Server configured, listening on: port 443, port 81
[root@login-1 certs]# ss -t | grep -e :http -e :80
LAST-ACK 0      1               10.0.3.240:59462          74.125.193.99:https
ESTAB    0      0               10.0.3.240:53178          92.122.160.95:https
ESTAB    0      0               10.0.3.240:43826          104.82.170.85:https
ESTAB    0      0      [::ffff:10.0.3.240]:https   [::ffff:82.1.125.18]:56555

notice that it's only listening on 443 (https) and 81 (not so sure it is but hey) this is good.

now we can install certbot dnf install -y certbot and run it against our domain which you have to point at the public ip certbot certonly -d <public FQDN> . In the following example I and doing a dry run as I have run this before but you should remove this. 

[root@login-1 certs]# certbot certonly -d nuig-cluster.define-technology.com --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Spin up a temporary webserver (standalone)
2: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Simulating renewal of an existing certificate for nuig-cluster.define-technology.com
The dry run was successful.

mine didn't add the renew job to cron or a timer so I did the following to auto renew the certs 

cat >/etc/cron.daily/certbot-renew <<EOF
#!/bin/bash
/usr/bin/certbot renew &>/dev/null
EOF
chmod a+x /etc/cron.daily/certbot-renew

now you should have the certs in /etc/letsencrypt/<fqdn>/ directory I tried putting this path directly into the apache config files but if fails what does work is to link to these. I put these links into /etc/pki/tls/certs and /etc/pki/tls/private 

ln -s /etc/letsencrypt/live/nuig-cluster.define-technology.com/fullchain.pem  /etc/pki/tls/certs/nuig-cluster.define-technology.com.crt
ln -s /etc/letsencrypt/live/nuig-cluster.define-technology.com/privkey.pem /etc/pki/tls/private/nuig-cluster.define-technology.com.key

[root@login-1 certs]# ll /etc/pki/tls/private/nuig-cluster.define-technology.com.key  /etc/pki/tls/certs/nuig-cluster.define-technology.com.crt
lrwxrwxrwx. 1 root root 70 Nov  7 12:42 /etc/pki/tls/certs/nuig-cluster.define-technology.com.crt -> /etc/letsencrypt/live/nuig-cluster.define-technology.com/fullchain.pem
lrwxrwxrwx. 1 root root 68 Nov  7 12:40 /etc/pki/tls/private/nuig-cluster.define-technology.com.key -> /etc/letsencrypt/live/nuig-cluster.define-technology.com/privkey.pem

finally modify the /etc/ood/config/ood-portal run /opt/ood/ood-portal-generator/sbin/update_ood_portal and restart httpd and opdemand-dex

we need to edit the ssl section I made mine: 

cat /etc/ood/config/ood-portal

<...snip... />

# List of SSL Apache directives
# Example:
#     ssl:
#       - 'SSLCertificateFile "/etc/pki/tls/certs/www.example.com.crt"'
#       - 'SSLCertificateKeyFile "/etc/pki/tls/private/www.example.com.key"'
# Default: null (no SSL support)
ssl:
#- 'SSLCertificateFile /etc/pki/tls/certs/localhost.crt'
#- 'SSLCertificateKeyFile /etc/pki/tls/private/localhost.key'
- 'SSLCertificateFile /etc/pki/tls/certs/nuig-cluster.define-technology.com.crt'
- 'SSLCertificateKeyFile /etc/pki/tls/private/nuig-cluster.define-technology.com.key'
- 'SSLProtocol all -TLSv1.1 -TLSv1 -SSLv2 -SSLv3'
- 'SSLCipherSuite ALL:+HIGH:!ADH:!EXP:!SSLv2:!SSLv3:!MEDIUM:!LOW:!NULL:!aNULL'
- 'SSLHonorCipherOrder On'
- 'SSLCompression off'
- 'SSLSessionTickets Off'

<...snip... />

/opt/ood/ood-portal-generator/sbin/update_ood_portal

systemctl restart httpd ondemand-dex

systemctl status httpd ondemand-dex

i.e. I commented out the old SSL cert liines from the stackhpc appliance (we will need to override this in the playbooks later) and put in the new links

hosting grafana behind OOD

you have probably worked out that you can use the <fqdn>/node/<hostnmame>/port syntax to point a link to https://nuig-cluster.define-technology.com/node/controller/3000/ and get apparent SSL access to the http hosted grafana ui behind ood but you might get stuck wondering why you keep getting redirected to https://nuig-cluster.define-technology.com/login.

wonder no more!

it's not a ood portal redirect issue this IS working perfectly it's an issue with the default grafana config on the controller node (NOT LOGIN). We need to change /etc/grafana/grafana.ini in the #HTTP options in the server setting and disable authentication the text below is the after state: 

# HTTP options
[server]
http_addr = controller
http_port = 3000
domain = controller.cluster.internal
root_url = https://nuig-cluster.define-technology.com/node/controller/3000
protocol = http
enforce_domain = False
socket =
cert_key =
cert_file =
enable_gzip = False
static_root_path = public
router_logging = False
serve_from_sub_path = True

in this I had to change just the

  • serve_from_sub_path
  • root_url

settings

I also had to add in this to disable auth as I have nothing to hide here 

# Users management and registration
[users]
allow_sign_up = False
auto_assign_org_role = Viewer
default_theme = dark
[emails]
welcome_email_on_sign_up = False

# Authentication
[auth]
disable_login_form = False
oauth_auto_login = False
disable_signout_menu = False
signout_redirect_url =
[auth.anonymous]
enabled = True
org_name = Main Org.
org_role = Viewe

and then restart this on the controller 

systemctl restart grafana-server

everything below this line is ancient but STILL useful Dashboard.png

It provides the following apps out of the box which I shall go through the configuration of using my trusty virtual TrinityX cluster on my file server.

  1. Home directory browser with editor
  2. Active Job Viewer
  3. Job creator
  4. Cluster Shell
  5. Remote Visualisation

here are screenshots of each of the working modules:

OOnDemandFilesBrowser.png OOnDemandActiveJobViewer.png OOnDemandJobCreator.png OOnDemandWebTerminal.png OOnDemandRemoteDesktop1.png OOnDemandRemoteDesktop2.png

Install RPMs on Portal node

yum install centos-release-scl
yum install https://yum.osc.edu/ondemand/1.5/ondemand-release-web-1.5-1.el7.noarch.rpm
yum install ondemand
systemctl start httpd24-httpd

Configure The Portal Node

Configure Authentication to use LDAP and allow remote access to nodes using reverse proxy

Backup the original blank config and configure basic ldap authentication against the cluster LDAP for now. If you want something more advanced then read the ood docs. Note: fill in the LDAP URL according to the cluster config on your system. You can find this in the sssd.conf file normally along with the search dn which is appended before the ?uid below. In a default TrinityX cluster the url will be <controller hostname>.cluster:636/ou=People,dc=local?uid the .cluster is the internal domain name and cannot be removed as it will fails the ssl hostchecks

Note that the last three lines configure the reverse proxy settings. the most important is the host_regex line. If a host does not match this then no proxy will connect to the node this is to stop your portal server being used to redirect traffic to node you do not control. In TrinityX we use the internal TLD domain suffix .cluster '[\w.-]+\.cluster' limits access to these servers. If you wanted to be more specific then you could use something like node\d+\.cluster or (rvis|interactive)\d+\.cluster to limit access to more specific hosts in the cluster.

some LDAP servers require uncommenting the AuthLDAPGroupAttribute and AuthLDAPGroupAttributeIsDN lines 

cp /etc/ood/config/ood_portal.yml /etc/ood/config/ood_portal.yml.orig
cat >> /etc/ood/config/ood_portal.yml << EOF
auth:
  - 'AuthType Basic'
  - 'AuthName "private"'
  - 'AuthBasicProvider ldap'
  - 'AuthLDAPURL "ldaps://trinityx.cluster:636/ou=People,dc=local?uid"'
#  - 'AuthLDAPGroupAttribute memberUid'
#  - 'AuthLDAPGroupAttributeIsDN off'
  - 'RequestHeader unset Authorization'
  - 'Require valid-user'
host_regex: '[\w.-]+\.cluster'
node_uri: '/node'
rnode_uri: '/rnode'
EOF

put a comment in /opt/rh/httpd24/root/etc/httpd/conf.modules.d/01-ldap.confand explicitly set LDAPLibraryDebug to 0 (off) as for some stupid reason failing LDAP authentications will NOT result in an error in /var/log/httpd24/error_log. If you see an internal server error after login and nothing in the logs then set LDAPLibraryDebug to 1 and it may point you in the right direction (the misssing .cluster) for me 

cat >> /opt/rh/httpd24/root/etc/httpd/conf.modules.d/01-ldap.conf << EOF
# change the following LDAPLibraryDebug line to 1 if you get 500 (internal server) errors after login
LDAPLibraryDebug 0
EOF

Now we need to run the portal config generator and restart the service 

/opt/ood/ood-portal-generator/sbin/update_ood_portal
systemctl try-restart httpd24-httpd.service httpd24-htcacheclean.service

If you access the website now it should you prompt you for a login and you should be able to login as a normal user with your cluster credentials and then display the most basic portal page, it will look bare compared to the above screenshots.

Configure Slurm and Remote Desktop sessions

Configure Slurm and remote access plugins

Lets fix that now and tell OOD how to use Slurm and create remote Visualisation settings I will cover VirtualGL for 3d accelerated using cluster GPU nodes another time. If you can install nvidia drivers that *REALLY WORK* then VGL is easy.

In the following the login host should be set to a LOGIN node. I am using the controller node here as I have no login node. I have added my user to the admins group in LDAP to allow me to login to the controller node. You could also disable the filter restricting non-root logins to the controller node if you wish. If you do not then you will get a rather unhelpful Authentication failed message when you use the cluster shell functions.

Note that the following uses shared apps and modules to make virtualgl and websockify work so that you do not need to add them to the nodes and re-provision them.

See the /etc/ood/config/clusters.d/my_cluster.yml file? you can call this anything you like like the cluster name and it should correspond to the title: "My Cluster" section. You can have multiple clusters. While I am mentioning this see the cluster: "cluster" in the job section? This is the slurm cluster name in case this have been customised from the default. 

mkdir -p /etc/ood/config/clusters.d/
cat > /etc/ood/config/clusters.d/my_cluster.yml << EOF
---
v2:
  metadata:
    title: "My Cluster"
  login:
    host: "trinityx.cluster"
  job:
    adapter: "slurm"
    cluster: "cluster"
    bin: "/usr/sbin/"
    conf: "/etc/slurm/slurm.conf"
    bin_overrides:
      sbatch: "/usr/bin/sbatch"
      squeue: "/usr/bin/squeue"
      scontrol: "/usr/bin/scontrol"
      scancel: "/usr/bin/scancel"
  batch_connect:
    basic:
      script_wrapper: |
        module purge
        %s
    vnc:
      script_wrapper: |
        module purge
        module add turbovnc websockify
        export WEBSOCKIFY_CMD="websockify"
        %s
EOF

Configure remote Desktop form

All the services are now configured ready for the bc_desktop module note we have NOT installed TurboVNC or websockify yet. We will get there later. Lets sort out the bc_desktop form first. There are lots of config options here to allow you to hide or set sensible defaults. Basically if you set a value in line in the yaml e.g. desktop: mate the option will be hardcoded and NOT show up to the user. If you set it using the value: option like this: 

  desktop:
    value: "xfce"

then it will be set as a default and the user can modify it. You can set things to null to hide them if they are not needed. you can use the label override to change the label on the form and you may add markdown formatted help descriptions to fields with the help: override.

Do NOT bother configuring the bc_vnc_resolution. The default is to resize the remote desktop based on the size of the window at the client end dynamically.

if you must try the MATE desktop YMMV but when I used it I had problems with dconf being unable to write into /var/run/$UID/ as it did not exist for the user. I could have created a slurm prolog to do this as root but:

  1. I prefer XFCE which doesn't need this.
  2. XFCE takes up less disk space on my tiny VM config.
  3. I am lazy and once something works and know I can fix the the other way I will wait for someone to really need it before I do
mkdir /etc/ood/config/apps/bc_desktop -p
cat > /etc/ood/config/apps/bc_desktop/my_cluster.yml << EOF
---
title: "Remote Desktop"
cluster: my_cluster
attributes:
  bc_account:
    help: "this should be left blank most of the time"
  desktop: "xfce"
  bc_vnc_idle:
    value: 180
    label: "Idle timeout"
    help: This is the time you have to connect to a session before it is automatically terminated
  node_type: null
form:
  - bc_vnc_idle
  - desktop
  - bc_account
  - bc_num_hours
  - bc_num_slots
  - node_type
  - bc_queue
  - bc_vnc_resolution
  - bc_email_on_started
EOF

Now we reconfigure the portal and restart the webserver to make our changes take effect 

/opt/ood/ood-portal-generator/sbin/update_ood_portal
systemctl try-restart httpd24-httpd.service

Configure the compute node image

luna chroot compute
yum groupinstall xfce
yum install numpy
systemctl disable gdm
exit
luna osimage pack compute

Configure Modules for TurboVNC and websockify

TurboVNC

I cheat and install TurboVNC from the RPM downloaded from the TuboVNC website locally and then copy the /opt/TurboVNC directory into the shared apps folder and then use the following module for the computes 

yum install turbovnc-2.2.2.x86_64.rpm
cp /opt/TurboVNC /trinity/shared/apps/ -r


cat /trinity/shared/modules/tr17.10/x86_64/compiler/turbovnc/2.2.2
#%Module
#
# @name:    TurboVNC
# @version:  2.2.2
# @packaging: BIOS IT
#

# Customize the output of `module help` command
# ---------------------------------------------
proc ModulesHelp { } {
   puts stderr "\tAdds $name to your environment variables"
   puts stderr "\t\t\$PATH, \$MANPATH"
}

# Customize the output of `module whatis` command
# -----------------------------------------------
module-whatis   "loads the [module-info name] environment"

# Define internal modulefile variables (Tcl script use only)
# ----------------------------------------------------------
set   name      TurboVNC
set   version   2.2.2
set   prefix    /trinity/shared/apps/$name/$version

# Check if the path exists before modifying environment
# -----------------------------------------------------
if {![file exists $prefix]} {
   puts stderr "\t[module-info name] Load Error: $prefix does not exist"
   break
   exit 1
}

# Update common variables in the environment
# ------------------------------------------
prepend-path   PATH              $prefix/bin

prepend-path   MANPATH           $prefix/man

setenv         TURBONVNC_DIR     $prefix

Websockify

I install this on the controller and set PYTHONPATH and use the --home= option so that it will end up in the shared apps folder The setup.py will try to install numpy on the node and in Centos 7 this breaks, pre-empt by installing from rpm FIRST 

yum install numpy
git clone https://github.com/novnc/websockify
cd websockify
mkdir -p /trinity/shared/apps/websockify/0.8.0
PYTHONPATH=/trinity/shared/apps/websockify/0.8.0/lib/python python ./setup.py install --home=/trinity/shared/apps/websockify/0.8.0

 cat /trinity/shared/modules/tr17.10/x86_64/libraries/websockify/0.8.0
#%Module
#
# @name:    websockify
# @version:  0.8.0
# @packaging: BIOS-IT
#

# Customize the output of `module help` command
# ---------------------------------------------
proc ModulesHelp { } {
   puts stderr "\tAdds websockify to your environment variables"
   puts stderr "\t\t\$PATH, \$MANPATH"
}

# Customize the output of `module whatis` command
# -----------------------------------------------
module-whatis   "loads the [module-info name] environment"

# Define internal modulefile variables (Tcl script use only)
# ----------------------------------------------------------
set   name      websockify
set   version   0.8.0
set   prefix    /trinity/shared/apps/$name/$version/

# Check if the path exists before modifying environment
# -----------------------------------------------------
if {![file exists $prefix]} {
   puts stderr "\t[module-info name] Load Error: $prefix does not exist"
   break
   exit 1
}

# Update common variables in the environment
# ------------------------------------------
prepend-path   PATH              $prefix/bin

prepend-path   LD_LIBRARY_PATH   $prefix/lib
prepend-path   LIBRARY_PATH      $prefix/lib
prepend-path   PYTHONPATH        $prefix/lib/python

prepend-path   MANPATH           $prefix/share/man

setenv         WEBSOCKIFY_DIR         $prefix
Retrieved from "http://wiki.define-technology.com/mediawiki-1.35.0/index.php?title=OpenOnDemand&oldid=32768"