Using a your laptop internet connection on a remote cluster
Many time you have no internet on a cluster install but you need to install rpms.
If you have internet access on your laptop and can ssh to the cluster even via a gateway node you do not controll you can do this using nothing but SSH
Short Version (Socks Proxy)
For when you have a VM/instance/node with internet access and the system your on doesn't have any
# build-server has internet
ssh build-server -D8888
# then on system you ssh from
# in yum.conf
proxy=socks5h://127.0.0.1:8888Preparing your laptop/proxy host
on your machine edit /etc/ssh/sshd_config add
GatewayPorts yes
restart sshd
this allows remote access to the proxy port
if you want to share the proxy at the OTHER end you also need to do the same (in this case it would be required on 10.11.0.5 if you want the compute nodes to use the proxy too
SSH session 1 to create your proxy
This creates a port forward from localhost:1337 to the remote host (nas2) could be localhost if your laptop has internet access. nas2 is a machine I can directly connect to with internet access
antony@dell-laptop:~$ ssh -D 1337 -q -C -N nas2
Optional SSH session 2 to directly access your cluster head node from your laptop
This is only needed if you cannot ssh directly into the head node and this could be a more complex setup involving multiple chained or nested tunnels if needed. Here I forward 2323 to the head node ssh port (10.11.0.5:22)
antony@dell-laptop:~$ ssh -i shanal -L2323:10.11.0.5:22 ubuntu@bios-landing.sanger.ac.uk
Final SSH session to create the reverse tunnel
this creates a tunnel from the node you want internet access on to the proxy you created in step 1 so it forwards 8080 on head node to port 1337 on my laptop
antony@dell-laptop:~$ ssh -i ~/.ssh/sangerkey -R8080:localhost:1337 root@localhost -p 2323
inside this session we can then configure yum to use my proxy like so:
configure yum to use the proxy server edit /etc/yum.conf and add the following line:
proxy=socks5h://127.0.0.1:8080
clean the yum caches and see if you can access the repos
yum clean all [root@head ~]# yum repolist Loaded plugins: fastestmirror Determining fastest mirrors epel/x86_64/metalink | 22 kB 00:00 * base: anorien.csc.warwick.ac.uk * epel: epel.mirror.far.fi * extras: centos.mirroring.pulsant.co.uk * updates: anorien.csc.warwick.ac.uk OpenHPC | 1.6 kB 00:00 OpenHPC-updates | 1.2 kB 00:00 base | 3.6 kB 00:00 epel | 5.4 kB 00:00 extras | 2.9 kB 00:00 updates | 2.9 kB 00:00 (1/10): OpenHPC/group_gz | 1.7 kB 00:01 (2/10): base/7/x86_64/group_gz | 165 kB 00:01 (3/10): OpenHPC/primary | 155 kB 00:03 (4/10): base/7/x86_64/primary_db | 6.0 MB 00:03 (5/10): epel/x86_64/group_gz | 88 kB 00:02 (6/10): epel/x86_64/updateinfo | 1.0 MB 00:00 (7/10): OpenHPC-updates/primary | 332 kB 00:04 (8/10): extras/7/x86_64/primary_db | 152 kB 00:00 (9/10): updates/7/x86_64/primary_db | 1.1 MB 00:01 (10/10): epel/x86_64/primary_db | 6.9 MB 00:04 OpenHPC 821/821 OpenHPC-updates 1866/1866 repo id repo name status OpenHPC OpenHPC-1.3 - Base 821 OpenHPC-updates OpenHPC-1.3 - Updates 1,866 base/7/x86_64 CentOS-7 - Base 10,097 epel/x86_64 Extra Packages for Enterprise Linux 7 - x86_64 13,411 extras/7/x86_64 CentOS-7 - Extras 304 updates/7/x86_64 CentOS-7 - Updates 311 repolist: 26,810 [root@head ~]#