HOWTO install a newer (or older) version of Python

sean

Note: the following Python versions are pre-installed on all Opalstack servers: 2.7, 3.6, 3.10, 3.11, 3.12
2023-11-28: updated to include OpenSSL build instructions.

You can install your own Python from source while in a SSH session as follows, just adjust the PYVER variable for the version you need:

# prep
mkdir -p ~/src ~/tmp ~/opt
export TMPDIR=~/tmp

# build openssl
cd ~/src
export OPENSSL_VER=1.1.1s
wget https://www.openssl.org/source/openssl-$OPENSSL_VER.tar.gz
tar -xvf openssl-$OPENSSL_VER.tar.gz
rm -rf openssl-$OPENSSL_VER.tar.gz
mv openssl-$OPENSSL_VER openssl
cd openssl
./config --prefix=$HOME/opt/ssl --openssldir=$HOME/opt/ssl
make
make install

# build python
cd ~/src
export PYVER=3.9.0
wget https://www.python.org/ftp/python/$PYVER/Python-$PYVER.tgz
tar zxf Python-$PYVER.tgz
cd Python-$PYVER
./configure --prefix=$HOME/opt --with-openssl=$HOME/opt/ssl --with-openssl-rpath=auto
make
make install

When that's done you'll have your new Python available at ~/opt/bin/pythonX.Y with its libraries available in ~/opt/lib/pythonX.Y.

That will work for reasonably recent versions of Python, ie 2.7+. If you need something older then you can install patched versions of Python with pyenv eg:

# prep
mkdir -p ~/src ~/tmp ~/opt
export TMPDIR=~/tmp

# build openssl
cd ~/src
export OPENSSL_VER=1.1.1s
wget https://www.openssl.org/source/openssl-$OPENSSL_VER.tar.gz
tar -xvf openssl-$OPENSSL_VER.tar.gz
rm -rf openssl-$OPENSSL_VER.tar.gz
mv openssl-$OPENSSL_VER openssl
cd openssl
./config --prefix=$HOME/opt/ssl --openssldir=$HOME/opt/ssl
make
make install

# install pyenv
git clone https://github.com/pyenv/pyenv.git ~/.pyenv
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile
echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init -)"\nfi' >> ~/.bash_profile
exec "$SHELL"

# install python
env CONFIGURE_OPTS="-with-openssl=$HOME/opt/ssl -with-openssl-rpath=auto" pyenv install 2.5.5
export PYENV_VERSION=2.5.5

In either case in order to use the new Python with a uwsgi or Django app installed via your dashboard, you'll need to replace the installed env virtualenv directory with a new one created with your self-installed Python and then install uwsgi and whatever other packages you need into that, eg:

cd ~/apps/appname
mv env env.old
~/opt/bin/python3.9 -m venv env
source env/bin/activate
scl enable devtoolset-11 -- pip install uwsgi django

karol

sean When I follow this instruction and create an env with the bespoke python version, the pip inside the env is not tied to the python I just downloaded. If I try python -m pip --version it says pip cannot be found. Any ideas?

sasa_buklijas

I understand that duo to WF migration of costumers you are very busy, but I would appreciate if up to date Python would be available in future.
It is 90% of my use case.

sean

sasa_buklijas we're using the newest Python3 that's available in our CentOS7 OS package repositories.

In 2021 we'll be rolling out new machines on CentOS 8 which should have a newer version.

ppo

FYI, I created a shell script to do that (and a bit more).

https://github.com/ppo/opalstack

dcd

ppo app-template-py script works great - thanks!! Looks like it's only configured for python template though -- messes with the django app template (removes uwsgi, start, stop, etc.) Thanks a ton for coding this up -- huge time saver.

ngkabra

Thanks for this: this is an important use case.

One feature request: could you install new versions of python system-wide (as /opt/local/pythonX.Y). We could still do the env updates ourselves.

ngkabra

I've been trying this with python3.10.0rc1 and having trouble. The python installation succeeds but afterwards I am unable to install anything with pip because it can't find openssl. Anyone know what step I'm missing?

The ./configure is finding openssl correctly

./configure --prefix=$HOME/opt 2>&1 | tee /tmp/conf.log
grep -i ssl /tmp/conf.log 
    checking whether compiling and linking against OpenSSL works... yes
    checking for --with-openssl-rpath... 
    checking for --with-ssl-default-suites... python

But after make && make install when I run pip I get this error:

~/opt/bin/pip3.10 install -U pip

I get this error:

WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping

What should I be doing in addition?

sean

ngkabra I tried Python-3.10.0rc1 and the make step gave the following warning at the end:

Could not build the ssl module!
Python requires a OpenSSL 1.1.1 or newer

It's happening because the OpenSSL in CentOS7 is too old.

To work around that you'll need to build your own OpenSSL from source and then use the various "with-openssl" configure options to point your install at that.

ngkabra

sean Thanks Sean,
I built my own openssl. If anyone else runs into this problem, this is what I did (Note: replace navin with your own username)

$ mkdir src && cd src && wget <openssl-url> && cd <openssl-directory>
$ ./config --prefix=/home/navin/opt/ssl --openssldir=/home/navin/opt/ssl && make && make install

Now install python:

$ cd src && wget <python-url> && cd <python-directory>
$ ./configure --prefix=/home/navin/opt --with-openssl=/home/navin/opt/ssl --with-openssl-rpath=auto && make && make install

This made everything work.

ngkabra

Also, if anyone is experimenting with python3.10, don't bother yet. As far as I can tell uwsgi does not yet support python3.10. As a result, although I got python3.10 working successfully, I couldn't actually get my app running because no uwsgi.

sean

ngkabra I've tested this and had no problems installing and running uwsgi on Python 3.10.

sean

karol pip should be installed by default on any new virtual environment.

If it's not working then first check to ensure that you have the correct environment activated, then run python -m ensurepip upgrade to install pip.

karol

sean

It doesn't seem to work the way you say. This env was created earlier using python 3.3.7. I want it because it's the highest version Django 1.6 supports.

[karol@opal4 apps]$ cd django_16_python3/
[karol@opal4 django_16_python3]$ source env/bin/activate 
(env) [karol@opal4 django_16_python3]$ python --version
Python 3.3.7
(env) [karol@opal4 django_16_python3]$ pip --version 
pip 21.3.1 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)
(env) [karol@opal4 django_16_python3]$ # not what I hoped for ^
(env) [karol@opal4 django_16_python3]$ python -m ensurepip upgrade
/home/karol/apps/django_16_python3/env/bin/python: No module named ensurepip

sean

karol in that case please try to install pip by following the get-pip.py instructions at:

https://pip.pypa.io/en/stable/installation/#get-pip-py

etienneh

In case anyone wants compressed instructions, here's some using pyenv. There's been no new version of 1.1 since 2018 so these commands should be stable and portable across users.

mkdir -p $HOME/opt/
mkdir -p $HOME/tmp/pyenv-builds/

cd ~/tmp
export OPENSSL_VER=1.1.1s
wget https://www.openssl.org/source/openssl-$OPENSSL_VER.tar.gz
tar -xvf openssl-$OPENSSL_VER.tar.gz
rm -rf openssl-$OPENSSL_VER.tar.gz
mv openssl-$OPENSSL_VER openssl
cd openssl

./config --prefix=$HOME/opt/ssl --openssldir=$HOME/opt/ssl
make
make install

env TMPDIR=$HOME/tmp/pyenv-builds CONFIGURE_OPTS="-with-openssl=$HOME/opt/ssl -with-openssl-rpath=auto" pyenv install 3.10.X