Learning about Python's virtual environments is critical if you want to be more confident as a Python programmer. It is one of the concepts that is mostly overlooked.
My Python programming journey started in a similar way. When I started with Python programming back in 2017, I was just using the system's main Python to do all of my work. It did not bother me much because I was not doing many projects. Later I moved to PyCharm for Python and it would create virtual environments automatically for me every time I created a new project. That's when I learned what a virtual environment is. To those who are not familiar with it, it is a Python environment, and libraries, interpreters, and scripts installed on it are isolated from other Python environments.
One of the main reasons you would want to use a virtual environment is that it isolates your project dependencies. If you install every package into the same Python environment, you will probably mess up your Python environment and it might not behave as desired. For example, one of your projects might use TensorFlow version 1 and another might use TensorFlow version 2. Obviously, you cannot install both versions in one environment so you end up installing each version every time you switch projects. TensorFlow depends upon other packages and might require their specific versions.
The vanilla way to use Python's virtual environment is to use virtualenv, a module native to Python. It is one of the easiest ways to install a virtual environment, but it can get unmanageable. You usually install a virtual environment in your project directory like this.
pip3 install virtualenv
virtualenv venv
It creates a folder named venv (you can name it anything). Inside this folder, a separate Python environment is created. It looks something like this:
The way you activate this Python environment is by :
>>source venv/bin/activate
(venv)>>which python
(venv)>>/home/user/venv/bin/python
You will know you are using the virtual environment called venv, because you will see (venv) at the beginning of the terminal line. If you are really skeptical and want to know whether this particular Python is being used you can run the command which python. It will show you the path of the Python that is being used.
You can deactivate it by running the following command:
(venv)>>deactivate
>>which python
>>/usr/bin/python
If you run which python command again after running the deactivate command. You will be pointed to the system's main Python. This is your system's main Python and you probably do not want to install everything here. That is the major point of this post!
With virtual environments, it is also easier to manage multiple versions of Python in your system. Even though new versions of Python are released, some applications, libraries, and frameworks still work on older versions of Python. When you want to use different frameworks that require different versions of Python, you can just use virtual environments.
When you run the above commands to create a virtual environment, you normally create a virtual environment using the system's main python (i.e. the python that is in your system's path). If you want to create a virtual environment with a different Python version, make sure that you install the specific Python version from here: https://www.python.org/downloads/. Note the path where the Python is being installed.
To install a specific version of Python in the virtual environment you can use the --python flag following the virtualenv command:
>>virtualenv venv --python=/usr/bin/python3.9
Virtual environments make your life much easier and let you have more control over Python's environments. It also teaches you how Python works as well.