macOS Big Sur: Python 3.7 Installation with Homebrew

Malte Hellmeier -

If you plan to develop Python on your macOS machine, there are several ways to set it up: Install it directly, use a package manager like Homebrew, use a container or environment system like pipenv or docker and many more.

I really like the idea of the Homebrew package manager and wanted to go this way. I know that environment systems are a better solution, but I often want to execute some lines of Python code without the work overhead of setting up a virtual system. Thus, installing Python with some popular packages  like numpy, pandas or matplotlib suits me best. But during the setup on a clean macOS Big Sur system, I faced some problems. Therefore, I decided to make a step-by-step tutorial to show and fix all upcoming issues.

Pre-requirements

Make sure that your system is up-to-date and you have got Homebrew installed. If not, follow up with their instructions.

Install Python

Using Python 3.9 leads to problems with most of the applications. Thus, installing Python 3.7 should be the right way. We start by opening up a terminal and installing it:

brew install python@3.7

Next, add the PATH to your system. If you run ZSH, add the following lines to your .zshrc:

export PATH="/usr/local/opt/python@3.7/bin:$PATH"
export LDFLAGS="-L/usr/local/opt/python@3.7/lib"
alias pip='pip3'

In the end, link the version with

brew link python@3.7

Reset pip

When working with Python, pip is an important tool. To get rid of old cache data, remove the following folder and recreate it:

rm -rf ~/Library/Caches/pip
mkdir ~/Library/Caches/pip

Install plugins and tools

Install numpy

Installing numpy should be easy, right? I just tried it with pip install numpy (don't worry, we don't have to use pip3 because we defined an alias above) but it failed! It results in the following error message:

RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, see site.cfg.example for information. Otherwise report this to the vendor that provided NumPy.
RankWarning: Polyfit may be poorly conditioned

Uff. The problem is still reported and solutions are discussed in the related GitHub issue. For me, using an older numpy version and openblas solved it. Let's first install numpy version 1.18.0:

pip install numpy==1.18.0

Next, install it with openblas:

pip cache remove numpy
brew install openblas
OPENBLAS="$(brew --prefix openblas)" pip install numpy

That's it!

Install pandas

It's really easy and works as expected:

pip install pandas

Install matplotlib

Similar to numpy, pip install matplotlib fails with the following error:

Collecting matplotlib
  Downloading matplotlib-3.3.3.tar.gz (37.9 MB)
     |████████████████████████████████| 37.9 MB 5.2 MB/s
    ERROR: Command errored out with exit status 1:
     command: /usr/local/opt/python@3.7/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/42/0xlmvl91723_7gwdjwvkhbkc0000gn/T/pip-install-d0kw9jgt/matplotlib/setup.py'"'"'; __file__='"'"'/private/var/folders/42/0xlmvl91723_7gwdjwvkhbkc0000gn/T/pip-install-d0kw9jgt/matplotlib/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/42/0xlmvl91723_7gwdjwvkhbkc0000gn/T/pip-install-d0kw9jgt/matplotlib/pip-egg-info
         cwd: /private/var/folders/42/0xlmvl91723_7gwdjwvkhbkc0000gn/T/pip-install-d0kw9jgt/matplotlib/
    Complete output (62 lines):

    Edit setup.cfg to change the build options; suppress output with --quiet.

    BUILDING MATPLOTLIB
      matplotlib: yes [3.3.3]
          python: yes [3.7.9 (default, Nov 20 2020, 23:58:42)  [Clang 12.0.0
                      (clang-1200.0.32.27)]]
        platform: yes [darwin]
     sample_data: yes [installing]
           tests: no  [skipping due to configuration]
          macosx: yes [installing]

    running egg_info
    creating /private/var/folders/42/0xlmvl91723_7gwdjwvkhbkc0000gn/T/pip-install-d0kw9jgt/matplotlib/pip-egg-info/matplotlib.egg-info
    writing /private/var/folders/42/0xlmvl91723_7gwdjwvkhbkc0000gn/T/pip-install-d0kw9jgt/matplotlib/pip-egg-info/matplotlib.egg-info/PKG-INFO
    writing dependency_links to /private/var/folders/42/0xlmvl91723_7gwdjwvkhbkc0000gn/T/pip-install-d0kw9jgt/matplotlib/pip-egg-info/matplotlib.egg-info/dependency_links.txt
    writing namespace_packages to /private/var/folders/42/0xlmvl91723_7gwdjwvkhbkc0000gn/T/pip-install-d0kw9jgt/matplotlib/pip-egg-info/matplotlib.egg-info/namespace_packages.txt
    writing requirements to /private/var/folders/42/0xlmvl91723_7gwdjwvkhbkc0000gn/T/pip-install-d0kw9jgt/matplotlib/pip-egg-info/matplotlib.egg-info/requires.txt
    writing top-level names to /private/var/folders/42/0xlmvl91723_7gwdjwvkhbkc0000gn/T/pip-install-d0kw9jgt/matplotlib/pip-egg-info/matplotlib.egg-info/top_level.txt
    writing manifest file '/private/var/folders/42/0xlmvl91723_7gwdjwvkhbkc0000gn/T/pip-install-d0kw9jgt/matplotlib/pip-egg-info/matplotlib.egg-info/SOURCES.txt'
    init_dgelsd failed init
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/42/0xlmvl91723_7gwdjwvkhbkc0000gn/T/pip-install-d0kw9jgt/matplotlib/setup.py", line 298, in <module>
        cmdclass=cmdclass,
      File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
        return distutils.core.setup(**attrs)
      File "/usr/local/Cellar/python@3.7/3.7.9_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/local/Cellar/python@3.7/3.7.9_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/local/Cellar/python@3.7/3.7.9_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 297, in run
        self.find_sources()
      File "/usr/local/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 304, in find_sources
        mm.run()
      File "/usr/local/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 535, in run
        self.add_defaults()
      File "/usr/local/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 571, in add_defaults
        sdist.add_defaults(self)
      File "/usr/local/Cellar/python@3.7/3.7.9_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/command/sdist.py", line 228, in add_defaults
        self._add_defaults_ext()
      File "/usr/local/Cellar/python@3.7/3.7.9_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/command/sdist.py", line 311, in _add_defaults_ext
        build_ext = self.get_finalized_command('build_ext')
      File "/usr/local/Cellar/python@3.7/3.7.9_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/cmd.py", line 299, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/usr/local/Cellar/python@3.7/3.7.9_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "/private/var/folders/42/0xlmvl91723_7gwdjwvkhbkc0000gn/T/pip-install-d0kw9jgt/matplotlib/setup.py", line 90, in finalize_options
        for package in good_packages
      File "/private/var/folders/42/0xlmvl91723_7gwdjwvkhbkc0000gn/T/pip-install-d0kw9jgt/matplotlib/setup.py", line 91, in <listcomp>
        for ext in package.get_extensions()
      File "/private/var/folders/42/0xlmvl91723_7gwdjwvkhbkc0000gn/T/pip-install-d0kw9jgt/matplotlib/setupext.py", line 345, in get_extensions
        add_numpy_flags(ext)
      File "/private/var/folders/42/0xlmvl91723_7gwdjwvkhbkc0000gn/T/pip-install-d0kw9jgt/matplotlib/setupext.py", line 469, in add_numpy_flags
        import numpy as np
      File "/usr/local/lib/python3.7/site-packages/numpy/__init__.py", line 286, in <module>
        raise RuntimeError(msg)
    RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, see site.cfg.example for information. Otherwise report this to the vendor that provided NumPy.
    RankWarning: Polyfit may be poorly conditioned

    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Fix it by first upgrading your existing tools:

pip install --upgrade pip setuptools wheel

Install scikit-learn

Works as expected:

pip install -U scikit-learn

Upgrading pip doesn't work with Homebrew

As a developer and security-oriented user, you want to keep your system up-to-date. Sadly, the upgrade benefit from Homebrew doesn't seem to work with pip because it will break if you try it. When using a simple command like pip -v, it will result in the following error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 567, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 884, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 775, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (pip 20.3 (/usr/local/lib/python3.7/site-packages), Requirement.parse('pip==20.0.2'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/opt/python@3.7/bin/pip3", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3238, in <module>
    @_call_aside
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3222, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3251, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 569, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 582, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 770, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pip==20.0.2' distribution was not found and is required by the application

They are also discussed in the related GitHub issues for Homebrew and pip.

To solve it, upgrade pip first. Then, edit the pip version in /usr/local/opt/python@3.7/libexec/bin/pip. That's it!

Now you should have a working development system.

References

  • Header Image Resources: