Mac OS X
Building Scrapy’s dependencies requires the presence of a C compiler and development headers. On OS X this is typically provided by Apple’s Xcode development tools. To install the Xcode command line tools open a terminal window and run:
There’s a known issue that prevents
pip from updating system packages. This has to be addressed to successfully install Scrapy and its dependencies. Here are some proposed solutions:
(Recommended) Don’t use system python, install a new, updated version that doesn’t conflict with the rest of your system. Here’s how to do it using the homebrew package manager:
PATHvariable to state that homebrew packages should be used before system packages (Change
.zshrcaccordantly if you’re using zsh as default shell):
echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc
.bashrcto ensure the changes have taken place:
brew install python
Latest versions of python have
pipbundled with them so you won’t need to install it separately. If this is not the case, upgrade python:
brew update; brew upgrade python
(Optional) Install Scrapy inside an isolated python environment.
This method is a workaround for the above OS X issue, but it’s an overall good practice for managing dependencies and can complement the first method.
virtualenv is a tool you can use to create virtual environments in python. We recommended reading a tutorial like http://docs.python-guide.org/en/latest/dev/virtualenvs/ to get started.
After any of these workarounds you should be able to install Scrapy:
pip install Scrapy