Sunday, 8 January 2017

Installing Scrapy in Windows 7 64-bit

It's been a long time that I want to try out some kind of program to crawl websites and extract info/data from them. Recently, I've tried powershell which had a decent outcome. However, it was not easy as the language was designed for some other purpose. Thus, I decided to check scarpy out.

The following would be the steps to install Scrapy in Windows 7 64-bit.


  1. It starts with the installation of Microsoft Visual C++ Compiler for Python 2.7 which is downloaded from Microsoft.
  2. Install Python 2.7.13 (using python-2.7.13.amd64.msi) which is downloaded from python.org.
  3.  Then install Python for Windows Extensions (using pywin32-220.win-amd64-py2.7.exe). 
  4. Once the installation is completed, the folder path that python is installed must be added to PATH environment variable in Windows.
  5. Check the installation and path added by checking python and pip versions in command prompt.
    python --version
    pip --version
  6. Once python and pip are displaying versions correctly, scrapy can be installed using
    pip install scrapy
  7. For me, the installation faces some issue when it looks for libxml related libraries.

        c:\users\admin\appdata\local\temp\xmlXPathInit_h2bpm.c(1) : fatal error C108
    3: Cannot open include file: 'libxml/xpath.h': No such file or directory
        ****************************************************************************
    *****
        Could not find function xmlCheckVersion in library libxml2. Is libxml2 insta
    lled?
        ****************************************************************************
    *****
        error: command 'C:\\Users\\admin\\AppData\\Local\\Programs\\Common\\Microsof
    t\\Visual C++ for Python\\9.0\\VC\\Bin\\amd64\\cl.exe' failed with exit status 2

        ----------------------------------------
    Command "c:\python27\python.exe -u -c "import setuptools, tokenize;__file__='c:\
    \users\\admin\\appdata\\local\\temp\\pip-build-g2zcer\\lxml\\setup.py';f=getattr
    (tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close()
    ;exec(compile(code, __file__, 'exec'))" install --record c:\users\admin\appdata\
    local\temp\pip-erwjed-record\install-record.txt --single-version-externally-mana
    ged --compile" failed with error code 1 in c:\users\admin\appdata\local\temp\pip
    -build-g2zcer\lxml\
  8. To resolve the above, steps in stackoverflow.com is followed. It has been suggested to download lxml-3.6.4-cp27-cp27m-win_amd64.whl and install it using this command.
    pip install lxml-3.6.4-cp27-cp27m-win_amd64.whl
  9. Once it is successfully installed, re-run pip install scrapy again. It should be successful now.
  10. Finally, tutorial from scrapy.org is run with results.

Hope it helps to install scrapy in Windows 7 64-bit.


No comments:

Post a Comment