Skip to content

@scrapy Scrapy project

Change the repository type filter

All

    Repositories list

    29 repositories

    • scrapy

      Public
      Scrapy, a fast high-level web crawling & scraping framework for Python.
      pythoncrawlerframework
      pythoncrawlerframeworkscrapingcrawlingweb-scrapinghacktoberfestweb-scraping-python
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •11k•61k•443•191•Updated Apr 23, 2026Apr 23, 2026

    • sphinx-scrapy

      Public
      Sphinx extension for documentation in the Scrapy ecosystem
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •1•1•0•0•Updated Apr 22, 2026Apr 22, 2026

    • scurl

      Public
      Performance-focused replacement for Python urllib
      pythoncythonchromium
      pythoncythonchromiumgurlurlparse
      Python
      •
      Apache License 2.0
      •6•21•10•1•Updated Apr 13, 2026Apr 13, 2026

    • scrapyd

      Public
      A service daemon to run Scrapy spiders
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •579•3.1k•6•0•Updated Apr 8, 2026Apr 8, 2026

    • w3lib

      Public
      Python library of web-related functions
      pythonhacktoberfest
      pythonhacktoberfest
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •108•419•10•5•Updated Mar 20, 2026Mar 20, 2026

    • itemadapter

      Public
      Common interface for data container classes
      pythonmetadatapython3
      pythonmetadatapython3scrapyhacktoberfestpython-dataclassespython-attrs
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •12•69•10•2•Updated Mar 16, 2026Mar 16, 2026

    • scrapyd-client

      Public
      Command line client for Scrapyd server
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •144•774•5•0•Updated Feb 27, 2026Feb 27, 2026

    • scrapy-lint

      Public
      A linter for Scrapy projects.
      linterlinter-pluginscrapy
      linterlinter-pluginscrapyflake8flake8-plugin
      Python
      •
      MIT License
      •4•22•42•0•Updated Feb 25, 2026Feb 25, 2026

    • itemloaders

      Public
      Library to populate items using XPath and CSS with a convenient API
      hacktoberfest
      hacktoberfest
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •16•48•18•4•Updated Jan 29, 2026Jan 29, 2026

    • queuelib

      Public
      Collection of persistent (disk-based) and non-persistent (memory-based) queues for Python
      pythonpython3persistent
      pythonpython3persistenthacktoberfestqueuesnon-persistent
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •55•298•4•2•Updated Jan 29, 2026Jan 29, 2026

    • protego

      Public
      A pure-Python robots.txt parser with support for modern conventions.
      pythonrobots-txthacktoberfest
      pythonrobots-txthacktoberfestrobots-parser
      DIGITAL Command Language
      •
      BSD 3-Clause "New" or "Revised" License
      •29•86•7•0•Updated Jan 29, 2026Jan 29, 2026

    • parsel

      Public
      Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors
      csspythonxml
      csspythonxmlscrapingselectorsxpathlxmlhacktoberfest
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •160•1.3k•31•12•Updated Jan 29, 2026Jan 29, 2026

    • cssselect

      Public
      CSS Selectors for Python
      csspythonselectors
      csspythonselectorshacktoberfest
      Python
      •
      Other
      •63•309•18•4•Updated Jan 29, 2026Jan 29, 2026

    • scrapy-bench

      Public
      A CLI for benchmarking Scrapy.
      pythonweb-crawlerscrapy
      pythonweb-crawlerscrapycommand-line-toolbenchmark-suitescrapy-bench
      Python
      •
      MIT License
      •15•32•6•1•Updated Jun 28, 2025Jun 28, 2025

    • scrapy.org

      Public
      The scrapy.org website
      htmlhacktoberfest
      htmlhacktoberfest
      HTML
      •143•66•1•1•Updated May 8, 2025May 8, 2025

    • form2request

      Public
      Python 3.8+ library to build HTTP requests out of HTML forms
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •3•4•2•0•Updated Mar 21, 2025Mar 21, 2025

    • loginform

      Public
      Fill HTML login forms automatically
      Python
      •85•279•11•3•Updated Apr 24, 2024Apr 24, 2024

    • xtractmime

      Public
      https://mimesniff.spec.whatwg.org/ implementation for Python
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •2•13•0•0•Updated Jan 16, 2024Jan 16, 2024

    • quotesbot

      Public
      This is a sample Scrapy project for educational purposes
      Python
      •
      MIT License
      •783•1.4k•2•7•Updated Nov 29, 2023Nov 29, 2023

    • booksbot

      Public
      A crawler for http://books.toscrape.com
      Python
      •982•42•0•4•Updated Aug 8, 2023Aug 8, 2023

    • scrapely

      Public
      A pure-python HTML screen-scraping library
      HTML
      •274•1.9k•26•6•Updated Apr 4, 2022Apr 4, 2022

    • scrapy-itemloader

      Public archive
      [Archived] Library to populate Scrapy items using XPath and CSS with a convenient API
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •7•7•2•0•Updated May 5, 2020May 5, 2020

    • url-chromium

      Public
      url component from Chromium source code, forked from https://chromium.googlesource.com/chromium/src/url
      chromiumgurl
      chromiumgurl
      C++
      •
      BSD 3-Clause "New" or "Revised" License
      •2•4•0•0•Updated Aug 7, 2018Aug 7, 2018

    • base-chromium

      Public
      base component forked from Chromium source https://chromium.googlesource.com/chromium/src/base/
      C++
      •
      BSD 3-Clause "New" or "Revised" License
      •3•8•0•0•Updated Jul 31, 2018Jul 31, 2018

    • dirbot

      Public
      Scrapy project to scrape public web directories (educational) [DEPRECATED]
      Python
      •1k•1.6k•0•0•Updated Oct 27, 2017Oct 27, 2017

    • scrapy-bench-speedcenter

      Public
      Codespeed for scrapy-bench
      Python
      •2•2•0•0•Updated Aug 28, 2017Aug 28, 2017

    • pypydispatcher

      Public
      A fork of http://pydispatcher.sourceforge.net/ with PyPy support
      Python
      •
      Other
      •4•16•1•0•Updated Jul 3, 2017Jul 3, 2017

    • slybot

      Public
      55•223•5•0•Updated Apr 27, 2015Apr 27, 2015

    • gsoc2014-integration-tests

      Public
      GSoC2014 - Scrapy Integration tests project
      Shell
      •3•3•0•0•Updated Mar 18, 2014Mar 18, 2014
    ProTip! When viewing an organization's repositories, you can use the props. filter to filter by custom property.