mobile wallpaper 1mobile wallpaper 2mobile wallpaper 3mobile wallpaper 4
51 words
1 minute
Running pyspider on Windows 11 with Docker
2024-01-02

There were issues installing pyspider on Windows 11, with multiple errors occurring.

I found that the official website offers a Docker-based installation method.

Directly via Docker#

# mysql
docker run --name mysql -d -v /data/mysql:/var/lib/mysql -e MYSQL_ALLOW_EMPTY_PASSWORD=yes mysql:latest
# rabbitmq
docker run --name rabbitmq -d rabbitmq:latest
# phantomjs
docker run --name phantomjs -d binux/pyspider:latest phantomjs
# result worker
docker run --name result_worker -m 128m -d --link mysql:mysql --link rabbitmq:rabbitmq binux/pyspider:latest result_worker
# processor, run multiple instance if needed.
docker run --name processor -m 256m -d --link mysql:mysql --link rabbitmq:rabbitmq binux/pyspider:latest processor
# fetcher, run multiple instance if needed.
docker run --name fetcher -m 256m -d --link phantomjs:phantomjs --link rabbitmq:rabbitmq binux/pyspider:latest fetcher --no-xmlrpc
# scheduler
docker run --name scheduler -d --link mysql:mysql --link rabbitmq:rabbitmq binux/pyspider:latest scheduler
# webui
docker run --name webui -m 256m -d -p 5000:5000 --link mysql:mysql --link rabbitmq:rabbitmq --link scheduler:scheduler --link phantomjs:phantomjs binux/pyspider:latest webui

Using docker-compose#

services:
phantomjs:
image: binux/pyspider:latest
command: phantomjs
result:
image: binux/pyspider:latest
external_links:
- mysql
- rabbitmq
command: result_worker
processor:
image: binux/pyspider:latest
external_links:
- mysql
- rabbitmq
command: processor
fetcher:
image: binux/pyspider:latest
external_links:
- rabbitmq
links:
- phantomjs
command : fetcher
scheduler:
image: binux/pyspider:latest
external_links:
- mysql
- rabbitmq
command: scheduler
webui:
image: binux/pyspider:latest
external_links:
- mysql
- rabbitmq
links:
- scheduler
- phantomjs
command: webui
ports:
- "5000:5000"

Then just run: docker-compose up -d After running successfully, if you visit http://localhost<5000>/ and see the content below, it indicates that pyspider is running successfully.

202401022235683.png

Share

If this article helped you, please share it with others!

Running pyspider on Windows 11 with Docker
https://dreaife.tokyo/en/posts/docker-pyspider-win/
Author
dreaife
Published at
2024-01-02
License
CC BY-NC-SA 4.0

Some information may be outdated

Related Posts Smart
1
Python Web Crawler Environment Setup
spider Setting up a Python web crawler environment includes installing Python 3, request libraries (such as requests and selenium), parsing libraries (such as lxml and beautifulsoup4), databases (such as MySQL and MongoDB), storage libraries (such as PyMySQL and PyMongo), web libraries (such as Flask and Tornado), app crawling tools (such as mitmproxy and appium), and crawler frameworks (such as pyspider and scrapy). Installation commands and notes for each library are provided in detail.
2
Getting Started with Docker
infra Docker is a technology for solving microservice deployment problems by packaging applications and their dependencies into isolated containers, avoiding inconsistent environments and dependency conflicts. Compared with virtual machines, Docker starts faster and uses fewer resources. Its architecture includes images and containers, and users can share and obtain images through Docker Hub. Basic operations include creating and managing images and containers and using volumes for data persistence and host-container decoupling. Docker Compose can simplify distributed application deployment.
3
A Summary of My Thoughts on the Market on X
market Drawing on complex systems and game theory, this post unpacks the underlying logic of market dynamics, emotional reflexivity, and the consensus underpinning monetary value, helping readers look beyond short-term volatility and develop a deeper understanding of market expectations and inflation transmission.
4
Basic On-Chain Operations for EOA Wallets
WEB3 Quickly master the core on-chain operations of Web3 wallets. Covers the principles behind creating EOA and HD wallets, mnemonic generation and the BIP derivation process, an in-depth look at SIWE and EIP-712 on-chain verification, and the full lifecycle of transaction construction, signing, and broadcasting to help you understand the underlying logic of wallets.
5
About an EOA Wallet Signature Verification and Related Content
WEB3 A developer-focused guide to EOA wallet signature verification, covering secp256k1, ECDSA r/s/v signatures, public key recovery, keccak-256 address derivation, and how SIWE proves wallet ownership without exposing the private key.

Table of Contents