#Thug Distributed Task Queuing
It is the Distributed Version of Thug, which feed tasks into a Queue to be distributed among thug instances. Thug Instances then connect to the Queue and queue automatically distributes tasks among thug instances giving only one task at a time so that Thug Instances doesn't get loaded up with requests. Then Thug Instances process that tasks and give back the results to the Main Server.
##Thug Worker
It is the Thug Worker which waits for Celery to distribute tasks to it, from 2 different types of queues:
- Generic Queue(named: "generic").
- Geolocation Based Queue(named: according to user country code).
Then it processes the assigned task and returns back the result which stores in "redis" backend.
Folder: ThugD
Files:
geolocation.py: It is finding the user country code using its IP address and Team Cymru service.celeryconfig.py: It is the Configuration file of Celery which changes all major settings.main_server.py: It is main file in which Celery instance is declared and dynamically a new queue is made according to user country code.thug_instances.py: It contains the function to be run by thug instances working all over the world.
File: run_tasks.py
It is used to run tasks on remote workers.
Tools/Libraries to install:
sudo apt-get install rabbitmq-server: Works as Message Broker .sudo pip install -U celery: Helps in distributing tasks among running workers.sudo pip install flower: Helps in monitoring and managing remote workers.sudo pip install dnspython: Helps in dns querying to Team Cymru service.
Usage:
celery flower: Runs the flower sever.
Open http://localhost:5555/ in browser to access the tool.celery worker -A ThugD.main_server.thugd -l info -n w1: Running Celery worker in foreground with w1 as hostname. Multiple workers can be run using different hostnames.python run_tasks.py: Running 3 types of tasks(specified in file) on worker.sudo rabbitmqctl list_queues: List active queues. (Optional)
Refer to gh-pages for more information and documentation.