Advanced Search

scrapy-service.tac (Twisted Application Framework)

/ Published in: Python
Save to your folder(s)

Expand | Embed | Plain Text

Copy this code and paste it in your HTML 
  1. # This is a tac file that can be used for launching Scrapy as a service (in the [Twisted Application Framework](http://twistedmatrix.com/documents/current/core/howto/application.html) ) using twistd.
  2. # 
  3. # You can start the service with:
  4. # 
  5. #     twistd -ny scrapy-service.tac
  6. # 
  7. # And then schedule spiders with:
  8. # 
  9. #     scrapy queue add myspider
  10.  
  11. from twisted.application.service import Service, Application
  12. from twisted.python import log as txlog
  13.  
  14. from scrapy import log
  15. from scrapy.crawler import Crawler
  16. from scrapy.conf import settings
  17.  
  18. class CrawlerService(Service):
  19.  
  20.     def startService(self):
  21.         settings.overrides['QUEUE_CLASS'] = settings['SERVER_QUEUE_CLASS']
  22.         self.crawler = Crawler(settings)
  23.         self.crawler.install()
  24.         self.crawler.start()
  25.  
  26.     def stopService(self):
  27.         return self.crawler.stop()
  28.  
  29. def get_application(logfile, loglevel=log.DEBUG):
  30.     app = Application("Scrapy")
  31.     app.setComponent(txlog.ILogObserver, \
  32.         log.ScrapyFileLogObserver(open(logfile, 'a'), loglevel).emit)
  33.     CrawlerService().setServiceParent(app)
  34.     return app
  35.  
  36. application = get_application('scrapy.log')
  37.  
  38. # Snippet imported from snippets.scrapy.org (which no longer works)
  39. # author: pablo
  40. # date  : Aug 26, 2010
  41.  

Report this snippet


Comments

RSS Icon Subscribe to comments

You need to login to post a comment.