Advanced Search

Proxy Authentication in Crawler

/ Published in: Python
Save to your folder(s)

Expand | Embed | Plain Text

Copy this code and paste it in your HTML 
  1. # To authenticate the proxy, you must set the Proxy-Authorization header.  You *cannot* use the form http://user:pass@proxy:port in request.meta['proxy']
  2.  
  3. import base64
  4.  
  5. proxy_ip_port = "123.456.789.10:8888"
  6. proxy_user_pass = "awesome:dude"
  7.  
  8. request = Request(url, callback=self.parse)
  9.  
  10. # Set the location of the proxy
  11. request.meta['proxy'] = "http://%s" % proxy_ip_port
  12.  
  13. # setup basic authentication for the proxy
  14. encoded_user_pass=base64.encodestring(proxy_user_pass)
  15. request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass
  16.  
  17. # Snippet imported from snippets.scrapy.org (which no longer works)
  18. # author: redtricycle
  19. # date  : Nov 21, 2011
  20.  

Report this snippet


Comments

RSS Icon Subscribe to comments

You need to login to post a comment.