Revision: 68079
Initial Code
Initial URL
Initial Description
Initial Title
Initial Tags
Initial Language
at November 25, 2014 13:30 by zwidnypublic
Initial Code
// File: myscrapy/items.py
..
# -*- coding: utf-8 -*-
import scrapy
class FundItem(scrapy.Item):
stock = scrapy.Field() # 股票åç§°
deadline = scrapy.Field() # æˆªæ¢æ—¥æœŸ
fund = scrapy.Field() # 基金åç§°
code = scrapy.Field() # 基金代ç
count = scrapy.Field() # æŒä»“æ•°é‡(è‚¡)
percentcir = scrapy.Field() # å æµé€šè‚¡æ¯”例(%)
svalue = scrapy.Field() # æŒè‚¡å¸‚值(元)
percentnet = scrapy.Field() # å 净值比例(%)
// File: myscrapy/pipelines.py
..
# -*- coding: utf-8 -*-
from twisted.enterprise import adbapi
from scrapy.utils.project import get_project_settings
settings = get_project_settings()
class FundPipeline(object):
# The table you items.FundItem class map to, my table is named fund
insert_sql = """insert into fund (%s) values ( %s )"""
def __init__(self):
dbargs = settings.get('DB_CONNECT')
db_server = settings.get('DB_SERVER')
dbpool = adbapi.ConnectionPool(db_server, **dbargs)
self.dbpool = dbpool
def __del__(self):
self.dbpool.close()
def process_item(self, item, spider):
self.insert_data(item, self.insert_sql)
return item
def insert_data(self, item, insert):
keys = item.fields.keys()
fields = u','.join(keys)
qm = u','.join([u'%s'] * len(keys))
sql = insert % (fields, qm)
data = [item[k] for k in keys]
return self.dbpool.runOperation(sql, data)
// File: myscrapy/settings.py
..
# -*- coding: utf-8 -*-
...
ITEM_PIPELINES = {
'myscrapy.pipelines.FundPipeline': 500,
}
DB_SERVER = 'MySQLdb' # For detail, please see twisted doc
DB_CONNECT = {
'db': 'finance', # Your db
'user': 'root', #
'passwd': 'root', #
'host': '10.19.80.197', # Your Server
'charset': 'utf8',
'use_unicode': True,
}
...
Initial URL
Initial Description
To store you item into mysql database.
1. I say you have been have mysql database and the table what field is the same in item class that you wang to be stored
2. I just ensure you item insert into mysql, do not ensure the performance
3. Todo
Abstract pipeline class
Initial Title
Scraped item to mysql in scrapy
Initial Tags
mysql
Initial Language
Python