上一篇【網路爬蟲】臺灣證券交易所歷史資料教學(1) 提到如何從證交所網站剖析連接後，建立了抓取歷史股價的基本爬蟲，但是證交所網頁比較麻煩的地方在於每一次只能抓一個月的資料，所以必須要回溯過去撈取，再將資料拼湊起來。
創建日期序列
import pandas as pd
Dates = pd.date_range(start = '2000-01-01', end = '2020-09-01', freq = 'MS')Output：
DatetimeIndex(['2000-01-01', '2000-02-01', '2000-03-01', '2000-04-01',
'2000-05-01', '2000-06-01', '2000-07-01', '2000-08-01',
'2000-09-01', '2000-10-01',
...
'2019-12-01', '2020-01-01', '2020-02-01', '2020-03-01',
'2020-04-01', '2020-05-01', '2020-06-01', '2020-07-01',
'2020-08-01', '2020-09-01'],
dtype='datetime64[ns]', length=249, freq='MS')由於帶入網址時必須帶入字串格式的日期，所以先將日期序列格式轉為文字，而且格式為yyyymmdd
Dates = Dates.astype(str)Output：
Index(['2000-01-01', '2000-02-01', '2000-03-01', '2000-04-01', '2000-05-01',
'2000-06-01', '2000-07-01', '2000-08-01', '2000-09-01', '2000-10-01',
...
'2019-12-01', '2020-01-01', '2020-02-01', '2020-03-01', '2020-04-01',
'2020-05-01', '2020-06-01', '2020-07-01', '2020-08-01', '2020-09-01'],
dtype='object', length=249)帶入上次寫好的Get_StockPrice()函數中
產生各個日期的字串後，必須要把字串中間的「-」消除，所以使用replace(‘-‘,”)，用空白來取代減號；最後我們利用For迴圈來抓取證交所的網站，最後加一個time.sleep()就是要讓程式稍微停一秒鐘，讓對方比較不會感覺我們再用爬蟲程式攻擊網頁。
Symbol = '2330'
Dates = pd.date_range(start = '2000-01-01', end = '2020-09-01', freq = 'MS').astype(str)
for Date in Dates:
print(Get_StockPrice(Symbol, Date.replace('-','')))
time.sleep() 完整程式碼
import pandas as pd
import numpy as np
import json
import requests
import datetime
import time
def Get_StockPrice(Symbol, Date):
url = f'https://www.twse.com.tw/exchangeReport/STOCK_DAY?response=json&date={Date}&stockNo={Symbol}'
print(url)
data = requests.get(url).text
json_data = json.loads(data)
Stock_data = json_data['data']
StockPrice = pd.DataFrame(Stock_data, columns = ['Date','Volume','Volume_Cash','Open','High','Low','Close','Change','Order'])
StockPrice['Date'] = StockPrice['Date'].str.replace('/','').astype(int) + 19110000
StockPrice['Date'] = pd.to_datetime(StockPrice['Date'].astype(str))
StockPrice['Volume'] = StockPrice['Volume'].str.replace(',','').astype(float)/1000
StockPrice['Volume_Cash'] = StockPrice['Volume_Cash'].str.replace(',','').astype(float)
StockPrice['Order'] = StockPrice['Order'].str.replace(',','').astype(float)
StockPrice['Open'] = StockPrice['Open'].str.replace(',','').astype(float)
StockPrice['High'] = StockPrice['High'].str.replace(',','').astype(float)
StockPrice['Low'] = StockPrice['Low'].str.replace(',','').astype(float)
StockPrice['Close'] = StockPrice['Close'].str.replace(',','').astype(float)
StockPrice = StockPrice.set_index('Date', drop = True)
StockPrice = StockPrice[['Open','High','Low','Close','Volume']]
print(StockPrice)
return StockPrice
if __name__ == '__main__':
Symbol = '2330'
Dates = pd.date_range(start = '2010-01-01', end = '2020-09-01', freq = 'MS').astype(str)
data = Get_StockPrice(Symbol, Dates[0].replace('-',''))
for Date in Dates[1:]:
print(Date)
try:
data = pd.concat([data,Get_StockPrice(Symbol, Date.replace('-',''))], axis = 0)
time.sleep(5)
except:
pass
print(data)
