How do I feed in my own data into PyAlgoTrade?

Question

I'm trying to use PyAlogoTrade's event profiler

However I don't want to use data from yahoo!finance, I want to use my own but can't figure out how to parse in the CSV, it is in the format:

Timestamp      Low  Open   Close       High        BTC_vol     USD_vol       [8]      [9]
2013-11-23 00  800  860    847.666666  886.876543  853.833333   6195.334452  5248330  0
2013-11-24 00  745  847.5  815.01      860         831.255     10785.94131   8680720  0

The complete CSV is here

I want to do something like:

def main(plot):
    instruments = ["AA", "AES", "AIG"]
    feed = yahoofinance.build_feed(instruments, 2008, 2009, ".")

Then replace yahoofinance.build_feed(instruments, 2008, 2009, ".") with my CSV

I tried:

import csv

with open( 'FinexBTCDaily.csv', 'rb' ) as csvfile:
     data = csv.reader( csvfile )

def main( plot ):
    feed = data

But it throws an attribute error. Any ideas how to do this?

Mayle

Answer 1

Step A: follow PyAlgoTrade doc on GenericBarFeed class

On this link see the addBarsFromCSV() in CSV section of the BarFeed class in v0.16

On this link see the addBarsFromCSV() in CSV section of the BarFeed class in v0.17

Note

- The CSV file must have the column names in the first row.
- It is ok if the Adj Close column is empty.
- When working with multiple instruments:
--- If all the instruments loaded are in the same timezone, then the timezone parameter may not be specified.
--- If any of the instruments loaded are in different timezones, then the timezone parameter should be set.

addBarsFromCSV( instrument, path, timezone = None )
Loads bars for a given instrument from a CSV formatted file. The instrument gets registered in the bar feed.

Parameters:
(string) instrument – Instrument identifier.
(string) path – The path to the CSV file.
(pytz) timezone – The timezone to use to localize bars.
Check pyalgotrade.marketsession.

---

Next:
A BarFeed loads bars from CSV files that have the following format:

Date       Time,    Open,    High, Low,  Close,   Volume,      Adj Close
2013-01-01 13:59:00,13.51001,13.56,13.51,13.56789,273.88014126,13.51001

Step B: implement a documented CSV-file pre-formatting

Your CSV data will need a bit of sanity ( before will be able to be used in PyAlgoTrade methods ),
however it is doable and you can create an easy transformator either by hand or with a use of a powerful numpy.genfromtxt() lambda-based converters facilities.

This sample code is intended for an illustration purpose, to see immediately the powers of converters for your own transformations, as CSV-structure differs.

with  open( getCsvFileNAME( ... ), "r" ) as aFH:
                         numpy.genfromtxt(  aFH,
                                            skip_header     = 1,    # Ref. pyalgotrade
                                            delimiter       = ",",
                                            #                                v     v       v       v       v       v 
                                            #                       2011.08.30,12:00,1791.20,1792.60,1787.60,1789.60,835
                                            #                       2011.08.30,13:00,1789.70,1794.30,1788.70,1792.60,550
                                            #                       2011.08.30,14:00,1792.70,1816.70,1790.20,1812.10,1222
                                            #                       2011.08.30,15:00,1812.20,1831.50,1811.90,1824.70,2373
                                            #                       2011.08.30,16:00,1824.80,1828.10,1813.70,1817.90,2215
                                            converters      = { 0:  lambda aString: mPlotDATEs.date2num( datetime.datetime.strptime( aString, "%Y.%m.%d" ) ),       #_______________________________________asFloat ( 1.0, +++ )
                                                                1:  lambda aString: ( ( int( aString[0:2] ) * 60 + int( aString[3:] ) ) / 60. / 24. )               #           ( 15*60 + 00 ) / 60. / 24.__asFloat < 0.0, 1.0 )
                                                                #                                    HH:                       :MM                                                HH      MM
                                                                }
                                            )

Gullett

Answer 2

I suggest to create your own Rowparser and Feed, which is much easier than it sounds, have a look here: yahoofeed

This also allows you to work with intraday data and cleanup the data if needed, like your timestamp.

Another possibility, of course, would be to parse your file and save it, so it looks like a yahoo feed. In your case, you would have to adapt the columns and the Timestamp.

Pentahedron

Answer 3

You can use pyalgotrade.barfeed.addBarsFromSequence with list comprehension to feed in data from CSV row by row/bar by bar. Basically you create a bar from each row, pass OHLCV as init parameters and extra columns with additional data in a dictionary. You can try something like this (with all the required imports):

data = pd.DataFrame(index=pd.date_range(start='2021-11-01', end='2021-11-05'), columns=['Open', 'High', 'Low', 'Close', 'Adj Close', 'Volume', 'ExtraCol1', 'ExtraCol3', 'ExtraCol4', 'ExtraCol5'], data=np.random.rand(5, 10))
feed = yahoofeed.Feed()
feed.addBarsFromSequence('instrumentID', data.index.map(lambda i:
        BasicBar(
            i,      
            data.loc[i, 'Open'],
            data.loc[i, 'High'],            
            data.loc[i, 'Low'],             
            data.loc[i, 'Close'],           
            data.loc[i, 'Volume'],          
            data.loc[i, 'Adj Close'],           
            Frequency.DAY,        
            data.loc[i, 'ExtraCol1':].to_dict())    
).values)

The input data frame was created with random values to make this example easier to reproduce, but the part where the bars are added to the feed should work the same for data frames from CSVs given that the valid column names are used.

Carmarthenshire