is backwards-compatible with the previous format.
In USEquityLoader, use dailyreader's trading_calendar.
This is backwards compatible and will fall back to the NYSE calendar if
the reader doesn’t have a calendar specified.
Instead of having separate ExchangeCalendar and TradingSchedule objects, we
now just have TradingCalendar. The TradingCalendar keeps track of each
session (defined as a contiguous set of minutes between an open and a close).
It's also responsible for handling the grouping logic of any given minute
to its containing session, or the next/previous session if it's not a market
minute for the given calendar.
Limit the date range to one day, instead of a year.
On my local machine the following goes from taking 12 seconds to 0.6 seconds.
`$ nosetests -x tests/test_algorithm.py:TestMiscellaneousAPI`
In preparation of adding futures, add equity to the names of both the
classes and methods for writing bcolz data. Futures data will use a
different minutes per day with a separate reader. This change will allow
both equity and futures fixtures to be side by side.
Also, break out the method which generates the dataframes and trading
days member into fixtures (`EquityMinuteBarData` and
`EquityDailyBarData`) on which the `*BarReader` fixture depends. This
fixture is separated out to enable reader/writers in different formats
to use the same data setup. (There is internal code which needs to write
minute and daily bar data in a database format.)
The `DataSource` class and other classes derived from it are no longer
used. Instead `DataPortal` and various `MinuteBarReader` and
`DailyBarReaders` should be used.
Instead of inferring it from the minute/daily writer, we now require the
first trading day to be passed explicitly, so the creator of the
DataPortal controls what is used as the first trading day.
Refactor AlgorithmSimulator so that DAY_END is emitted for both
minute and daily emission, and that handling of end-of-minute
and end-of-day are separated
Updates the BcolzMinuteBarWriter.write api to allow users to pass their
data as a stream instead of requiring that they loop over their data
externally. This matches the API presented by BcolzDailyBarWriter.
Previously, whenever we try to access a missing value on the Positions
dict, we return a default Position and save it to the dict. Instead,
just return the Position
Changes BcolzDailyBarWriter to not be an abc, data is passed as an
iterator of (sid, dataframe) pairs to the write method.
Changes the AssetsDBWriter to be a single class which accepts an engine
at construction time and has a `write` method for writing dataframes for
the various tables. We no longer support writing the various other data
types, callers should coerce their data into a dataframe themselves. See
zipline.assets.synthetic for some helpers to do this.
Adds many new fixtures and updates some existing fixtures to use the new
ones:
WithDefaultDateBounds
A fixture that provides the suite a START_DATE and END_DATE. This is
meant to make it easy for other fixtures to synchronize their date
ranges without depending on eachother in strange ways. For example,
WithBcolzMinuteBarReader and WithBcolzDailyBarReader by default should
both have data for the same dates, so they may use depend on
WithDefaultDates without forcing a dependency between them.
WithTmpDir, WithInstanceTmpDir
Provides the suite or individual test case a temporary directory.
WithBcolzDailyBarReader
Provides the suite a BcolzDailyBarReader which reads from bcolz data
written to a temporary directory. The data will be read from
dataframes and then converted to bcolz files with
BcolzDailyBarWriter.write
WithBcolzDailyBarReaderFromCSVs
Provides the suite a BcolzDailyBarReader which reads from bcolz data
written to a temporary directory. The data will be read from a
collection of CSV files and then converted into the bcolz data through
BcolzDailyBarWriter.write_csvs
WithBcolzMinuteBarReader
Provides the suite a BcolzMinuteBarReader which reads from bcolz data
written to a temporary directory. The data will be read from
dataframes and then converted to bcolz files with
BcolzMinuteBarWriter.write
WithAdjustmentReader
Provides the suite a SQLiteAdjustmentReader which reads from an in
memory sqlite database. The data will be read from dataframes and then
converted into sqlite with SQLiteAdjustmentWriter.write
WithDataPortal
Provides each test case a DataPortal object with data from temporary
resources.
More documentation to follow in release notes.
Based on lazy-mainline branch, see for more details.
Also-By: Jean Bredeche <jean@quantopian.com>
Also-By: Andrew Liang <aliang@quantopian.com>
Also-By: Abhijeet Kalyan <akalyan@quantopian.com>
Renames zipline.utils.test_utils to zipline.testing
Adds zipline.testing.fixtures.ZiplineTestCase to manage setup and
teardown and adds mixins to define fixtures like an asset finder or
trading calendar.
Previously we have capitalized input strings at different levels in
our code: in the user-facing API methods and in the asset finder.
This commit moves input string capitalization exclusively to the API
method to which the string was supplied. Specifically, the string is
capitalized by a preprocess API method decorator. The preprocess
decorator passes the input string to the newly defined
ensure_upper_case() method, which returns a TypeError if the argument
supplied is not a string.
ensure_upper_case() is defined in a new file, zipline/utils/input_validation.py.
The existing expect_types() method is also moved there.
Various tests in tests/test_assets.py are modified to account for the
fact that the asset finder method lookup_symol() no longer capitalizes
its supplied argument.
'instant fill' execution may be on the path to deprecation, (the removal
is currently proposed by the branch which removes events in favor of
operating over dts.), so remove instant_fill from the close position
tests so that the input and output is the same between the proposed
branch and here.
(This was one of the few places where instant fill was used that
was not testing instant fill behavior explicitly, so may be better to
align this test with the rest of the suite.)
Also, change test methods to make it more clear on which day the values
the expected and actual results differed.
lookup_symbol_resolve_multiple was identical to lookup_symbol, except that lookup_symbol performed upper-casing of the input string and lookup_symbol would return Nones. Now, lookup_symbol has a kwarg 'default_None=True' and all symbols are upper-cased on insertion and request.
This commit removes the ability to reference a shared TradingEnvironment through the zipline.finance.trading module. In place, the classes that require a TradingEnvironment, or its child AssetFinder, contain their own references to those objects.
This commit also adds serialization utilities that allow for the pickling/unpickling of objects without unintentionally their TradingEnvironments or AssetFinders.
This patch lays the groundwork for a compute engine designed to
facilitate construction of factor-based universe screening and portfolio
allocation. It contains:
A new module, `zipline.modelling`, containing entities that can be used
to express computations as dependency graphs. Each node in such a graph
is an instance of the base `Term` class, defined in
`zipline.modelling.term`. Dependency graphs are executed by instances
of `FFCEngine`, defined in `zipline.modelling.engine`.
A new module, `zipline.data.ffc`, containing loaders and dataset
definitions for inputs to the modelling API.
New `TradingAlgorithm` api methods: `add_factor`, and `add_filter`.
These methods can only be called from `initialize`, and are used to
inform the algorithm that each day it should compute the given terms.
Computed factor results are made available through a new attribute of
the `data` object in `before_trading_start` and `handle_data`. Computed
filter results control which assets are available in the factor matrix
on each day.
Attack the startup bottleneck of creating the asset finders caches for a
large universe, which was between 1-2 seconds on development and
production machines.
Instead, allow the AssetFinder to be passed a sqlite3 file that has
already been populated and then hydrate asset objects only when an
equity is referenced for the first time.
To create aforementioned sqlite3, create an AssetFinder with an db_path
and `create_table` set to True. If `create_table` is set to False, the
prepopulated data in the sqlite file found at db_path will be used.
Default behavior is to use an in memory database.
Behavior that changes:
- Fuzzy lookup now only works on one character, that character needs to be
specified at write/metadata consumption time, since the fuzzy lookup key
is created by dropping the character from each symbol.
- Overwriting partially written metadata is no longer
supported. i.e. some unit tests allowed for inserting just the identifier,
and then later updating the symbol, end_date, etc.
Instead of building an upsert behavior at this time, this patch
changes the unit tests so that the data for each asset is only
inserted once.
Other notes:
- populate_cache is now removed, since there is no longer a two step
process of inserting metadata and then realizing that metadata into
assets. _spawn_asset is rolled into insert_metadata, so that a call to
insert_metadata both converts the metadata and makes it available in
the data store.
The lookup of future contract by individual symbol is a constraint on
incoming changes of changing how the asset finder stores data.
(i.e. the asset finder is changing so that there are separate tables for
both futures and equities.)
Since this lookup is not yet fully supported, we can add it back in on
top of the new asset finder.
Test sources are now defined by the sim_params period_start and period_end, rather than by the period_start and a defined 'count' of bars. This allows us to consider the sim_params.period_end as the canonical definition of the end of a simulation.
Remove pieces that are no longer used now that the simple transforms are
wrappers around history via the SIDData object.
Move window length related pieces into batch_transform, since the rest
of the utils module is no longer used.
Currently, `order_percent()` and `order_target_percent()` both operate as a percentage of `self.portfolio.portfolio_value`. This PR lets them operate as percentages of other important MVs.
(also adds `context.get_market_value()`, which enables this functionality)
For example:
```python
order_percent('AAPL', 0.5)
order_percent('AAPL', 0.5, percent_of='cash')
order_target_percent('MSFT', 0.1, percent_of='shorts')
tech_stocks = ('AAPL', 'MSFT', 'GOOGL')
tech_filter = lambda p: p.sid in tech_stocks
for stock in tech_stocks:
order_target_percent(stock, 1/3, percent_of_fn=tech_filter)
```
- NotHalfDay only worked at midnight
- week_(start|end) were actually month_(start|end)
- Removes check_args from api.
- Default offset of 30mins for market_(open|close)
schedule_function takes a date rule, a time rule, and a function and
will call the function, passing context and data only when the two rules
fire. This allows for code that is conditional to the datetime of the
algo.
This is implemented internally with `Event` objects which are pairings
of `EventRule`s and callbacks.
handle_data becomes a special event with a rule that always fires. This
makes the logic for handling events more complete and compact.
The initialization of perf_tracker had been moved from __init__
in TradingAlgorithm to _create_generator. This caused perf_tracker
to not be ready when portfolio requested it. portfolio was consequently
not ready for access in init. portfolio can now be accessed in init
again, assuming valid sim_params are passed. Otherwise it will be
available in handle_data() after _create_generator() is called.
Created a new flag in TradingAlgorithm that enables subclasses to
decide if they want to handle setting self.initialized = True.
Before it was the responsibility of an overriding subclass to set
initalized = True. This was causing problems because it's easy to
forget this. Now it is the responsibility of TradingAlgorithm
unless explicity stated otherwise.
Previously, calling order() in initalize resulted in a weird
stack trace. It now returns a well formulated error that is
readable to the user through the API. Adding a slippage
kwarg to test_algorithm and simfactor was necessary because
slippage can only be called during init. Previously initaliazed
was never set to true and calls to init-only function were sprinkled
around the code in non-init sections. Code changes were to enforce
init-only rules.
There were sevaral places you could supply sim_params
in TradingAlgorithm (__init__, run). This got confusing
as its not clear who updated what and which one was the
correct one to use at each time.
Then there were to ways to define data_frequency, one in
__init__() and one in the sim_params which also added code
complexity.
This refactor makes it explicit that sim_params are to be
passed to __init__() only. Moreover, data_frequency is
only stored in sim_params. For backwards compatibility,
it can still be supplied separately but will link to
the one in sim_params.
For example, you could create new sim params via:
sim_params = create_simulation_parameters(data_frequency='minute')
algo = MyAlgo(sim_params)
algo.run(data)
In addition, perf_tracker only gets initialized in one place:
_create_generator() which should also make the various ways
of running an algorithm more deterministic.
This also fixes a bug with SimulationParameters where
you could not change the period_start. Unfortunately, the
current implementation still requieres an implicit call to
update the internal variables.
Filter out empty lists from `get_open_orders` so that we have consistent
behavior between the case where a user has never placed an order and the case
where the user has placed an order but it has been executed or cancelled.
A nice side-effect, which was the impetus for this change, is that you can
check if you have any open orders by doing:
```
len(get_open_orders()) == 0
```
Also adds a test for the behavior of `get_open_orders`, which was previously
lacking.
Adds four new methods to the Zipline API that can be used as circuit-breakers
to interrupt the execution of an algorithm. The API methods are:
`set_max_position_size`
`set_max_order_size`
`set_max_order_count`
`set_long_only`
Internally, these methods are implemented by each registering a TradingControl
callback object with the TradingAlgorithm. During
TradingAlgorithm.__validate_order_params (and thus before any side-effects of
the order call occur), each callback's `validate` method is called with
information about the order to be placed and the algorithm's current state,
raising an exception if the callback detects that an error condition has been breached.
Add a test case in test_algorithms to verify that appropriate exceptions are
thrown if an algorithm makes a call to the order api with a stop/limit price
and a style.
Add `style` parameter to order_value, order_percent, order_target,
order_target_percent, and order_target_value methods. The style parameter is
forwarded to the underlying call to `order`.
Adds a test algorithm that tries to buy with very high limit prices/very low
stop prices and tries to sell with very low limit prices/very high stop prices.
This adds a new data source that emits events
with certain user-specified frequency (minute
or daily).
This allows users to backtest and debug an
algorithm in minute mode to provide a cleaner
path towards Quantopian.
A symbol() lookup feature was added to Quantopian.
By adding the same API function to zipline we can
make copy&pasting of a zipline algo to Quantopian
easier.
This is a step towards the goal of uniting Quantopian scripts
and zipline.
To make the syntax of zipline identical to Quantopian
we break out the API methods (like order) and turn them into
functions. To access the algo object we add a thread local reference
to the current algorithm that is accessed in the API functions.
TradingAlgorithm now takes either a string or two functions
(initialize and handle_data) that it executes.
Use api method decorator for methods available in algoscript.
Ported appropriate algorithm tests from internal code.
To help prevent algorithms from operating on positions that are
not in the existing universe of stocks.
Formerly, iterating over positions would return positions for stocks
which had zero shares held. (Where an explicit check in algorithm
code for `pos.amount != 0` could prevent from using a non-existent
position.)
The `test_multi_source_as_input_no_start_end` test is designed
to test for a lack of simulation parameters, so make that they are
not passed.
Was unnoticed because sim_params passed as an arg are dropped.
The start and end of the simulation parameters should be 'normalized'
i.e. midnight timestamped.
However, the algorithm tests were using the timestamp of the
first and last trade, which were in market times,
i.e. 9:30 AM and 4:00 PM EST.
Fix passing the sim_params that is used to create the trade_history,
instead of having the sim_params inferred from the source.
(Also may want to consider fixing the logic that infers the date
range from the sources provided.)
Also, add a `num_days` option to `factory.create_simulation_parameters`
so that the a date range that covers the desired number of days is covered.
Since the default sim_params were covering a year, while the test only
supplies 4 values, causing an alignment issue with the record test,
since a years worth of results were returned, but there were only 4 events.
Global state for the financial simulation environment is accessed through the
zipline.finance.trading module, which now contains a module variable:
environment.
Parameters are passed into an algorithm as a keyword argument, sim_params.
SimulationParameters creates a trading day index for the test period that
can be used to find trading days, calculate distance between trading days,
and other common operations. The sim params index is just selected from the
global state.
================
Details:
- adding delorean to the requirements.
- made index symbol a parameter for loading the benchmark data. changed
messagepack storage to be symbol specific.
- ported risk, performance, algorithm, transforms, batch transforms
and associated tests to use simulation parameters and global environment
- factory and sim factory use global state and sim params
- factory method parameter names now reflect the class expected
Takes the value set for a variable on handle_data and records it,
e.g.:
```
def initialize(self):
self.incr = 0
self.record_variables(['incr'])
def handle_data(self, data):
self.incr += 1
```
Would record a variable of `incr`.
Emits the recorded variables as part of the daily performance.
This batch combins work from:
Thomas Wiecki <thomas.wiecki@gmail.com> (@twiecki)
fawce <fawce@quantopian.com> (@fawce)
