This commit adds initial support for international markets in the Pipeline API
via addition of the concept of "domains".
Significant changes on this branch include:
- Added `zipline.pipeline.domain` and the `Domain` interface.
- Added several canonical domain definitions:
- GENERIC
- US_EQUITIES
- CA_EQUITIES
- GB_EQUITIES
- Added `zipline.country` as a place to define canonical country code names.
- Made `PipelineLoader` an interface and added new `domain` parameter to
signature of `load_adjusted_array`.
- Added a new system for "generic" and "specialized" DataSet and BoundColumn
objects.
- Added a new system for inferring a domain from a pipeline that contains a mix
of generic and non-generic terms.
- Reworked the built-in pricing dataset. USEquityPricing dataset is now a
specialized version of a generic EquityPricing dataset, and most built-in
factors are now implemented in terms of EquityPricing rather than
USEquityPricing.
- Removed `zipline.data.us_equity_pricing`.
- Moved BcolzDailyBar{Reader,Writer} to `zipline.data.bcolz_daily_bars`.
- Moved SQLiteAdjustment{Reader,Writer} to `zipline.data.adjustments`.
New dependencies added as part of this work:
- iso3166 (for canonical country code definitions)
- python-interface (for strict interface definitions).
See https://github.com/quantopian/zipline/issues/2265 for a full description of
the design for domains.
From pep-0008:
```
Always use a def statement instead of an assignment statement that binds a
lambda expression directly to an identifier.
Yes:
def f(x): return 2*x
No:
f = lambda x: 2*x
The first form means that the name of the resulting function object is
specifically 'f' instead of the generic '<lambda>'. This is more useful for
tracebacks and string representations in general. The use of the assignment
statement eliminates the sole benefit a lambda expression can offer over an
explicit def statement (i.e. that it can be embedded inside a larger expression)
```
Allows ``__funcname`` to be passed to preprocessors like expect_types
and expect_dtypes to override the name displayed in error messages.
This is useful for providing clearer errors for ``__init__`` and
``__new__`` methods in classes.
Instead of having separate ExchangeCalendar and TradingSchedule objects, we
now just have TradingCalendar. The TradingCalendar keeps track of each
session (defined as a contiguous set of minutes between an open and a close).
It's also responsible for handling the grouping logic of any given minute
to its containing session, or the next/previous session if it's not a market
minute for the given calendar.
Changes BcolzDailyBarWriter to not be an abc, data is passed as an
iterator of (sid, dataframe) pairs to the write method.
Changes the AssetsDBWriter to be a single class which accepts an engine
at construction time and has a `write` method for writing dataframes for
the various tables. We no longer support writing the various other data
types, callers should coerce their data into a dataframe themselves. See
zipline.assets.synthetic for some helpers to do this.
Adds many new fixtures and updates some existing fixtures to use the new
ones:
WithDefaultDateBounds
A fixture that provides the suite a START_DATE and END_DATE. This is
meant to make it easy for other fixtures to synchronize their date
ranges without depending on eachother in strange ways. For example,
WithBcolzMinuteBarReader and WithBcolzDailyBarReader by default should
both have data for the same dates, so they may use depend on
WithDefaultDates without forcing a dependency between them.
WithTmpDir, WithInstanceTmpDir
Provides the suite or individual test case a temporary directory.
WithBcolzDailyBarReader
Provides the suite a BcolzDailyBarReader which reads from bcolz data
written to a temporary directory. The data will be read from
dataframes and then converted to bcolz files with
BcolzDailyBarWriter.write
WithBcolzDailyBarReaderFromCSVs
Provides the suite a BcolzDailyBarReader which reads from bcolz data
written to a temporary directory. The data will be read from a
collection of CSV files and then converted into the bcolz data through
BcolzDailyBarWriter.write_csvs
WithBcolzMinuteBarReader
Provides the suite a BcolzMinuteBarReader which reads from bcolz data
written to a temporary directory. The data will be read from
dataframes and then converted to bcolz files with
BcolzMinuteBarWriter.write
WithAdjustmentReader
Provides the suite a SQLiteAdjustmentReader which reads from an in
memory sqlite database. The data will be read from dataframes and then
converted into sqlite with SQLiteAdjustmentWriter.write
WithDataPortal
Provides each test case a DataPortal object with data from temporary
resources.
Add a cache interface which supports expirable entries with a changeable
backend for the cache into which they are entered.
The default cache is a `dict` but could swapped for
`cachetools.LRUCache` or any other cache which supports `__get__`,
`__set__`, and `__del__`.
So that consumers can change the use of `CachedObjects` stored in a
cache from:
```
self._cache = {}
...
try:
obj = self._cache[key]
try:
return obj.unwrap(dt)
except Expired:
pass
except KeyError:
pass
...
self._cache[key] = CachedObject(value, new_expiration)
```
to:
```
self._cache = ExpiringCache(LRUCache(maxsize=6))
...
try:
return self._cache.get(key, dt)
except KeyError:
# Get fresh value
...
self._cache.set(key, value, new_expiration)
```
More documentation to follow in release notes.
Based on lazy-mainline branch, see for more details.
Also-By: Jean Bredeche <jean@quantopian.com>
Also-By: Andrew Liang <aliang@quantopian.com>
Also-By: Abhijeet Kalyan <akalyan@quantopian.com>
Renames zipline.utils.test_utils to zipline.testing
Adds zipline.testing.fixtures.ZiplineTestCase to manage setup and
teardown and adds mixins to define fixtures like an asset finder or
trading calendar.
loaders
This allows people to set their cutoff time to the time they will
actually execute 'before_trading_start'. Currently this is just passed
to the constructor of the loader; however, I would like to make this
managed by the algorithm simulation runner. This would help keep all of
the loaders in sync and lock 'before_trading_start's execution to the
time the data is queried for.
EarningsCalendar loader.
- Moves most of AdjustedArray back into Python. The window iterator is
the only part that's performance-intensive.
- Adds a bootleg templating system for creating specialized versions of
AdjustedArrayWindow for each concrete type we care about.
- Adds support for differently dtyped terms in pipeline. This allows us
to use datetime64s which are needed in the EarningsCalendar.
- Adds EarningsCalendar dataset for the next and previous earnings
announcements in pipeline.
- Adds in memory loader for EarningsCalendar.
- Adds blaze loader for EarningsCalendar.
