Discrete actions multi-data, multi-asset setup intro.

Problem formulation

Consider setup with one riskless asset acting as broker account cash and K risky assets. For every risky asset there exists track of historic price records referred as data-line. Apart from assets data lines there [optionally] exists number of exogenous data lines holding some information and statistics, e.g. economic indexes, encoded news, macroeconomic indicators, weather forecasts etc. which are considered relevant to decision-making. It is supposed for this setup that:

1. there is no interest rates for any asset;
2. broker actions are fixed-size market orders (`buy`, `sell`, `close`); short selling is permitted;
3. transaction costs are modelled via broker commission;
4. 'market liquidity' and 'capital impact' assumptions are met;
6. time indexes match for all data lines provided;

Model

The problem is modelled as discrete-time finite-horizon partially observable Markov decision process for equity/currency trading:

- *for every asset* traded agent action space is discrete `(0: `hold` [do nothing], 1:`buy`, 2: `sell`, 3:`close` [position])`;
- environment is episodic: maximum  episode duration and episode termination conditions
  are set;
- for every timestep of the episode agent is given environment state observation as tensor of last
  `m` time-embedded preprocessed values for every data-line included and emits actions according some stochastic policy.
- agent's goal is to maximize expected cumulative capital by learning optimal policy;

Environment setup explanation:

    1. This environment expects Dataset to be instance of `btgym.datafeed.multi.BTgymMultiData`, which sets
    number,  specifications and sampling synchronisation for historic data for all assets and datalines
    one want to consider jointly.

    2. Internally every episodic asset data is converted to single bt.feed and added to environment strategy
    as separate named data-line (see backtrader docs for extensive explanation of data-lines concept).
    Strategy is expected to properly handle all received data-lines.

    3. btgym.spaces.ActionDictSpace and order execution. Strategy expects to receive separate action
    for every asset in form of dictionary: `{data_line_name_1: action, ..., data_line_name_K: action}`
    for K assets added, and issues orders for all assets within a single strategy step.
    It is supposed that actions are discrete [for this environment] and same for every asset.
    Base actions are set by strategy.params.portfolio_actions, default is: ('hold', 'buy', 'sell', 'close')
    which equals to `gym.spaces.Discrete` with depth `N=4 (~number of actions: 0, 1, 2, 3)`. 
    That is, for `K` assets environment action space will be a shallow dictionary 
    `(DictSpace)` of discrete spaces:
    `{data_line_name_1: gym.spaces.Discrete(N), ..., data_line_name_K: gym.spaces.Discrete(N)}`

        Example::

            if datalines added via BTgymMultiData are: ['eurchf', 'eurgbp', 'eurgpy', 'eurusd'],
            and base asset actions are ['hold', 'buy', 'sell', 'close'], than:

            env.action.space will be:
                DictSpace(
                    {
                        'eurchf': gym.spaces.Discrete(4),
                        'eurgbp': gym.spaces.Discrete(4),
                        'eurgpy': gym.spaces.Discrete(4),
                        'eurusd': gym.spaces.Discrete(4),
                    }
                )
            single environment action instance (as seen inside strategy):
                {
                    'eurchf': 'hold',
                    'eurgbp': 'buy',
                    'eurgpy': 'hold',
                    'eurusd': 'close',
                }
            corresponding action integer encoding as passed to environment via .step():
                {
                    'eurchf': 0,
                    'eurgbp': 1,
                    'eurgpy': 0,
                    'eurusd': 3,
                }
            vector of integers (categorical):
                (0, 1, 0, 3)

    4. Environment actions cardinality and encoding. Note that total set of environment actions for `K` assets
    and `N` base actions is a `cartesian product of K sets of N elements each`. 
    It can be encoded as `vector of integers,
    single scalar, binary or one_hot`. As cardinality skyrockets with `K`, 
    `multi-discrete` action setup is only suited
    for small number of assets.

        Example::

            Setup with 4 assets and 4 base actions [hold, buy, sell, close] spawns total of 256 possible
            environment actions expressed by single integer in [0, 255] or binary encoding:
                vector str :                            vector int:     int:   binary:
                ('hold', 'hold', 'hold', 'hold')     -> (0, 0, 0, 0) -> 0   -> 00000000
                ('hold', 'hold', 'hold', 'buy')      -> (0, 0, 0, 1) -> 1   -> 00000001
                ...         ...         ...
                ('close', 'close', 'close', 'sell')  -> (3, 3, 3, 2) -> 254 -> 11111110
                ('close', 'close', 'close', 'close') -> (3, 3, 3, 3) -> 255 -> 11111111

    Internally there is some weirdness with encodings as we jump forth and back between
    dictionary of names or categorical encodings and binary encoding or one-hot encoding.
    As a rule: strategy operates with dictionary of string names of actions, environment 
    sees action as dictionary of integer numbers while policy estimator operates with 
    either binary or one-hot encoding.

    5. Observation space: is nested DictSpace, where 'external' part part of space 
    should hold specifications for every data line added.

        Example::

            if datalines added via BTgymMultiData are:
                'eurchf', 'eurgbp', 'eurgpy', 'eurusd';

            environment observation space should be DictSpace:
             {
                'raw': spaces.Box(low=-1000, high=1000, shape=(128, 4), dtype=np.float32),
                'external': DictSpace(
                    {
                        'eurusd': spaces.Box(low=-1000, high=1000, shape=(128, 1, num_features), dtype=np.float32),
                        'eurgbp': spaces.Box(low=-1000, high=1000, shape=(128, 1, num_features), dtype=np.float32),
                        'eurchf': spaces.Box(low=-1000, high=1000, shape=(128, 1, num_features), dtype=np.float32),
                        'eurgpy': spaces.Box(low=-1000, high=1000, shape=(128, 1, num_features), dtype=np.float32),
                    }
                ),
                'internal': spaces.Box(...),
                'datetime': spaces.Box(...),
                'metadata': DictSpace(...)
            }

Reference:

• MultiDataFeed: https://kismuz.github.io/btgym/btgym.datafeed.html#btgym.datafeed.multi.BTgymMultiData
• ActionSpace: https://kismuz.github.io/btgym/btgym.html#btgym.spaces.ActionDictSpace
• MultiDiscreteEnv: https://kismuz.github.io/btgym/btgym.envs.html#btgym.envs.multidiscrete.MultiDiscreteEnv
• PortfolioEnv: https://kismuz.github.io/btgym/btgym.envs.html#btgym.envs.portfolio.PortfolioEnv