Improve split-repair of multi-day intervals. Because split error can occur within a multi-day interval, e.g. mid-way through week, need to repair each OHLC column separately
Increase robustness of repair 'Adj Close'
Limit price-repair recursion depth to 2
Various important changes to yfinance logging:
Remove handler from YF logger as breaks standard logging practice.
Improve 'download' log message grouping
Move custom DEBUG log formatting into 'yf.enable_debug_mode()', which if called adds the handler. The custom DEBUG log formatting is:
- 'MultiLineFormatter' improves appearance of multi-line messages in any logging mode. Adds leading indent to all lines below first, to align with the first line which is indented by %levelname%.
- Whitespace padding is added to end of %levelname% string up to 8 characters. This resolves different level strings e.g. INFO & DEBUG creating different indents. But this only automatically applies to first line of message - for multi-line messages, 'MultiLineFormatter' extracts amount of padding and applies to all other lines.
- Add leading indent proportional to function depth, because DEBUG generates a lot of messages particularly when repairing price data - same function can be called recursively. Implemented in 'IndentLoggerAdapter', a simple wrapper around the logger object.
- 'log_indent_decorator' inserts 'Entering/Exiting %function%' debug messages, helps organise.
Main change is fixing price-repair of 1d 'Adj Close'. 1d repair uses 1h data, but 1h is never div-adjusted. For correct 1d 'Adj Close', extract div-adjustment from the good 1d data, and calculate it for any bad 1d data. A new unit test ensures correctness.
Other changes:
- bug fix in split-repair logic to handle price=0
- improve unit test coverage on price dividend
- add 1wk interval to split-repair unit test
Stumbled upon another type of 100x price error - Yahoo may switch a symbol from e.g. cents -> $ on some recent date, so recent prices are 100x different. The new split-repair is perfect for this - set change to 100 and ignore 'Volume'.
An out-of-range dividend was breaking merge with 1mo-prices, so fixed that. Also replaced the mega-loop with Numpy, much clearer now. Improved its tests.
Recent release action generated deprecated error: "Node.js 12 actions are deprecated. Please update the following actions to use Node.js 16: actions/checkout@v2, actions/setup-python@v2."
So simply increasing versions to match latest GitHub usage docs, hopefully works.
New logging in `download` stores the tracebacks, but the logic was faulty, this fixes that.
Also improves error handling in `download`.
Unit tests should have detected this so improved them:
- add/improve `download` tests
- disable tests that require Yahoo decryption (because is broken)
- fix logging-related errors
- improve session use
Improve logging messages related to price data fetches:
- fix 'debug is deprecated' msg
- append user args to 'may be delisted' msg - interval & dates/period
- improve formatting of 'cannot reconstruct' msg
- hide errors in 'history()' while accessing 'fast_info[]'
