Quick scan for cycles🙏🏻
The followup for
As I told before, ML based algorading is all about detecting any kind of non-randomness & exploiting it (cuz allegedly u cant trade randomness), and cycles are legit patterns that can be leveraged
But bro would u really apply Fourier / Wavelets / 'whatever else heavy' on every update of thousands of datasets, esp in real time on HFT / nearly HFT data? That's why this metric. It works much faster & eats hell of a less electicity, will do initial rough filtering of time series that might contain any kind of cyclic behaviour. And then, only on these filtered datasets u gonna put Periodograms / Autocorrelograms and see what's going there for real. Better to do it 10x times less a day on 10x less datasets, right?
I ended up with 2 methods / formulas, I called em 'type 0' and 'type 1':
- type 0: takes sum of abs deviations from drift line, scales it by max abs deviation from the same drift line;
- type 1: takes sum of abs deviations from drift line, scales it by range of non-abs deviations from the same drift line.
Finnaly I've chosen type 0 , both logically (sum of abs dev divided by max abs dev makes more sense) and experimentally. About that actually, here are both formulas put on sine waves with uniform noise:
^^ generated sine wave with uniform noise
^^ both formulas on that wave
^^ both formulas on real data
As you can see type 0 is less affected by noise and shows higher values on synthetic data, but I decided to put type 1 inside as well, in case my analysis was not complete and on real data type 1 can actually be better since it has a lil higher info gain / info content (still not sure). But I can assure u that out of all other ways I've designed & tested for quite a time I tell you, these 2 are really the only ones who got there.
Now about dem thresholds and how to use it.
Both type 0 and type 1 can be modelled with Beta distribution, and based on it and on some obvious & tho non mainstream statistical modelling techniques, I got these thresholds, so these are not optimized overfitted values, but natural ones. Each type has 3 thresholds (from lowest to highest):
- typical value (turned off by default). aka basis ;
- typical deviation from typical value, aka deviation ;
- maximum modelled deviation from typical value (idk whow to call it properly for now, this is my own R&D), aka extension .
So when the metric is above one of these thresholds (which one is up to you, you'll read about it in a sec), it means that there might be a strong enough periodic signal inside the data, and the data got to be put through proper spectral analysis tools to confirm / deny it.
If you look at the pictures above again, you'll see gray signal, that's uniform noise. Take a look at it and see where does it sit comparing to the thresholds. Now you just undertand that picking up a threshold is all about the amount of false positives you care to withstand.
If you take basis as threshold, you'll get tons of false positives (that's why it's even turned off by default), but you'll almost never miss a true positive. If you take deviation as threshold, it's gonna be kinda balanced approach. If you take extension as threshold, you gonna miss some cycles, and gonna get only the strongest ones.
More true positives -> more false positives, less false positives -> less true positives, can't go around that mane
Just to be clear again, I am not completely sure yet, but I def lean towards type 0 as metric, and deviation as threshold.
Live Long and Prosper
P.S.: That was actually the main R&D of the last month, that script I've released earlier came out as derivative.
P.S.: These 2 are the first R&Ds made completely in " art-space", St. Petersburg. Come and see me, say wassup🤘🏻
