Test your trading ideas

Trading Platform Limitations

October 18, 2015 by Shaun Overton 6 Comments

This post was authored by Ben Fulloon, a respected trader and subscriber to OneStepRemoved.

I developed an awesome strategy with a drawdown ratio of 13.67. Sounds amazing, right? Too bad that my trading platform overstated the results by more than double!

It’s important to learn about both your brokers and platforms limitations. Sometimes these intricacies only become apparent through time and experience. It’s so frustrating when your trading platform doesn’t function or report results as expected.

In this article I’ll point out two limitations of NinjaTrader 7, one bad limitation and one which can actually turn out surprisingly better for the trader in certain situations. However, this is more to do with the broker I’m using and not the platform itself.

NinjaTrader is definitely not the only platform that has limitations: MetaTrader, TradeStation, X-Trader, Matlab, etc. all have limitations for quantitative finance.

I’ll just be writing about NinjaTrader in this article to keep it fairly short and easy to read. I am also not intending to make out NinjaTrader as being a bad platform either. But, there are definitely some improvements that could be made to make it a lot easier and more convenient for quantitative traders to develop and trade strategies.

The first quirk relates to the broker I’m using. Specifically, it’s the day trade margins that I care about. These day trade margins end 15 minutes before the close of the session. For instance the ES (Emini S&P500) has a day trade margin of $500, which ends at 4:00pm CT that then reverts back to the full trading margin of $5060 before the session closes at 4:15pm CT. (Times stated are correct at time of Writing, The ES now closes at 4:00pm CT and the Day Trade margin ends at 3:45pm CT)

I’ll show you a screenshot of the results of a day trading strategy that I developed. This strategy trades the ES, NQ (Emini Nasdaq 100) and the YM (Emini Dow) all at the same time. The easiest way to exit on close with NinjaTrader is setting “Exit on Close” to true which will then exit on the close of the session.

According to the results the strategy makes a total of $332,771.60 with a maximum drawdown of $25,912.27 since 2008 to now. This is a drawdown ratio of 12.84. That’s oustanding!

The issue is… and you knew there’d be a problem… is that the strategy exits at 4:15pm CT. Day trading margin ends at 4:00pm CT. The strategy is therefore highly likely to get a margin call with a small account size.

It makes sense to tweak the strategy to make best use of the day trading margin. Ninjatrader offers a custom session template, which in this case I made end at 4:00pm CT. The results of the custom session template is as follows.

The exact same strategy applied to the same instruments to avoid a margin call makes $335,819.30 with a maximum drawdown of $24,560.51. This is a drawdown ratio of 13.67.

I didn’t change the strategy with the goal of improving the drawdown ratio AND the profit. But hey, I’ll take it. Finding a limitation in the platform can actually benefit you in some situations.

This strategy is based on trading 3 different instruments. The ES, the NQ and the YM. The problem is that I backtested it using an instrument list in NinjaTrader. What this means is they’re all tested separately. NinjaTrader then combines the test results for you as a total result like the results of the screenshots above.

Here’s what it looks like when you test them as an instrument list. This shows the different profits and drawdowns of the individual instruments.

Now at first glance it reads that the trader would have made $335,819.30 with a maximum drawdown of $24,560.51 if they traded all three instruments together. Don’t you agree?

The problem is that this is incorrect. NinjaTrader doesn’t actually combine the results like you’d think. The trader still would have made roughly that money. However, all the statistics aren’t quite correct.

To show this I recreated the exact same strategy however it will trade the ES, NQ and YM all at the same time instead of trading them separately like it does by default. These are the results when you program it into a multi-instrument strategy

It makes $335,915.30 which is roughly the same amount, but it has a maximum drawdown of $59,937.60 instead of the $24,560.51 it originally looked like it would be. This makes it a drawdown ratio of 5.60, which is a lot worse than the original 13.67.

If the trader decided to trade based upon the maximum drawdown of $24,560.51, they may get a nasty shock when the drawdown turns out to be twice as bad as they were expecting.

Incorrect calculations on such an important metric could jeopardize an account. You might assume that you can get away with half of the equity that’s actually required to trade the strategy. Oops?!?

The misleading statistics in NinjaTrader makes this strategy look really nice. But when the drawdown is more than double what it appeared that it would have been originally, you might get a nasty shock.

This is why it’s important to learn both your platforms and brokers limitations as early as possible. You don’t want to learn these limitations the hard way.

In a few weeks time, I’ll reveal a simple way to create multi-instrument strategies which show more accurate metrics. Stay tuned for my next article in the series.

The Art of Crafting an Automated Trading Strategy

September 21, 2015 by Lior Alkalay 28 Comments

For nearly every trader, there is one, all-consuming question: how do you build a profitable strategy?

Naturally, one would expect to dive straight into the statistics and begin crunching the numbers. Yet, before we delve further into the mechanics – and believe me there are plenty – I want to talk a bit about the philosophy and, indeed, the rules behind a good strategy.

With all the numbers needing to be crunched, it’s unquestionably important to have a rough sketch of your ideas for building a strategy. Only with that rough sketch in hand can you then move onto the blue prints of actually constructing a sound, viable strategy.

There is No Perfect Trading Strategy

Having said that, let’s get something perfectly clear and out of the way: Your first rule of trading is to understand that there is no perfect strategy. But don’t let that stop you from crafting the best trading strategy possible. There are some steps to take before you even begin to draft your strategy.

If you hope to one day be able to gauge the market you need to first find your philosophical starting point. That is the acceptance of a single immutable truth which you must first embrace.

Before you dive into the practise, the data to the equation and, of course, the money, you need a reference point. And that reference point is simply this: The market has a heartbeat. That means there is no one-size-fits-all strategy that can predict the market each and every time. That’s because the market is not static; it’s constantly moving and the rules are constantly changing.

Many, many times I’ve seen traders try to optimise their strategy to account for every conceivable condition. The next thing you know, they’ve not only lost direction but they’ve lost a bundle. As I’ve said, no single strategy works all the time. Thus, building a successful strategy calls for you to predict the market at certain very specific points. Moreover, you need to recognize the “dead zones” where the strategy is not predictive, and make it a point not to engage.

Stick to the Rules

Your second rule to producing a viable trading strategy is to have clear mechanical rules of engagement. That is, clear conditions as to when you open a trade and when you close a trade. And you need to stick with that rule, no matter what. If you allow any leeway you actually change the mechanism, thus making it impossible to measure.

Let’s say your first rule of engagement looks like this: When the 120 days EMA crosses the 60 EMA from above, you buy. This time, though, it appears to you that the pair you’re trading is still bearish. You decide that you’re not going to open a trade after all.

Though in this instance it was the right call, you’ve interfered with the rules. Thus the results you got are not a byproduct of your strategy. Instead, they’re the result of your judgment call. When a trader interferes with the rules of engagement they eliminate the ability to assess its effectiveness. So why is that a tragedy?

Because it means you’ll never know where you’re really getting it wrong. And if you never learn from your mistakes you’ll always make them. This is the most dangerous, even lethal, pitfall, and one that has burned thousands of traders with so-called “good strategies.” So, if you want to make sure your strategy is successful (and not just randomly successful), stick to the rules.

Four Phases of Testing

So now that we’ve established the philosophy it’s time to get down to business. How do you test whether or not your strategy is worth implementing? The testing process constitutes four different phases:

In-Sample Testing
Optimization
Out-of-Sample Testing
Forward Testing (i.e. paper trading)

In-Sample Testing

When you think about testing a strategy what instinctively comes to mind? Back testing, or testing your strategy based on historical data. But while back testing is one of the more important parts of testing, it might also create some misconceptions.

For example, if you test your strategy on an entire set of data, how will you know how it performed when market conditions changed? To tackle this problem professionals use what’s called In-Sample Out-of-Sample.

How you perform the test is relatively simple. The historical data is divided into two parts, the In- Sample and Out-of Sample. In-Sample represents about 2/3 of the testing period while Out-of-Sample accounts for the remaining 1/3. You can see how it plays out in the illustration below.

The In-Sample will be the preliminary test of your strategy, the first dry run, if you will. If your strategy doesn’t do well in the In-Sample test, it means you might have to ditch that strategy and go back to the drawing board.

However, if the In-Sample testing shows an ascending curve of returns that’s good news! It means you’ve got something to work with. Now it’s time to tighten the screws, just like a mechanic, and in the world of trading that means optimizing your strategy.

Optimizing Your Trading Strategy

Now, optimizing a strategy is perhaps the most mathematically intensive part of carving out a strategy. Even if math is not your forte, it’s important enough not to ignore. There are three methods we can use: Correlation, return distribution and curve fitting. Let’s look at how we’d put them to use.

Of course, as a case study we will use the simplest trading strategy that traders use to ride a trend is the moving average cross. The moving average cross works like this: If the fast moving average (short period) is above the slow moving average (longer period) that’s a buy signal.

Conversely, if the fast moving average is below the slow moving average then that’s a sell signal. Now, let’s say that we decide to open a position but only if certain parameters are met. The question is how can you know if those parameters are, in fact, the optimal ones? Well, that’s where our statistical methodologies will come in handy.

Correlation, Return Distribution and Curve Fitting

The first method is correlation. Essentially, you switch to a different set of parameters that better correlates to the market. Let me elaborate; say your first test or trial was 120 days for the long average and 30 days for the short average (or 120, 30). Then you tested a few more options, let’s say 120, 14 and then 60, 30 days.

Next you compare the correlation of each data set. The closer the R² Correlation Coefficient is to 1, the better. That means this strategy is better at predicting the market. What if you get an R value that is closer to -1?

Well, that’s good, too, in its own way. It means you should be selling rather than buying, because the market is moving in the opposite direction. (Of course, with the MA crosses it’s very unlikely to get a -1 correlation anyway.)

Now, if you get a value close to 0 that’s not so good. That signals that there’s no or very little correlation between your strategy and the market. If you had positive gains on the first back testing and the correlation is 0 then the reality is your success was random and not indicative.

As you can see from the three options, our first choice was actually the best correlated to the market. That suggests that the parameters we chose in this particular case were optimal.

Distribution of Returns

After we checked to see which strategy better correlated to the market there is another dimension to consider. Let’s say one set of parameters is better correlated to the market thus it’s more successful. Another set of parameters is not quite as correlated but a successfully executed trade yields more on average per trade.

As seen in charts above, we can see that that makes for an interesting insight. Our preliminary parameters (120, 30) had a better distribution of returns, meaning returns per trade are stable rather than fluctuating. And that makes sense because it has a higher correlation to the market, as indicated by the first test we made.

Now, we get an interesting result, and one that you are likely to encounter. Our first parameters made more constant returns because the correlation to the market was higher. However, the average return per trade was lower than the second set of numbers (120,14).

This is quite puzzling. How would you determine, then, which strategy is better? Is it better to earn less per trade but constantly or to earn more per trade but less constantly?

To answer this question we must move to our final staple in optimization. That is to compare the curves created when using the two parameters and see which one works to our advantage, i.e. constant smaller gains or less constant bigger gains.

Comparing the Curves

When we overlay the curves we get our answer: The first set of parameters (120,30) is still preferable. While having inferior results at the beginning, the high fluctuations of the second option means that your results were more random.

That randomness would eventually lead to a strategy that does not efficiently predict the market. However, if the returns on the second set of numbers were vastly superior then the volatility in returns might be worth the risk.

But in this case, the 120, 30 had proven superior returns at the end, compensating us for taking less risk and being more predictive in the market. So, now that we’ve finished our preliminary optimization we’re ready to test our strategy with the Out-of-Sample test phase.

Out-Sample Testing

By running the Out-of-Sample strategy after the other tests you’ll gain valuable insight as to how your strategy reacts to different market conditions than initially considered.

Now we need to check the results of each. There are two different strategies (A and B) that were tested with both the In- and Out-of-Sample. As you can see, Strategy A had been rather successful on the In-Sample (2/3) but once checked against the entire data set it performed rather poorly.

In contrast, Strategy B had done well on both parts of the In- and Out-of-Sample, meaning you’ve got something viable. Thus, there is a greater chance that your strategy is what every trader wants – a money maker.

Graduating to Paper Trading

Congratulations! If you’ve reached this level then you’ve been successful in generating returns. Now it’s time to try out your trading strategy on live data. We’re not yet trading with actual money but it’s the closest thing, and should give you a good idea as to how well your strategy will perform live.

You may find that there are glitches that you didn’t at first notice. Or perhaps the entry or exit of each trade could be improved with another indicator. These are things you might only discover when trading goes live. After some adequate sampling, something akin to the Out-of-Sample length and – voilà – you’ve crafted a trading strategy.

Shaun put together a free, 6-step checklist to help traders build their own automated trading systems. It’s the same one that he used for QB Pro. If you’re feeling overwhelmed with where to begin your own strategy, then that free handout is the logical place to start.

The Dimension of Time in Trading

July 9, 2015 by Lior Alkalay 6 Comments

The trading blogosphere is full of articles about the importance of timing your trade. Certainly, it’s important; identifying the right trend, knowing when to open or close – all integral to the trade. But here is the thing, there’s another dimension and timing is only half the story. You see, in the real world, it’s not just about the timing of your entry and exit but also the time between. In fact, the duration of your trade matters a lot more than you might think. In this article I’ll discuss the various aspects in which time can affect both short and long term strategies.

Duration and Returns

Let’s consider a theoretical trading situation, looking at two strategies. The first is a high frequency one designed at gaining 1 pip, with every trade taking a single second. We’ll call this Strategy A. The second strategy, Strategy B, is designed to generate a profit of 10 pips every 10 seconds. So which is the better strategy? On paper, both should generate 60 pips per minute hence equal returns. But in the real world? Well, that’s an entirely different story.

In real life, when you aggregate the results of the two strategies, you’re going to be in for a big surprise. Under Strategy A you had a profit of 50 pips after 1 minute. But under Strategy B you had a profit of 66 pips. Now which strategy do you think is better? Of course, you’d pick B. Interestingly, per trade, each strategy profited exactly as planned. Strategy A profited 1 pip and Strategy B profited 10 pips per trade.

So what made the difference? The time. As you can see below in the two charts, strategy A, when executed, took a little longer on average than initially expected. From the measurement of the duration of trades it clustered around 1.2 seconds per trade rather than 1 second. In Strategy B, the measurement of duration per trade, on average, is about 9 seconds. When executed, Strategy B trades took less time than initially expected. Eventually, at the end of the day, Strategy B returned 66 pips or 32% more than Strategy A’s 50 pips. That means Strategy B is the superior choice.

Now, let’s look at this a bit closer still. We’re going to take the duration of each of the trades executed and measure the standard deviation (a simple exercise in Excel). You would discover that in Strategy A, while the average might be 1 second per trade, the standard deviation is high. That suggests the cycles are not equal thus the strategy is less efficient. On the other hand, Strategy B has a low standard deviation, meaning most trades are close to 10 seconds. Therefore, it has more predictable returns and, hence, a more effective strategy. When you measure a strategy, understand that beyond returns and risk, the time each trade takes matters. And in fact, it can matter quite a lot.

Duration and Swing Trading

So now we understand how the nuances of time can affect the eventual returns of each strategy. Yet when it comes to longer duration trades (i.e. weeks or months) there is still another element. In this case, it is the amount of time the trade has been positive.

Let’s look at another theoretical scenario. This time, we have Strategies C and D, each has the identical duration of two weeks. Likewise, both have the same standard deviation of duration and roughly the same returns. But one strategy is riskier than the other. The question is which? The answer is the strategy that has been profitable for the least amount of time. Say you opened two trades; Strategy C was profitable for 12 of the 14 days it took to reach its target. Strategy D had been negative for the first 12 days before producing steep returns in the last 2. Strategy D, then, is clearly more risky because you are risking a loss for a longer time. This also suggests that, perhaps, the entry signal for Strategy D is not very well calibrated. Strategy C is clearly more effective and less risky, on average.

The Bottom Line

Of course, there are many more aspects to measuring time effectiveness in a trading strategy. There are also other dimensions to examine beyond just time. But the lesson here is clear. The next time you measure your own trading strategy remember that time isn’t just about timing.

Autocorrelation

July 5, 2015 by Shaun Overton 11 Comments

It’s the 4th of July weekend. My wife is running errands this afternoon and my youngest son is napping. I can’t go outside for a run – it’s July in Texas (i.e., incredibly hot) and, what if the baby wakes up? The only ways to spend my holiday afternoon was either to a) watch TV or 2) dive into Michael Halls-Moore’s recent article, “Series Correlation in Time Series“.

I’m going to skip the math and keep this very layman friendly. There are some key take aways that even the most mathematically challenged traders can learn.

The Layman’s Guide to Correlation

Let’s go easy on the mathiness and remind ourselves what correlation is. Correlation is directly related to the angle between two points.

It’s easiest to understand in two dimensions because that’s what fits on a piece of paper or computer screen. The points (1,0) and (2,0) share an angle of 0°. Basically, the points lie on the same line.

Correlation is directly related to the angle between two points. These points lay on the same line, so the angles between each other are 0°. The fact that they are on the same line makes them 100% correlated.

Correlation is the cosine of the angle between two points. Get out a calculator and try it. The cosine of 0° is 1.

cos(0) = 1.0, which is 100% correlation.

Look at the image above. The points (1,0) and (2,0) share the same line. Doesn’t it make sense that points on the same line are 100% correlated? Points that are less than 100% correlated lie on different lines.

To keep this intuitive, you probably know from trading that a 60% correlation is worth paying attention to, but it’s not very predictive. What does that mean geometrically?

We want to get the angle where cos⁻¹(θ) = 0.6. Please don’t get caught up on the cosine stuff. I just want you to realize how this relates to angles.

The angle with a 60% correlation is 53°. They’re kind of close together, but not too close.

Those lines are pretty far apart. A 60% correlation between two forex pairs puts you in the same ballpark, but you’re hardly tracking pip for pip.

Forex prices are like points on the graphs above. 1 bar ago, 2 bars ago… 16,000 bars ago the prices of EURUSD were 1.11342, 1.11297…. 1.31974. The historical data on EURUSD forms a point in space. It’s 16,000 dimensional space, which is completely incomprehensible mentally, but it’s still a point. That means you can draw a line to it!

We usually talk about correlations in forex by comparing two currency pairs. Most traders know that EUR/JPY and GBP/JPY are correlated. You’re drawing a line between 0 and the EURJPY point, then drawing a line between 0 and the GBPJPY point, then calculating the angle to determine the correlation.

The EURUSD example only lets us draw one line. What if we compare the EURUSD against itself with a lag? That would let us draw two EURUSD lines!

The memory function

Using the EURUSD example, you naturally expect a 100% correlation if you drew the same line twice. It’s the same series of prices. That on its own doesn’t tell us too much.

The idea of autocorrelation is to lag the prices. Right now is 15:00. I can take the prices from 15:00, 14:00 and 13:00 and compare it with a one hour delay. The new delayed series are the prices from 14:00, 13:00 and 12:00. A one hour gap is 99.9% correlated with the original price series.

Michael’s article encouraged me to look at bigger and bigger time lags.

The correlation of EURUSD decreases against itself as the time lag increases.

The number on the horizontal axis is the time lag, done in groups of 100. The vertical axis shows the correlation.

EURUSD loses all its information after around 3,000 hours of data. That’s about the point where the correlation function reaches 0%.

Autocorrelation is often called the memory function. Traders can use this to ask themselves the question, “How far back in time can I go and still obtain useful information?” I can tell you with high confidence that if you’re trading one hour charts, it’s useless to consider anything beyond 3,000 hours.

My personal threshold for significance is a 75% correlation. EURUSD maintains that autocorrelation through 800 bars of data, which you can see on the chart if you zoom in closely.

The take-away for traders is that once you go past around 1,300 bars back, your information rapidly becomes less and less valuable.

What’s going on?

April 22, 2015 by Shaun Overton 9 Comments

It’s been a bumpy month by any definition. We made a ton of money in the aftermath of last month’s Fed announcement, only to give it all back the next week. QB Pro recovered most of the earlier gains, then last week’s drawdown took it all back again. It’s been painful.

The good news is that the new changes to QB Pro are rolled out. Several of you sent in emails asking about new currencies like GBPNZD and AUDCAD appearing in your account. Kudos to you for paying close attention to the trading.

The total currencies traded in the basket is up to 16 pairs. While the max leverage is unchanged at 36:1 (still very, very high), the leverage per pair is only 2.25:1. Future losses like the one from last week will still occur.

The difference is that the size of the positions is reduced by over 2/3. The impact of getting caught in losing trades that are all reflective of USD weakness decreases significantly. We’re now trading a mix of AUD, CAD, CHF, EUR, GBP, JPY, NZD, USD and XAG. No one currency should dominate the performance.

The system also does extremely well on emerging market currencies. I’m holding off on adding RUB, MXN and others until I determine the impact of the spreads on overall profitability. They’d do amazing if we could trade for free!

Short term performance expectations for QB Pro

We’re coming into the summer, which is when the forex market traditionally falls into the doldrums. That’s generally a good thing for QB Pro. The markets whipsaw up and down without really going anywhere.

The alternative is that the Fed hikes rates in June and sends the market into a USD buying frenzy. That’s also good news. Most of the money that QB Pro made over the past 8 months was driven by USD strength. A rate hike would unleash chaos in emerging markets and equities. That’s the kind of condition to push volatility into our new crosses, creating opportunities for us to trade.

QB Pro 2.0 isn’t happening

I’m extremely disappointed. After several thousand dollars in programming expenses, and not to mention the 100+ hours that I spent coding myself, the QB Pro 2.0 change is a wash.

I had a trusted developer audit my code to make sure I wasn’t doing something stupid like trading on future prices or anything. Neither him nor myself caught anything from December until March.

Towards the end of last month, a single line of code ruined it all. One of my key features was deciding when to bail on trades and go the opposite direction. Well, it turned out that I accidentally introduced data snooping into the backtesting platform. I pre-calculated when losing trades occurred to calculate probabilities.

In plain English, my goal was to calculate “If today was a big loser, then do the opposite tomorrow.”

What I accidentally coded was “If tomorrow is a big loser, then do the opposite.” If only that were possible!

I don’t want to muddle up the explanation with code examples. Suffice it to say that the idea didn’t work out when I took away the ability to look into the future.

There are some features of the 2.0 system that I wish to analyze in the coming months, but for now it’s going to have to take a back seat.

What’s next?

My plan is to sit tight for a few weeks to ensure that the new pairs are working as intended. Whenever I am personally satisfied with the system behavior, I intend to increase the amount of capital in my account.

Don’t hold my feet to the fire. This part is a subjective process, so I can’t put a precise time frame on it. If and when I am satisfied – and it’s going very well the first few days – then I will make a decision about increasing my capital at risk.

If and when I choose to increase my capital in the account, I will then re-open QB Pro to new traders.

PS: I hope that the drawdowns encourage some of you to withdraw profits the next time the opportunity presents itself. You don’t want to lose more than you are comfortable risking.

How to Use Machine Learning in Your Trading

April 5, 2015 by Shaun Overton 2 Comments

Machine learning presents many unique and compelling advantages for traders looking for an edge in the market. Just in the last year we have seen a huge amount of resources from the world’s top hedge funds, like Bridgewater Associates, dedicated to exploring these techniques.

While using machine learning or artificial intelligence seems incredibly complex and difficult to implement, there are still ways to leverage their capabilities without needing a PhD in math or science.

In this post, we’ll go through 3 different ways that you can use techniques from machine learning to improve your own trading.

Indicator Selection

One of the most important decisions is deciding on which indicators to use to trade. Whether you are a technical or fundamental trader, or you just use price action to trade, your success is going to be largely dependent on the indicators that you use and how you interpret them.

Luckily, there are many different methods for selecting your indicators and this is known as “feature selection” in the machine-learning world.

Using a Decision Tree to Select Your Indicators

Decision trees are very versatile algorithms that have the benefit of being easily interpretable. Given a large dataset of indicators and the price movement of the asset, a decision tree will find the indicators, and indicator values, that best split the data between price increases and price decreases. Indicators closer to the top of the tree are seen as better predictors than those closer to the bottom of the tree, and following a particular branch will allow you to easily find interdependencies and relationships between the indicators.

The decision tree will also give you a set of rules that you can use to trade based on those indicators, but you must be sure to properly prune the tree and test for overfitting.

The decision tree is a powerful, visual tool that can help you decide which combinations of indicators to trade and at what values to trade them. You can find a tutorial on how to build a strategy with a decision tree here or for a more general guide, in R here is a good resource.

Optimization

Once you have the basis for your strategy, the next step is optimization, or choosing the correct parameter values to maximize your chance of success. Many strategies have a wide variety of parameters, such as indicator settings, the entry and exit conditions, stop loss and take profit levels, and position sizing, that make “brute force” methods of trying every single combination extremely difficult and time consuming, if at all even possible.

Solving these types of problems is another area where machine learning excels.

Optimizing a Strategy Using Genetic Algorithms

Genetic algorithms mimic the process of natural selection by creating a unique set of “child” strategies that contains a mixture of the best “parent’” strategies, with a chance of random mutation.

The process begins by encoding your strategy into an array. For example it could read as something like:

Indicator 1 Period

Indicator 2 Period

Buy Condition

Sell Condition

Risk:Reward Ratio

Where a 50 – 200 period moving average cross, with a 50 pip to 100 pip risk-to-reward ratio would be:

50 – period Moving Average

100 – period Moving Average

Indicator 1 crosses above Indicator 2

Indicator 1 crosses below Indicator 2

50:100

You would then generate a large population of strategies with random variations of these parameters. These strategies all have different combinations of moving average periods, entry and exit conditions, and risk-to-reward ratios.

Next, you would test this population by running each strategy over a test set and selecting the top strategies based on a performance metric of your choice.

Finally, you randomly combine the traits of the top strategies, with a small chance of “mutating” some of the parameters, to create a new generation of “child” strategies. You then repeat the evaluation procedure and once again mate the top performing strategies from this new generation. This leads to a survival of fittest scenario where only the top strategies “survive” to pass along their genes to the next generation

Repeat this process a large number of times or until a certain performance criteria is reached and you are left with a very robust strategy built from generations of the best performing strategies!

You do have to make sure that you select an appropriate performance metric (such as risk-adjusted return) and always test the final strategy over data that wasn’t used to build the strategy to ensure that you aren’t overfitting to a particular data set.

This is a very powerful and robust method that has been successful in a wide variety of applications, including the world of trading. You can find a more detailed description here and a tutorial on how to implement it in R here.

Live Trading

One of the more attractive aspects of machine learning is having an algorithm that is able to learn and adapt to changing market conditions. However, this creates a “blackbox” strategy that, if you do not completely understand how the algorithms work and thoroughly tested it yourself, is very difficult to trust on a live account. Not knowing when or why a strategy is entering a trade can be a scary proposition.

However, there are ways to get the benefits of an intelligent, algorithmic approach while still maintaining transparency and understanding in your strategy.

Association Rule Learning

Association Rule Learning is the process of deriving a set of clear, understandable rules from the patterns uncovered by a machine-learning algorithm.

Algorithms, like the Apriori algorithm, search a dataset of indicators, indicator values, and the resulting price movement to produce a set of conditions; basically “if-then” statements, that lead to the highest-performing results. However, it’s still difficult to know exactly where these rules are coming from, the Apriori algorithm requires a fairly large number of parameters to be tuned and this process does not lend itself well to changing market conditions.

With TRAIDE, we took the process one step further and allow you to see the patterns found by an ensemble of machine-learning algorithms, from which you can create your own trading rules. These rules are then easy to implement and adjusted to changing market conditions, all without requiring any programming or mathematical experience. You are able to get the benefits of using machine-learning algorithms to trade while still maintaining complete transparency, an understanding of your strategy, and including your own domain expertise in your trading.

Using machine learning and artificial intelligence to find an edge in the market does not need to be exclusively owned by only the largest financial institutions. As this technology becomes more accessible and these techniques more common, you too can use machine learning to improve your trading.

Tad Slaff

CEO/Co-founder

Inovance

QB Pro – February

March 2, 2015 by Shaun Overton 1 Comment

The first full month with all clients on the system was awesome. The MAM earned $45,199.63 across all client accounts. My company account made $1,247 on a starting balance of $2,500.

Withdraw profits

Remember that you should withdraw anything above your desired risk level out of the account on a monthly basis. I removed $750 from my personal account on Friday. That’s a total withdrawn profit of $4,997.66 on a single deposit of $2,000. The total lifetime profit on the account is $5,997.66.

Now that I have two months of risk capital completely set aside, I’m finally at a point where I’m totally comfortable upping the balance at risk. The starting balance for March 2015 is $3,000.

Future plans are to risk anywhere from 20-50% of lifetime profits on a monthly basis in order to compound returns without risking a complete wipeout. I’m only doing that because I have $4,000 set aside to recapitalize the account if things go bad.

Risk level

I polled my existing traders and the consensus ranged from leaving the risk level unchanged to dropping the risk by ~25%. I left the leverage unchanged after the poll, but the next week changed my mind.

The account equity was around $120,000 at the time (we started the month at $132,749.27). Watching 5 open positions levered 58:1 on an account that size kept me awake at night. The agreement we all made was that this is a high risk account and everyone accepts the very real risk of loss.

I never expected to raise that much money (my goal was 40k) and it’s seriously altered my thinking. The performance fees even managing 100k are substantial. Like any intelligent businessman, I intend to do everything in my power to continually earn that business.

Translation: I really, really, reaaaaally don’t want to lose your money.

The image above represents an out of sample equity curve for the upcoming trading system running a maximum allowed leverage of 7:1. It’s QB Pro 2.0. The chart shows the profit and loss in dollars over a year and a half period.

The tests were conducted in a custom backtester that I spent about 40 hours developing. A developer has audited the code looking for flukes to explain the performance – we haven’t been able to disprove the results.

We’ve been working on converting the code into MetaTrader 4 for two weeks now. Whenever the code is ready, I’ll launch it in a demo account to confirm identical performance on the same data. I don’t have an ETA on the changes. Trust me when I say it’s a top priority.

Assuming all goes well (fingers crossed!) on backtesting and demo trading, then the next step will be to trade it live on very low risk settings. My goal is to keep your accounts alive and healthy until I’m able to roll out the updated system. That’s why I ultimately lowered the leverage in the MAM accounts by 25%. My company accounts also lowered the leverage by 25% – I’m taking the same risks as my traders.

The historical risks of blowup on 70:1 leverage are undetectable on a historical basis with version 2.0. That’s compared to the 8.5% monthly risk of blowing up that I told everyone before they started trading QB Pro.

Is the expected monthly risk of blowing up 0%? No! Of course not. If a Swiss central bank type of move occurs and you have a leveraged position in the wrong direction, well, it’s lights out for the strategy. No amount of backtesting can guard against that type of loss.

One of the forthcoming, key changes to the strategy is that it won’t be purely mean reverting. It should know when to throw in the towel and start trend trading instead of fighting it. This a critical change and will allow me to fully sleep at night. The trading performance won’t be all or nothing. My expectation is that the risk will decrease dramatically while offering traders the same levels of potential returns whenever it’s available.

Should I bet the farm?

Obviously not. I’ve answered this person’s inquiry and am satisfied that the car is a superfluous asset (it’s just sitting there, so why not sell it?). That said, I get extremely nervous when I get emails like this.

I’m asking because I want to sell my car to make money to fund to account.

The risk of QB Pro is presently unchanged. It’s lower than I originally quoted because the leverage is lowered, but it’s still very substantial. If you’re thinking about adding money into the account, then please do not add more money than you are willing to lose.

That’s true regardless of the trading strategy. If you can’t lose it, then don’t risk it. Period. End of story.

If you’ve been thinking about adding money into the account but it’s not pure “punting money”, i.e., it’s more like an investment, then I’d encourage you to hold off until the version 2.0 is available. I feel more comfortable managing those types of funds and would prefer that you hold off until I have something less historically risky to offer.

MAM Fees

If you have a question on the fees charged in your account, then please email info@quantbar.com. I want to make sure you understand how the fee is calculated if you have any concerns.

Remember that your fees are charged based on the account equity from a month end to end basis. The account equity is taken on Jan 31. We made a profit in February, so the fee is the profitable difference between the equity on Feb 27 and Jan 31, assuming it’s above your high watermark.

It is not based on the closing trade balance. Because QB Pro had open trades on Jan 31, the balance reports you run in MetaTrader will differ from equity profit you made in February. Again, this is due to the fact that open trades existed on Jan 31. The difference should be relatively minor.

If you see a major discrepancy such as 50% of profits or something, then please let me know. I know of one instance where the MAM software didn’t recognize balance changes of a client as deposits, so they were instead marked as profits and he was almost overcharged. If you see an error, I very much want to make it right. I’m dependent on the software’s accuracy and you spotting the discrepancy is the only way it’ll ever be corrected.

Disclaimer: Dominari Online Courses Ltd, of which I am the 100% beneficial owner, does not accept US clients under any circumstance.

Choosing The Right Strategy

November 12, 2014 by Eddie Flower 3 Comments

Traders use a variety of strategies in the markets, all based on two forms of analysis: Fundamental analysis and technical analysis. Although institutions and other large traders often use a combination of these two analytical styles, most independent traders rely on strategies based largely on technical analysis.

Let’s take a look at both analytical styles as they apply to trading forex.

Fundamental analysis

In the stock markets, equities traders are sometimes able to value a company (and therefore predict its share price) if they know all the information about that company. That’s because the share price of the company reflects the value of its known assets. By knowing a company, the equity trader knows what its share price should be.

However, in forex markets using fundamental analysis alone is far less effective, because it’s extremely difficult to value an entire country’s economy in order to predict its currency’s value. Most forex traders use exclusively technical analysis.

When full scale fundamental analysis is applied to forex markets, it’s most often used as a way to predict longer-term trends. And, some traders use data such as news releases in the short term to generate trades or confirm signals. So, along with their mainstay technical analysis, some traders incorporate fundamental data.

Here are some of the fundamental indicators commonly used by forex traders:

★ Non-Farm Payroll

★ Consumer Price Index (CPI)

★ Purchasing Managers Index (PMI)

★ Durable Goods Sales

★ Retail Sales

For best results, savvy traders also pay attention to various meetings of government officials and industry conferences, and other venues where market-moving quotes and commentary can be found.

Meetings are scheduled to discuss inflation, interest rates and other issues that directly affect currency prices. These meetings and conferences are often reported in the industry press before they reach mainstream media. The important event for fundamentals-based forex traders is the Federal Open Market Committee (FOMC) press conference and meeting transcript.

Forex traders can follow meetings and conferences and become highly knowledgeable specialists, and profit by knowing a particular market better than most others.

Technical analysis

Technical analysis is by far the most common basis for forex strategies. Using technical analysis in forex is different than in equities, because the forex time frame is 24 hours worldwide whereas many stocks don’t trade overnight, so their price movements are different.

Traders use a huge variety of individualized systems, often built by knowledgeable EA providers, with many different indicators. Here are just a few of the most common indicators and theories used in technical analysis:

★ Elliott Waves

★ Parabolic SAR

★ Gann Theory

★ Fibonacci Numbers

★ Pivot points

Traders craft many different strategies based on technical analysis, especially by combining multiple indicators. Other developers create trading systems based on finding historical buying and selling patterns that are expected to be repeated.

Developing a personal strategy

Successful forex traders develop and fine-tune their strategies over time. Some traders focus on a particular tool or calculation, while others user a broader approach and experiment with a combination of technical and fundamental analysis.

Many new traders wisely start out by “paper trading” or using a demo account with a forex broker. And, experienced traders almost invariably develop new systems with backtesting before trying them in real time. Lack of experience can cause you to lose your capital, so it’s important to take the time to practice before committing significant money to any new trading system.

Regardless of whether you use technical indicators alone, or incorporate fundamentals as well, if you have the discipline to learn your target markets and trade confidently while carefully managing risks, then your strategy has an excellent chance to succeed.

Do you rely on technical indicators? Fundamental indicators? Or, a combination of both?

Probability Tools For Better Forex Trading

June 16, 2014 by Eddie Flower 25 Comments

In order to be successful, forex traders need to know the basic mathematics of probability. After all, it’s difficult to achieve and maintain trading gains without first having the ability to understand the numbers and measure them.

Many traders use a combination of black box indicators to develop and implement trading rules. Yet, the difference between a “good” trader and a great one is his or her understanding of the metrics and methods for calculating performance and gains.

Probability and statistics are the key to developing, testing and profiting from forex trading. By knowing a few probability tools, it’s easier for traders to set trading goals in mathematical terms, create and operate effective trading strategies, and assess results.

It’s helpful to review the most basic concepts of probability and statistics for forex trading. By understanding the math of probability, you’ll know the logic used by mechanical trading systems and expert advisors (EA).

Normal distribution

The most basic tool of probability in forex trading is the concept of normal distribution. Most natural processes are said to be “normally distributed.”

“Uniform distribution” implies that the probability of a number being anywhere on a continuum is about equal. This is the sort of distribution that would result from artificially spreading objects as evenly as possible across an area, with a uniform amount of spacing between them.

However, instead of a uniform distribution, a currency-pair’s price will likely be found within a certain area at any given time. This is its “normal distribution,” and probability tools can show an approximation of where that price is likely to be found.

Normal distribution offers forex traders predictive power regarding the likelihood that a currency-pair price will reach a certain level during a certain time frame.

Computers use a random-number generator to calculate the means (averages) of forex prices in order to determine their normal distribution.

If a large number of sample prices are checked, the normal distribution will form the shape of a bell curve when plotted graphically. The greater the number of samples, the smoother the curve will be.

The rules of simple averages are helpful to traders, yet the rules of normal distribution offer more useful predictive power. For example, a trader may calculate that the “average” daily price move of a forex pair is, say, 50 pips.

Yet, the normal distribution can also tell the trader the likelihood that a certain daily price move will fall between 30 and 50 pips, or between 50 and 70 pips.

According to the rules of normal distribution and standard deviation, approximately 68% of the samples will be found within one standard deviation of the mean (average), and about 95% will be found within two standard deviations of the mean. Finally, there is a 99.7% likelihood that the sample will fall within three standard deviations of the mean.

Normal distribution and standard deviation functions in expert advisors (EA) and trading systems help forex traders assess the probability that prices may move a certain amount during a given period of time.

Yet, traders should be cautious when using the concept of normal distribution alone for purposes of risk management. Even though the probability of a rare event (such as a price decrease of 50%) may seem low, unforeseen marketplace factors can make the possibility much higher than it appears during normal distribution calculations.

Reliability of analysis depends on quantity and quality of data

When modelling normal distribution curves, the amount and quality of input price data is very important. The greater the number of samples, the smoother the curve will be. Also, to avoid calculation errors resulting from insufficient data, it’s important that each calculation be based on at least thirty samples.

So, for testing a forex-trading strategy by estimating the results from sample trades, the system developer must analyze at least 30 trades in order to reach statistically-reliable conclusions regarding the parameters being tested. Likewise, the results from a study of 500 trades are more reliable than those from an analysis of only 50 trades.

Dispersion and mathematical expectation to estimate risk

For forex traders, the most important characteristics of a distribution are its mathematical expectation and dispersion. Mathematical expectation for a series of trades is easy to calculate: Just add up all the trade results and divide that amount by the number of trades.

If the trading system is profitable, then the mathematical expectation is positive. If the mathematical expectation is negative, the system is losing on average.

The relative steepness or flatness of the distribution curve is shown by measuring the spread or dispersion of price values within the area of mathematical expectation. Typically, the mathematical expectation for any randomly-distributed value is described as M(X).

So, dispersion can be defined as D(X) = M[(X-M(X)]².

And, a dispersion’s square root is called its standard deviation, shown in mathematical shorthand as sigma (σ).

Dispersion and standard deviation are critically important for risk management in forex trading systems. The higher the value of the standard deviation, the higher will be the potential drawdown, and the higher the risk. Likewise, the lower the value for standard deviation, the lower will be the drawdown while trading the system.

For example, below is a sample risk assessment for a test of a forex trading system:

Trade Number X (Trade Gain or Loss)

1 -17.08
2 -41.00
3 147.80
4 -159.97
5 216.97
6 98.30
7 -87.75
8 -27.83
9 12.34
10 48.14
11 -60.92
12 10.62
13 -125.43
14 -27.82
15 88.02
16 32.94
17 54.82
18 -160.10
19 -83.37
20 118.40
21 145.65
22 48.43
23 77.39
24 57.49
25 67.75
26 -127.10
27 -70.19
28 -127.60
29 31.31
30 -12.55

In the above example based on the minimum number of thirty trades for an adequate sample, it’s important to note that the mathematical expectation is positive, so the forex trading strategy is indeed profitable.

However, the standard deviation is high, so in order to earn each dollar the trader is risking a much larger amount; this system carries significant risk.

Here’s the rest of the math: To determine the mathematical expectation for this group of trades, add together all the trades’ gains and losses, then divide by 30. This is the mean value M(X) for all the trades. In this case, it equals an average gain of $4.26 per trade. Thus far, the system looks promising.

Next, to calculate the standard deviation of the dispersion, the above average $4.26 is subtracted from the results of each trade, then it’s squared, and the sum of all these squares is added together. The sum is divided by 29, which is the total number of trades minus 1.

By using the formula for Dispersion of (X) = M[(X-M(X)]² given above, here’s a check of the calculation from the first trade in our example:

Trade 1: -17.08 – 4.26 = -21.34, and (-21.34)² = 455.39

The same calculation is performed for each trade in the test series. In this example, the dispersion over the series equals 9,353.62 and by definition its square root equals the standard deviation (σ), which in this case is $96.71.

Thus the forex trader sees that the risk for this particular system is fairly high: The mathematical expectation is indeed positive, with a mean profit of $4.26 per trade, yet the standard deviation is high when compared with that profit.

It can be seen that the trader is risking about $96.71 for each opportunity to earn $4.26 in profit. This risk may be acceptable, or the trader may choose to modify the system in search of lower risk.

Z-score

Beyond the riskiness of a particular trading system, forex traders can also use normal distribution and standard deviation to calculate the Z-score, which indicates how often profitable trades will occur in relation to losing trades.

During the process of developing a winning forex trading system, the trader may wonder how many of the profitable trades seen during testing were “random,” and how many consecutive losing trades must be tolerated in order to achieve winning trades.

For example, let’s assume the average expected profit from a given forex trading system is four times less than the expected loss amount from each stop-loss order triggered while trading this system.

Some traders may assume that the system will win over time, as long as there is an average of at least one profitable trade for each four losing trades. Yet, depending upon the distribution of wins and losses, during real-world trading this system may draw down too deeply to recover in time for the next winner.

Normal distribution can be used to generate a Z-score, sometimes called a standard score, which lets traders estimate not only the ratio of wins to losses, but also how many wins/losses are likely to occur consecutively.

A positive Z-score represents a value above the mean, and a negative Z-score represents a value below the mean. To obtain this value, the trader subtracts the population mean from an individual raw value then divides the difference by the population standard deviation.

The basic standard score calculation for a raw score designated as x is:

Z = (x – μ) / σ

Where μ is the population mean and σ is the population standard deviation. It’s important to understand that calculating the Z score requires that the trader know the parameters of the population, not merely the characteristics of a sample taken from that population.

Z represents the distance between the population mean and the raw score, expressed in units of the standard deviation. So, for a forex trading system:

Z = [N x (R – 0.5) – P] / [(P x (P – N)] / (N – 1)]½

When:

N is the total number of trades during a series;
R is the total number of series of winning and losing trades;
P equals 2 x W x L
W is the total number of winning trades during a series
L is the total number of losing trades during a series

Individual series can be represented by a consecutive sequence of pluses or minuses (for example ++++ or —). R counts the number of such series.

Z can offer an assessment of whether a forex trading system is operating on-target, or how far off-target it may be.

Just as importantly, a trader can use Z-score to determine whether a trading system contains fewer or greater series of winners and losers than expected from a random sequence of trades– In other words, whether the outcomes of consecutive trades are dependent upon each other.

If the Z-score is near 0, then the distribution of trade results is near the normal distribution. The score of a sequence of trades may indicate a dependency between the results of those trades.

This is because a normal random value will deviate from the average value by not more than three sigma (3 x σ) with a certainty of 99.7%. Whether the Z value is positive or negative will inform the trader about the type of dependence: A positive Z value indicates that the profitable trade will be followed by a loser.

And, positive Z indicates that the profitable trade will be followed by another profitable one, and a loser will be followed by another loss. This observed dependency lets the forex trader vary the position sizes for individual trades in order to help manage risk.

Sharpe Ratio

The Sharpe Ratio, or reward-to-variability ratio, is one of the most valuable probability tools for forex traders. As with the methods described above, it relies on applying the concepts of normal distribution and standard deviation. It gives traders a method to check the performance of a trading system by adjusting for risk.

The first step is to calculate the Holding Period Returns (HPR). For example, a trade which resulted in a profit of 10% has a HPR calculated as 1 + 0.10 = 1.10 while a trade which loses 10% is calculated as 1 – 0.10 = 0.90.

Likewise, HPR can be calculated by dividing the after-trade balance amount by the before-trade amount. The Average Holding Period Returns (AHPR) is then calculated by adding up all individual holding-period returns, then dividing by the number of trades.

AHPR by itself produces an arithmetic average which may not properly estimate the performance of a forex trading system over time. Instead, a trading system’s investment efficiency can be more closely estimated by using the Sharpe Ratio, which shows how AHPR minus the risk-free rate of long-term investment returns relates to the standard deviation of the trading system.

Sharpe Ratio = [AHPR – (1 + RFR)] / SD

When AHPR is the average holding period return, RFR is the risk-free rate of return from “safe” investments such as bank interest rates or long-term T-bond rates, and SD is the standard deviation.

Since more than 99% of all random values will fall within a distance of ±3σ around the mean value of M(X) for a given trading system, the higher the Sharpe Ratio, the more efficient the trading system.

For example, if the Sharpe Ratio for normally-distributed trade results is 3, it indicates that the probability of losing is less than 1% per trade, according to the 3-sigma rule.

The concepts of normal distribution, dispersion, Z-score and Sharpe Ratio are already incorporated into the logarithms of EAs and mechanical trading systems, and their usefulness is invisible to most traders.

Yet, by knowing how these basic probability tools work, forex traders can have a deeper understanding of how automated systems perform their functions, and thereby enhance the probability of winning trades.

Are you currently using probability tools to increase your own chance for success?

System Parameter Permutation Beats Data Mining Bias

May 16, 2014 by Eddie Flower 6 Comments

Recently, an interesting new perspective has emerged regarding trading-system development. The National Association of Active Investment Managers (NAAIM) has just announced a $10,000 Wagner Award for Dave Walton of StatisTrade for his pioneering work in exploring a new method for trading-system development, which he calls System Parameter Permutation (SPP).

It’s a well-merited prize, since SPP neatly solves the age-old issue of data mining bias.

This article will summarize the implications of System Parameter Permutation, and the entire 33-page paper can be downloaded here. SSP is exciting because it opens up an entirely new horizon in trading-system development. It has the power to help boost quant traders to the next level, so it’s worth taking a careful look.

Many traders have been frustrated by trading system performance that doesn’t live up to expectations, and it seems that data-mining bias (DMB) is mostly to blame. The effects of DMB are misunderstood and underestimated by most traders and system developers.

Even well-seasoned institutional quants fall prey to the effects of DMB, and very few traders have been able to overcome it. The value of SPP lies in its power to mitigate DMB and allow developers to effectively design mechanical trading systems according to their probability of future success, not past success.

System Parameter Permutation seems the perfect tool for active traders who only use mechanical trading systems, and it’s said to be effective for trading systems using any time frame. It helps traders answer these two fundamental questions:

What is the long-term performance expectation for a given trading system?

What is the worst-case short-term performance (i.e. drawdown) that must be tolerated in order to achieve that long-term performance expectation?

Defining the parameters for System Parameter Permutation

SPP is intended to be used only with fully-mechanical trading systems using quantitative algorithmic trading rules. So, it’s perfect for today’s traders.

In order to keep the explanation in his award-winning paper simple and easy-to-follow, the author keeps the definitions and instructions simple.

For the sake of simplicity, the author’s primary study of System Parameter Permutation was based on ETFs using “long” trades only, since shorts would complicate the input assumptions regarding borrowed shares, callbacks, dividends, and interest charges. The author used a historical simulation period of about seven and a half years. Commissions and other costs were calculated at typical levels.

The results of the SPP-fueled simulations were limited to four metrics: Compounded annual returns, maximum drawdown, annualized information ratio, and the annualized standard deviation of daily returns. To compare the effectiveness of SPP in predicting trading-system performance, the simulations were also checked using legacy out-of-sample (OOS) methods.

How to mitigate data mining bias?

Data mining bias, also called over-optimization, curve-fitting or over-fitting, is a trading-system developer’s worst enemy.

Most developers build DMB into their systems without understanding what it is and how it poisons systems. As a result, their systems are doomed to perform worse in the future than historical back-testing would have them believe.

The issues caused by DMB result from the preconditions inherent during the system-development process, namely randomness and the multiple-comparison approach. DMB causes the resulting performance metrics to be inflated on the side of success.

Data mining in order to find the best set of trading rules means that developers end up with only the best results of historical performance, which are not the same as the rules for best future performance.

In fact, the law of statistical regression toward the mean indicates that the “good luck” of the past is unlikely to be repeated in the future when using that same set of trading rules.

Savvy system developers use various methods to mitigate DMB, including cross-validation by examining system performance after regression to the mean has already occurred. Or, they may attempt to compensate for bias, or multiply their results by a deflation factor in hopes of neutralizing the effects of data mining bias.

Thus, DMB creates systematic, difficult-to-quantify errors by focusing on the results from good luck, while ignoring the likelihood of bad luck. In contrast, System Parameter Permutation accounts for the occurrence of both good and bad luck.

Using System Parameter Permutation to determine system performance

The limitations of DMB mitigation mean that developers often suffer from inaccurate predictions about trading system performance going forward. In contrast, SPP offers a handy method for accurately estimating performance as well as a way to test the statistical significance of results, free of data mining bias.

Best of all, SPP works very well alongside the standard optimization tools used in commercially-available trading software packages.

Beyond avoiding the issues caused by DMB, System Parameter Permutation also enables traders and developers to objectively check the performance of a trading system’s long-term “edge.” And, it allows them to determine a system’s short-term “worst case” performance. For systems already in use, SPP helps determine when and whether the old rules no longer work.

How SPP works

System Parameter Permutation works by generating a large set of sampling distributions of a system’s performance metrics. Each individual point in the distribution results from a historical simulation of portfolio effects. From these sampling distributions, developers and traders can evaluate the system based on any desired performance metrics.

SPP uses the statistics from these sampling distributions to estimate the system’s performance as well as providing measurements of the statistical significance of the distributions.

In contrast to standard optimization methods, System Parameter Permutation does not merely pick a single “ideal” set of parameters to be used to create a set of trading rules that would have been historically successful. Instead, SPP uses all the performance data for all sets of parameters that were evaluated during the optimization.

For each metric, SPP generates a sampling distribution that incorporates the hypothetical-trade results from all combinations of parameters. This approach is far different from DMB compensation or cross-validation, since those methods use only the result of a single “best” set of trades in order to predict the system’s performance.

SPP relies on the median performance in each distribution for several reasons: (1) The median is not influenced by DMB; (2) the shape of the distribution curve is unimportant; and (3) the median is unaffected by outliers.

The steps to SPP

To generate the performance-metric sampling distributions, a developer must first determine an appropriate set of parameter ranges for the trading system, then create a sampling distribution. SPP is based on the following steps:

1. Determine the scan ranges of the parameters for the system;

2. Divide each individual parameter scan range into the desired number of observation points;

3. Perform exhaustive optimization of every possible combination of parameter values using a historical simulation during the chosen time period;

4. Combine together these simulated results from each and every variant to build a sampling distribution regarding each desired performance metric, such as compound annual return (CAR) and maximum drawdown.

With the System Parameter Permutation method, each point in the sampling distribution is derived from the simulation run according to an individual system variant. Depending upon the time and computing power available to the trading system developer, any number of performance metrics may be checked.

The cumulative distribution function (CDF) is then examined for each metric, in order to estimate system performance and arrive at statistical inferences.

It’s critically important to choose SPP parameter scan ranges carefully to avoid data mining bias. For example, if SPP is repeated several times using different scan ranges in search of better results, then it may become infected by a positive bias.

Using SPP to estimate the long-term performance of a trading system

The most important question to be answered by System Parameter Permutation regards the long-term performance of a given trading system. The most accurate long-run estimates are obtained from the sampling distributions based on all available market data. As indicated above, the median value of the distribution offers the best performance estimate for each metric.

Also, traders and system developers can test the statistical significance of these performance estimates, whether in terms of absolute returns or measured against a benchmark. When using SPP, confidence levels and p-values can be estimated directly by using the CDF.

System Parameter Permutation also estimates short-term and worst-case performance

Even though trading system developers are naturally focused on a system’s potential for long-term gains, SPP is also useful for estimating the drawdowns which must be endured in order to achieve those gains.

• All market data are divided into blocks of time equal to the length of the chosen short-term time period (t);

• These time blocks may overlap with adjoining blocks, depending on the time-frame chosen for trading signals, e.g. hourly within a day, or monthly within a year;

• The result is a number (m) of time blocks;

• The above-listed Steps 1 through 4 are performed for all (m) time blocks individually;

As with estimating the long-term periods, the length of the short-term periods depends on the trader’s preferences and trading objectives.

So, if a trading system has (n) combinations of parameters, in total (m x n) optimization permutations are calculated from a historical time period t to generate a sampling distribution for each metric being examined during the chosen short-term time-frame. Just as with the long-term performance study, the trader may choose any number of metrics for short-term study.

Sampling distributions from the short-term SPP process produce far more individual samples that have a higher variation than those generated by the long-term SPP process. However, each distribution has a shorter time-frame and therefore represents fewer closed trades in each sample.

So, the standard error of each short-term sample is greater. When the standard error increases, the variation of the sampling distribution likewise increases.

Armed with these sampling distributions, a trader can make probability-driven decisions about whether to trade a system or not. First, the trader decides on a probability level that he or she considers improbable yet tolerable as a worst-case scenario, perhaps a 1% to 5% loss.

Or, the trader may determine the worst case in view of the least-favorable-yet-most-tolerable performance level. The CDF of the short-term sampling distribution is then examined according to the chosen level of performance desired.

If the trader is unable or unwilling to tolerate the indicated probability of loss, then he or she should not trade that system. Thus, System Parameter Permutation provides traders with an objective risk-assessment and risk-management tool.

Why SPP works so well

System Parameter Permutation works because it leverages the statistical law regarding regression to the mean, instead of ignoring it as most other system-optimization methods do. As well, SPP takes advantage of modern computing power to quickly extract and utilize the maximum amount of information from all available market data.

Traditional optimization methods calculate performance metrics from the single best set of trades discovered during optimization. Yet, random resampling can result in problematic assumptions and data mining bias.

By using a large number of parameter-value combinations, SPP estimates the effect of mean-regression. Using all the available market data ensures that the system is exposed to the widest range of market conditions, and the results contain the smallest possible standard error.

In contrast to random resampling, when using SPP the random variations result from changing the entry and exit rules for hypothetical trades using actual market data. Thus, SPP accounts for the effects of both completed trades as well as randomly-skipped trades.

In other words, System Parameter Permutation lets trading system developers explore aspects of a system which otherwise would remain hidden yet possible during real trading.

SPP opens new doors for mechanical traders

Traditionally, developers have built their trading systems according to estimates of performance based on single-point optimization and measures of statistical significance inferred from a limited number of trades. Yet, System Parameter Permutation provides traders with more useful sampling distributions of performance metrics, and it accounts for all historical trades, whether or not they actually occurred.

SPP can help traders confidently predict both long-term gains as well as short-term drawdowns. Best of all, it can help quantitative trading system developers avoid data mining bias which robs them of both their confidence and profits.

What methods do you use to avoid curve fitting a system?