5
Optimize the Real-Time Performance of Pattern Scanners
Second installment on using the pattern scanner more effectively. You can find the first one here.
There are many techniques to improve scanning performance significantly by turning on/off certain options within the pattern scanner. There are also some global settings within NeoTicker that affect scanning in real-time. I will now discuss each item separately.
Pre-Filtering Process in Scanning
The Ranking setting of Last Price, Volume, etc. that depends on the latest real-time fields from your data vendor, all require an extra step of requesting these data from your data feed. If the symbol is not currently streamed by your data feed, then NeoTicker has to take some extra time to connect the symbol, getting the data, and then, disconnect the symbol to keep you from going over the symbol limit of your data feed subscription.
Choosing Ranking settings of Indicator or Pattern Conf%, pattern scanner will smartly skip this step of looking up those real-time fields.
When you see the Pre-Filtering message on the pattern scanner, it is this part of the scanner working. Thus you can tell easily if this part is skipped or not.
By a similar reason, in the Pattern Scanner Setup window, if you use the Current tab or Lookup tab filtering options, then the scanner will need to access the real-time fields during a scan. If all you need is simply scanning based on indicators, then avoid using these filtering options to improve the scanning performance.
Many of the filtering options can be implemented within context of an indicator. In case you wanted to scan for these criteria, then folding them into one indicator and scanning on indicators only will allow you to skip the Pre-filtering process.
This factor alone can cut the scanning time by half.
Use RAM Cache Properly
RAM Cache has 2 purposes for real-time usage in NeoTicker.
First, quote formulas in Quote Window, Dynamic Table, etc., and RTD/DDE connection to Excel all need some form of invisible data series and indicators loaded in the background and updated in real-time. RAM Cache is responsible for managing these series and their real-time updates. When you check the RAM Cache status those columns on data series and indicators show you how many series are defined under each symbol that you have some form of calculations going on in the background.
Second, there is the preloading of historical data in the 3 basic data type (tick, minute, daily). These preloaded data serve the purpose of fast data loading should you need them in a chart, creating new formula columns, etc. By utilizing this part of the RAM Cache, you can increase your scanner performance significantly if you scan by indicators only.
To use pattern scanner with RAM Cache, first you have to look into the Cache Manager, under the RAM Cache tab, open the RAM Cache Options window by pressing the Options button. In the opened RAM Cache Options window, under Smart Tools, make sure the setting is set to Disabled. Smart insertion of symbols into the RAM Cache is designed for people who track a very small number of symbols and that they are likely to switch among those symbols in the charts, etc. very frequently.
Next is to set the Preload Symbol List under the tab with same name. By providing the complete list of symbols that you are going to scan in real-time, the RAM Cache will preload all these symbols’ historical data. That way, the pattern scanners will no longer need to access your hard disk for any data, thus improving the speed of your scan.
Last part here is to change the Days To Load tab settings. Enable only those data types you need and set the number of days to load based on the indicator and/or data series requirements you have in your scanners.
For example, you are scanning on 15-minute timeframe, using 100 period simple moving average as one of the indicators. Then you need 5 days of data for the pattern scanner to function properly. So, if you want the scanner to save time by loading data from RAM Cache only, then you need to set the RAM Cache to preload minute data for 5 days.
Several factors can affect how much data you can preload.
1. If your data feed subscription will limit the number of symbols that you can stream in real-time. Thus you cannot preload more symbols than your subscription is allowed.
2. The amount of RAM you have with your computer can stop you from loading more historical data. NeoTicker can utilize upto 2G RAM within the Windows environment, thus if your computer has more RAM than that, you will be able to maximize your ability to preload data.
3. The number of days of data to preload can drastically affect the amount of RAM it is going to use. Pre-loading 200 symbols with 5 days of minute data may max out your computer’s resources, then when you want to pre-load 500 symbols you will have to cut down the number of days of data to preload.
Using RAM Cache properly can often cut the scanning time by half if it is based on indicators only.
Optimize Heavy Calculations with IDL indicators or Other Means
Not all indicators are created equal. When you use a script based indicator, it works much slower (sometimes 10x slower) than the formulas. And formula indicators are slower than IDL indicators.
So if you are using script indicators initially to build your scan, then you may want to rewrite them in either formula or port the indicator into IDL. The performance boost varies depending on the type of calculations you are carrying out.
If the performance you are getting is acceptable with formula indicators, then there is no need to push the envelope here as maintaining scanners with formula indicators is easier and more flexible than using specialized IDL indicators.
Consider Using Chain Scanning
When you have some indicator calculations that are extremely time consuming, then consider breaking up your scans to multiple parts.
For example, you have a scan that look for a price pattern and then certain indicator thresholds. By scanning for the price pattern part only, you can reduces the number of symbols from the original set into a much smaller set. And then utilize the pattern scanner chain scan option to send this smaller set of symbols into a scanner with only the indicator calculations.
Think of it this way. If you are looking for a short term bounce in the afternoon, and that you are using 30-minute bars looking for candidates to go long, then a scan on 300 symbols for the price pattern of a candlestick hammer formation would probably give you 30 to 40 symbols at 1 pm, right after lunch. Then sending those results into another scan for certain 5-min bar based criteria like indicator readings, etc. is better and faster than cramping all calculations into a single scan.
Having Invalid Symbols in the Scan Can Impact Performance
An issue seldom mentioned anywhere is that if you have an invalid symbol in your symbol list, it can slow down the pattern scanner a lot. The reason is that your data feed may not response well when you keep requesting for data from an invalid symbol.
Invalid symbols are common, especially for stocks, due to the fact that there are merger and aquisition activities, symbol change due to restructuring, symbol change due to moving from one exchange to another, delisted by the exchange, etc.
To keep your pattern scanner running in top performance, you may want to check for invalid symbols in your symbol lists once a week to remove or replace those invalid symbols.
Next installment on pattern scanner will focus on developing scans with pattern scanner.