Question

Answer

In version 10.1, Investor/RT introduced Smart Compression of Tick Data.  This is an option that can be turned in by going to "Setup | Preferences | Historical Data" from the main menu and checking the checkbox titled "Use Smart Compression of Tick Data".

During our testing, Smart Compression reduce the tick data (number of ticks) consistently by over 85%.  Users can now store approximately 7 times as much tick data in the same amount of space.  This compression has a number of very positive impacts on the performance and capability of Investor/RT: users can store more tick data, charts load and process quicker, databases verify and backup quicker, faster backtesting.  The list goes on and on.  Users that once store 3 days of tick data will now be able to store 20 days of tick data using less space in the database.

When Smart Compression is on, very few aspects of Investor/RT are adversely affected.  The one aspect that will affect most users will be tickbar charts.  If tickbar charts are a big part of your analysis, then Smart Compression might not be for you as tickbar charts will not function properly with compressed ticks.  However, ALL other periodicities (Volume Bars, Range Bars, Second Bars, etc) will be identical with Smart Compression turned on.  Also, advanced volume functionality like the Profile Indicator (Volume at Price / Delta at Price), the Volume Breakdown Indicator (Delta per Bar), and VWAP, will also be unaffected by  Smart Compression.  Certain results of the Volume Breakdown Indicator can be affected.  For a complete list of aspects of Investor/RT that might be effected by Smart Compression, click here.

When Smart Compression is on, ticks are stored in the database in a compressed format by combining similar ticks which share the price, bid, ask, and timestamp.  These similar consecutive ticks are stored as one tick with a volume that represents the cumulative volume of the component ticks.  Once this option is turned on, existing historical data is compressed, and all new tick data that flows into the database will be compressed.  This includes ticks that result from downloads/backfills and live ticks that are flushed to the database at the end of each session and each time the user quits the program.

Below is an example of a block of ticks that are displayed in both compressed and uncompressed (raw tick) formats.  Notice the first tick matches, then the price changes and the next 3 uncompressed ticks became on one compressed tick.  Then the bid and offer changed and the next 22 raw ticks became one compressed tick with a cumulative volume of 220 (sum of the size of the 22 raw ticks).

What aspects of Investor/RT can be adversely affected when Smart Compression is turned on?

In a nutshell, any aspect of of Investor/RT that relies knowing the actual number of raw ticks over a given period, will not work accurately with Smart Compression.  These include...

• Tickbar Charts or Periodicities
• Specific Volume Breakdown Indicator Options including:
  - VB is setup to be based on "Trades" instead of "Volume" (top right setting in Prefs window)
  - VB setup with any "Result" that is based on tick counts including:
            - Total Traders/Ticks
            - Average Size
            - Total Traders/Ticks
            - Big Trader %
            - Average Buy Size
            - Average Sell Size
            - Buy Traders/Ticks
            - Sell Traders/Ticks
  - VB setup with a volume filter "Consider Trades with Volume" above, below, or between a given level.

Analysis of Database Size Requirements for ES Data
1-Minute, Compressed Ticks, and Uncompressed Ticks
The Investor/RT database is limited to storing 2gb of intraday data (combined minute data and tick data on all symbols).  Exceeding this limitation can result in database corruption.  A recent analysis of data on the ES revealed that, on average, one day of 1-minute data required 0.072mb of database space, a day of compressed tick data required 0.83mb, and a day of uncompressed tick data required 6.17mb.  Based on these numbers, the Investor/RT database is capable of storing 27,397 days of 1-minute ES data, 2,410 days of compressed tick data on ES, or 324 days of uncompressed tick data on ES.  As a specific example, if you were retaining uncompressed tick data on 10 symbols with comparable volume to ES, you should limit your tick retention to 30 days.  However, compressed ticks would allow that limit to be raised to to 240 days.

Keep in mind, if you are retaining, for example, 1000 days of 1-minute data and 10 days of uncompressed tick data, the 1-minute data will take approximately 70mb of database space while the tick data will take 60mb of database space, for a total of 130mb of database space for each symbol setup with these retention settings (assuming similar volume to ES).

For more information on fine-tuning Investor/RT for optimal data retention, review the Investor/RT Tune-Up video.