In version 10.1, Investor/RT introduced Smart Compression of Tick Data.
This is an option that can be turned in by going to "Setup | Preferences
| Historical Data" from the main menu and checking the checkbox titled
"Use Smart Compression of Tick Data".
During our testing, Smart
Compression reduce the tick data (number of ticks) consistently by over
85%. Users can now store approximately 7 times as much tick data
in the same amount of space. This compression has a number of very
positive impacts on the performance and capability of Investor/RT: users
can store more tick data, charts load and process quicker, databases
verify and backup quicker, faster backtesting. The list goes on
and on. Users that once store 3 days of tick data will now be able
to store 20 days of tick data using less space in the database.
When Smart Compression is on, very few aspects of Investor/RT are
adversely affected. The one aspect that will affect most users
will be tickbar charts. If tickbar charts are a big part of
your analysis, then Smart Compression might not be for you as
tickbar charts will not function properly with compressed ticks.
However, ALL other periodicities (Volume Bars, Range Bars, Second Bars,
etc) will be identical with Smart Compression turned on. Also,
advanced volume functionality like the Profile Indicator (Volume at
Price / Delta at Price), the Volume Breakdown Indicator (Delta per Bar),
and VWAP, will also be unaffected by Smart Compression.
Certain results of the Volume Breakdown Indicator can be affected.
For a complete list of aspects of Investor/RT that might be effected by
Smart Compression, click
When Smart Compression is on, ticks are stored in the database in a
compressed format by combining similar ticks which share the price, bid,
ask, and timestamp. These similar consecutive ticks are stored as
one tick with a volume that represents the cumulative volume of the
component ticks. Once this option is turned on, existing
historical data is compressed, and all new tick data
that flows into the database will be compressed. This includes
ticks that result from downloads/backfills and live ticks that are
flushed to the database at the end of each session and each time the
user quits the program.
Below is an example of a block of ticks that are displayed in both
compressed and uncompressed (raw tick) formats. Notice the first
tick matches, then the price changes and the next 3 uncompressed ticks
became on one compressed tick. Then the bid and offer changed and
the next 22 raw ticks became one compressed tick with a cumulative
volume of 220 (sum of the size of the 22 raw ticks).
What aspects of Investor/RT can be adversely affected when Smart
Compression is turned on?
In a nutshell, any aspect of of Investor/RT that relies knowing the
actual number of raw ticks over a given period, will not work accurately
with Smart Compression. These include...
- Tickbar Charts or Periodicities
- Specific Volume Breakdown Indicator Options including:
- VB is setup to be based on "Trades" instead of "Volume" (top right
setting in Prefs window)
- VB setup with any "Result" that is based on tick counts including:
- Total Traders/Ticks
- Average Size
- Total Traders/Ticks
- Big Trader %
- Average Buy Size
- Average Sell Size
- Buy Traders/Ticks
- Sell Traders/Ticks
- VB setup with a volume filter "Consider Trades with Volume" above,
below, or between a given level.
Analysis of Database Size Requirements for ES Data
1-Minute, Compressed Ticks, and Uncompressed Ticks
The Investor/RT database is limited to storing 2gb of intraday data
(combined minute data and tick data on all symbols). Exceeding
this limitation can result in database corruption. A recent
analysis of data on the ES revealed that, on average, one day of
1-minute data required 0.072mb of database space, a day of compressed
tick data required 0.83mb, and a day of uncompressed tick data required
6.17mb. Based on these numbers, the Investor/RT database is
capable of storing 27,397 days of 1-minute ES data, 2,410 days of
compressed tick data on ES, or 324 days of uncompressed tick data on ES.
As a specific example, if you were retaining uncompressed tick data on
10 symbols with comparable volume to ES, you should limit your tick
retention to 30 days. However, compressed ticks would allow that
limit to be raised to to 240 days.
||Database Size / Day (ES)
||Number of Days Allowable
Keep in mind, if you are retaining, for example, 1000 days of
1-minute data and 10 days of uncompressed tick data, the 1-minute data
will take approximately 70mb of database space while the tick data will
take 60mb of database space, for a total of 130mb of database space for
each symbol setup with these retention settings (assuming similar volume
For more information on fine-tuning Investor/RT for optimal data
retention, review the
Investor/RT Tune-Up video.