Downloading and Retaining Data

Automatic Backfill

Investor/RT's automatic backfill mode is designed to keep daily and intra-day historical data up to date automatically without user involvement. Generally speaking, if you track fewer than 100 symbols, all historical data management should simply be done for you; there should not be a need to manually perform data downloading operations. It is important to understand the role of the intraday monitoring status in these automated download operations managed by Investor/RT.

Why/when should I perform a manual (partial or full) data download operation ?

Here are a few typical situations

Case 1: Users who track many hundreds or thousands of symbols are exceptional cases where a higher degree of user involvement is required to keep your historical data current. Due to symbol limit constraints, the automatic backfill mode might have been deactivated, requiring some manual involvement in the data download management

Case 2: you monitor a limited number of futures and have the automatic backfill mode activated. However, due to some external event or exceptional (such as an internet cut, a temporary issue with your live data provider, computer going into sleep mode, etc..), you just observed that you are missing some data on a chart

For the record, there are 2 types of data downloads

• either a partial data download, ie you request to download data for a specific period of time (current session / last 2 days, etc) so that you may fill a "data" gap (for one one the reason just mentionned in the second case), but this will keep the existing data from the database for the prior period. Such data download may only takes a few seconds to perform
• either a full data download. This is essentially a request to replace all of the data you have on file with fresh data from the historical data. When you add a brand new symbol to Investor/RT, for example, a full download is done automatically. If you just adjusted and increased the data retention settings for a particular instrument, then you will need to perform a full data download.

How do I perform a manual Data Download operation?

The different options are discussed on this dedicated page. The vast majority of the time, you will be using one of the Data download menu options that can be accessed through a right click on a chart, and which offers both options for a partial or Full data download.

 

Please note that this limitation is only for tick data (ie not for 1 min bar data). However, this means that any partial or full download operations, requesting more than one week of tick data, should be performed outside of the US RTH hours

 

Retaining longer term Tick data

Some users require more tick data than others. To accomplish this, you must first setup Investor/RT to retain larger amounts of tick data by going to File > Preferences > Data > Historical Data. The top section of the window labeled Default Historical Data Retention Settings affects all symbols and sets the default amount of data that will be retained each time Investor/RT is closed. The middle section of the window allows you to override the retention settings on a symbol by symbol basis. If you require more tick data to be kept, it is recommended that you increase the Tick Data retention settings for the individual symbols you need as opposed to doing it for all symbols which might increase the size of your database unnecessarily.

Now that you've set Investor/RT up to retain more tick data, the second step is to download the tick data. Press Alt-6 to open a Download Window and enter the number of Calendar days you require. Remember that the maximum amount of Tick data you can download during Regular Trading Hours is 8 Calendar Days. After the Regular Trading Hours session closes, you can download up to 180 Calendar Days.

Historical Data Management

Investor/RT has data retention settings that control how much tick data, one minute interval data, and daily bar data is retained for the symbols you track. These settings have reasonable defaults but you may wish to change them to retain more or less data depending on your requirements. Investor/RT is very flexible in this regard; there is a global setting for tick, one minute, and daily data retention, or you may override particular symbols for which you want a setting different than the default. These settings are adjusted in the Historical Data preferences window ( Setup > Preferences > Historical Data). The global setting is at the top; select any symbol and specify an override if you wish in the middle section of the window. Each trading day, Investor/RT collects or may download tick data for the trades for each symbol that you monitor intra-day. At the end of each trading session, Investor/RT will examine your historical data retention settings and purge any data that is beyond (older than) your stated preferences. For example, if you call for 5 days of tick data and 100 days of one minute data for a symbol, and Investor/RT finds that you now have 6 days of tick data, the oldest day of tick data is purged and replaced with one-minute data. Any one-minute data older than 100 trading days back will also be purged. Investor/RT historical data management is designed to keep the data you need on hand, automatically trimming the size of the database after each trading day in accordance with your settings.