Home                           

 Try Investor/RT             

 Investor/RT Tour           

 Getting Started              

 What's New                  

 Testimonials                 



MASTER INDEX
  SEARCH
  CHARTS
   MAIN TOOLBAR
   SETTING PREFERENCES
   TECHNICAL INDICATORS
   WHAT'S NEW
  ALARMS
  BACKTESTING
  CONFIRMING TRADES
   CUSTOM COLUMNS
  CUSTOM INDICATORS
  CUSTOM INSTRUMENTS
  CUSTOM PRICES
  DATA FEEDS
   DATABASE
   DOWNLOADING DATA
   FEED STATUS
   HEARTBEAT
   HELP
   MULTI-LINKING
   MULTI-PERIODICITY
  INSTRUMENT SETUP
  NASDAQ LEVEL II
  NEWS TICKER
  NEWS BROWSER
   OPTIMIZATION
  OPTIONS ANALYTICS
  PLACING TRADES
   PORTFOLIOS
  PRESETS
  PRINTING
   QUICK QUOTES
  QUOTE PAGES
   REALIZATION
  RTL LANGUAGE
   SCANS
   SCHEDULES
  SCOPE
   SYSTEM STATUS
   TICKER TAPE
   TIME AND SALES
  TRADING NOTES
   USER VARIABLES
  VERTICAL SCALE
 MASTER INDEX
DICTIONARY
FEEDBACK

 Investor/RT Tour
Database Backup and Recovery

(Also See: Tip: Database Utilities)
(Also See: Tip: Keeping Your Database Size Minimal)

This help topic has four sections....

Investor/RT Database Concepts
Investor/RT utilizes a powerful embedded database management system to record and maintain historical data, user preferences, and definitional data. The database contains many different kinds of records, for example there is a record for each instrument. These in turn are related (linked) to the records containing historical data for the instrument. Similarly, relationships are maintained between instruments and the quote pages in which they appear, between instruments and trading positions within portfolios, between instruments and news items and trading notes pertaining to the instrument. Preferences and definitions of charts, quotepages, scans, signals, custom indicators, and all other saved objects are kept in the data base.

The Investor/RT database is a collection of 59 files kept within a folder (directory) named data_f inside the Investor/RT startup directory/folder. Since many the individual files are inter-related, the entire collection of files is backed-up and restored as a whole.

Investor/RT has an automated daily back-up feature and built-in database recovery system that will restore the database back to its most recent back-up if the database is damaged in any way due to a power failure, system error or application failure. The Backup Preferences window allows the user to specify when daily backups are to be made and the location where the backup is kept. The back-up location is by default the backup_f folder (directory) within the data_f folder (directory). The back-up location may be on another hard drive or some other back-up volume if you like. Set the backup preferences for daily backup at some convenient time of day, or if you typically run Investor/RT all day and exit the software regularly each day, setup the preferences to backup when quitting Investor/RT.

For new Investor/RT users, during initial setup of Investor/RT, as instruments, quote pages, charts and portfolios are being defined, it is good practice to periodically issue the Control: Database Utilities: Backup Database command to make a database backup. This assures that all of the setup work up to that point will be protected in the event of a system or application failure. Consider adding a keyboard shortcut that will invoke the Backup Database utility. This can be done easily by creating a "Schedule" (using Setup: Schedule) and assigning a function key, e.g. F2, to the schedule. The schedule will have one "action", i.e. "Backup Database", which is one of the Database Functions you can specify as an action.

When Investor/RT starts up, it determines whether the database was closed normally the last time Investor/RT ended. If so, startup continues normally. If the database was not closed properly, Investor/RT launches the database maintenance utility application to correct minor errors and verify the integrity of the database. If the database is okay, startup continues normally. If the database is damaged, the user is given the opportunity to restore the database to the most recent backup. The database maintenance utility process can take anywhere from a few seconds to several minutes to complete. The time depends on the speed of the computer and hard disk, the number of instruments tracked, and the amount of historical data on file. Options are provided in Backup Preferences for automating the recovery process. It is strongly recommended that the user enable the option to verify the database before making backups and enable the option to automatically restore the database if errors are encountered. If these options are not enabled, Investor/RT cannot automatically protect against data loss or loss of the entire database in the event of system or application failure. The reason these protection features are "optional" is to provide flexibility to those users who have exceptionally large databases who wish to take special action in the event of a system or application failure.

Daily backup is necessary. Additionally, a weekly or monthly backup to some other media is advised just in case hard drive failure renders both the database and backup unrecoverable. We recommend at least a monthly backup to CD ROM. Again, in the early phases of setting up Investor/RT or when instruments, charts, quote pages, and portfolios are being defined, it is recommended that the database be backed up even more frequently using the Backup Database menu item in the Control menu, under Database Utilities.

Database Integrity
If you experience a system or application failure while Investor/RT is running there is a possibility that database integrity may be compromised. Investor/RT guards against this by performing frequent "checkpoints" to assure that all database updates are written to the hard drive periodically and every time you "save" an object of some kind, e.g. a quote page. Nevertheless, when a failure occurs, there is a possibility of database damage. Investor/RT checks the integrity of the database when the software is started again. If the database does have errors, it is imperative that steps be taken to eliminate the errors before proceeding to use the database. If you checkmark the option "Restore database at startup if verify finds errors", the process is entirely automatic. This is the recommended setting. If Investor/RT is setup as recommended to backup daily, to verify before making a backup, and to automatically restore if errors are detected at startup, then there should never be any problem with database integrity.

Further, the MS Windows and Macintosh OS X versions of Investor/RT both incorporate a "crash guard" feature. During execution of Investor/RT, if the software encounters a fatal error and must be terminated for any reason, the crash guard feature performs database housekeeping functions to properly close the database files before Investor/RT is terminated. This feature protects the database from damage due to application failures.

Rebuilding a New Database
If you find it necessary to rebuild your Investor/RT database, here are some tips that will assist you in salvaging as much data as possible from the damaged database.

First, set aside the preference file. On Macintosh system, the file is named preferences.d01 On the MS Windows platform the file name is preferen.d01 This file contain registration information and other user preferences. Make a copy of this file and place the copy on the desktop. Be sure to leave the original file in the data_f folder. Check the dbmaint.txt report to be sure that there were no errors reported in either the object.d01 or the custcol.d01 files. If these files show no error, set copies of them aside as well. These files contain chart, RTL objects (scans, signals, custom indicators), and custom column definitions. In summary here are the files you should copy and set aside on your desktop:

Preferen.d01
Object.d01
Custcols.d01

Start Investor/RT (i.e. running with the damaged database) and use the File: Export command to export the .All Symbols quotepage and each user defined quote page you wish to retain. The exported text files can be imported into the new database later to define the quote pages and instruments.

If database errors were detected in the object.d01 file, you can save all chart and RTL object definitions. Open each chart and RTL object and use File: Export to save their definitions to text files. Repeat this process for each RTL object and chart you wish to retain.

If you wish to save historical data, use the File: Export: Data command. For example to save all daily historical data, choose File: Export: Data, pick the ".All Symbols" quote page, choose Daily, and click the "Export Quote Page" button. A text file containing all daily data on file for every instrument in the ".All Symbols" quote page will be placed into a text file. Tick Data, weekly data, and monthly data may be similarly exported to text files. These files may later be used via the Import command to establish the historical data in the new database.

Having saved quote pages, historical data, charts and RTL objects if necessary, and having set aside the preferences, object, and custcols files as described above, quit Investor/RT and install a "fresh" database". To accomplish this you must first remove the data_f folder/directory from the Investor/RT folder (remove it or rename it in place if you like) then run the Investor/RT "Full Installer"; it will install a new starter database, creating a new data_f folder.

Before starting Investor/RT on the freshly installed database, place the "set aside" preferences, object and custcols files into the data_f folder, replacing the three files having the same name. In addition you MUST run the database maintenance function BEFORE starting Investor/RT. On the Windows platform run the database maintenance from the "Start" Menu in your Windows taskbar. On Macintosh, run the Investor/RT Database Maintenance program located in the Investor/RT Folder. If the maintenance utility continues to report errors, then contact Linn Software support for assistance; attach the dbmaint.txt report file to the email.

If the new database (with your old preferences, object and custcols files) verifies as okay, start Investor/RT. The application will start up with your registration and other preferences. The charts, scans, trading systems, custom columns will be available in the new database.

The next step is to import all of the instruments. Choose File: Import: QuotePage and import the ".All Symbols" quotepage you exported earlier. Repeat this same process for any user-defined quote pages exported earlier.

If your object.d01 file was unusable you can recover RTL objects and chart definitions saved earlier using File: Import to import each of them.
Finally, import any exported historical data. Choose File: Import: Data and pick "Daily Historical Data" as the Import Type and "ASCII Text File" as the Import File Type, and "Import Format" as the File Format. Then click the Import button and pick the file of daily data exported earlier. Repeat this process for tick data, weekly or monthly data, if applicable.

At this point you should be up and running with a new Investor/RT database. Before proceeding further, make sure the "Verify before Backup" checkbox is marked in your Backup Preferences. Then make a backup of the new database by choosing Backup Database from the Control: Database Utilities menu.

At the present time, there is no provision for transferring portfolio definitions and transactions. Portfolios will have to be created again in the normal fashion. As you continue the setup process, make periodic backups to assure all of your changes are backed up.

Database Troubleshooting
Version Consider this scenario: You employ all of the recommended settings, and have a verified daily backup (made yesterday) and at the end of the trading day, Investor/RT attempts to make a new backup and detects database errors. A message appears:

"Database errors were encountered. The requested backup was not performed. Press "Fix" to perform database maintenance, "Restore" to restore the most recent backup."

Investor/RT gives you the choice of either attempting the fix the existing database, or simply restoring the most recent backup. Attempting to fix the existing database is certainly worth a try. The "Fix" button runs the "Database Maintenance" utility which can very often fix minor database issues while preserving all of the setup work done since the last backup was made. If you have done little or no setup work or if reverting to the prior day's backup is acceptable, then lick "Restore". If you elect to fix the database, the database maintenance utility will run and inform you whether the database is okay. In the event that the database maintenance (fix) does not repair the database, you have two choices: either (1) restore the previous backup using the Restore Database command in the Control Menu, or (2) continue to troubleshoot the problem.

If you elect to continue troubleshooting, here are some tips. Quit Investor/RT; run the database maintenance utility independently of Investor/RT. MS Windows users can do this via the Start Menu, i.e.

Start-->Programs-->Investor/RT-->Database Maintenance

Macintosh users should launch the "Database Maintenance" application inside the Investor/RT Folder.

The Database Maintenance utility will then produce a text file named dbmaint.txt inside the admin directory/folder. You can email this file to Linn Software support for diagnosis and recommendations, or open the file yourself and inspect the report. In the report you may find lines of text reporting "Problems at record …", for example:

Problems at record 172:
  * set SL_SYM_T_TICK(27) error:
    invalid last-pointer:
    pointer-dba=[1:3952]
  * set SL_SYM_T_TICK(27) error:
    invalid first-pointer:
    pointer-dba=[1:3951]

If you see SL_SYM_T_TICK or SL_SYM_T_BAR, this indicates that the database error relates to intra-day (tick) data in the database. If you see SL_SYM_T_DAT, or SL_SYM_T_WKT, or SL_SYM_T_MOT, the error relates to daily, weekly, or monthly historical data. If all of the errors relate to either intra-day, or daily/weekly/monthly historical data, Investor/RT has initialization utilities that can repair the database while preserving all other data. The Initialize D/W/M Database utility will eliminate all daily, weekly, and monthly data from the database and repair any damaged linkages relating to the storage of daily, weekly, or monthly data for all ticker symbols. Before using this utility, you may wish to use the Export command to Export Daily (or weekly or monthly) data for all symbols to a text file. After initializing the D/W/M database, you can then use the Import command to bring the exported data back in. Similarly, the Initialize Intra-day Database utility will eliminate all intra-day historical data from the database and repair the linkages. Export one minute bar data for all symbols before initializing if you like; alternatively, you can download intra-day data as needed if your data service supports a historical intra-day server.

After initializing and reimporting historical data, run the database maintenance utility again to assure that the database now verifies okay.

Another type of database error you may see in the dbmaint.txt report involves the linkage SL_SYM_T_NEWS. Problems of this kind indicate an error in the news portion of the database. If you do not wish to restore your backup, you can eliminate the problem by using the "Initialize News Database" utility. All news items will be discarded from the database and all linkages will be repaired. After initializing the news database, run the database maintenance utility again to assure that the database now verifies okay.

Another type of database error you may see in the dbmaint.txt report involves the linkage SL_SYM_T_NOTE. Problems of this kind indicate an error in the trading notes portion of the database. If you do not wish to restore your backup, you can eliminate the problem by using the "Initialize Trading Notes" utility. All trading notes will be discarded from the database and all linkages will be repaired. After initializing the trading notes database, run the database maintenance utility again to assure that the database now verifies okay.

If errors persist, contact Linn Software support for further assistance and guidance.

More Database Details
All software products that provide charting must have some place to store the data. Metastock, for example, a widely used charting program, stores each stock's data in a file on the hard drive. If you track 1000 stocks you have 1000 historical files, each one containing the historical data for one instrument. Investor/RT on the other hand uses an integrated database management system to record all information about the stocks you are following and the historical data for each one. Investor/RT includes an embedded database system that manages all of this information. For example, Daily/Weekly/Monthly historical data for all instruments whether you track 10, 100, 1000, or 10,000 ticker symbols is kept in a single file. The database software embedded in IRT keeps track of which history records belong to which instrument. Another file keeps track of each instrument. There are linkages between this file (the symbol file) and the history file. The record for IBM in the symbol file contains information that tells the database system where in the history file to find the historical data for IBM. This is what I mean by "the files are linked or interrelated".

There are many reasons for using an embedded database system rather than a more simplistic "one file per" approach. When IRT does a scan on 1000 stocks it only has to open the historical file one time instead of opening and closing 1000 individual files. Scans run very very quickly in IRT. Another reason is that a database system is capable of managing very complex data relationships easily. The IRT database keeps track of instruments, quote pages, charts and their contents, layouts, scans, custom indicators, trading signals and trading systems, and much more. Most of these things are interrelated and the database system allows IRT to quickly determine, for example, which tickers belong in which quotepages, which portfolios contain positions on a particular stock, which indicators and instruments belong in each named chart, which instruments make up a custom instrument or even which custom instruments are used in other custom instruments.

We use the database system to great advantage in IRT to record all of your preferences and definitions and all of the complex relationships among them. The flip side is that the care and maintenance of this small collection of inter related files is very important. That's why we employ daily verification and backup methods and have a wide variety of database utility functions aimed at maintaining the integrity of the "database".