NASDAQ LEVEL II
TIME AND SALES
Database Backup and Recovery
(Also See: Tip: Database Utilities)
(Also See: Tip: Keeping Your Database Size
This help topic has four sections....
Investor/RT utilizes a powerful embedded database management system to
record and maintain historical data, user preferences, and definitional
data. The database contains many different kinds of records, for example
there is a record for each instrument. These in turn are related (linked)
to the records containing historical data for the instrument. Similarly,
relationships are maintained between instruments and the quote pages in
which they appear, between instruments and trading positions within
portfolios, between instruments and news items and trading notes
pertaining to the instrument. Preferences and definitions of charts,
quotepages, scans, signals, custom indicators, and all other saved objects
are kept in the data base.
The Investor/RT database is a collection of 59 files kept within a folder
(directory) named data_f inside the Investor/RT startup directory/folder.
Since many the individual files are inter-related, the entire collection
of files is backed-up and restored as a whole.
Investor/RT has an automated daily back-up feature and built-in database
recovery system that will restore the database back to its most recent
back-up if the database is damaged in any way due to a power failure,
system error or application failure. The Backup Preferences window allows
the user to specify when daily backups are to be made and the location
where the backup is kept. The back-up location is by default the backup_f
folder (directory) within the data_f folder (directory). The back-up
location may be on another hard drive or some other back-up volume if you
like. Set the backup preferences for daily backup at some convenient time
of day, or if you typically run Investor/RT all day and exit the software
regularly each day, setup the preferences to backup when quitting
For new Investor/RT users, during initial setup of Investor/RT, as
instruments, quote pages, charts and portfolios are being defined, it is
good practice to periodically issue the Control: Database Utilities:
Backup Database command to make a database backup. This assures that all
of the setup work up to that point will be protected in the event of a
system or application failure. Consider adding a keyboard shortcut that
will invoke the Backup Database utility. This can be done easily by
creating a "Schedule" (using Setup: Schedule) and assigning a function
key, e.g. F2, to the schedule. The schedule will have one "action", i.e.
"Backup Database", which is one of the Database Functions you can specify
as an action.
When Investor/RT starts up, it determines whether the database was closed
normally the last time Investor/RT ended. If so, startup continues
normally. If the database was not closed properly, Investor/RT launches
the database maintenance utility application to correct minor errors and
verify the integrity of the database. If the database is okay, startup
continues normally. If the database is damaged, the user is given the
opportunity to restore the database to the most recent backup. The
database maintenance utility process can take anywhere from a few seconds
to several minutes to complete. The time depends on the speed of the
computer and hard disk, the number of instruments tracked, and the amount
of historical data on file. Options are provided in Backup Preferences for
automating the recovery process. It is strongly recommended that the user
enable the option to verify the database before making backups and enable
the option to automatically restore the database if errors are
encountered. If these options are not enabled, Investor/RT cannot
automatically protect against data loss or loss of the entire database in
the event of system or application failure. The reason these protection
features are "optional" is to provide flexibility to those users who have
exceptionally large databases who wish to take special action in the event
of a system or application failure.
Daily backup is necessary. Additionally, a weekly or monthly backup to
some other media is advised just in case hard drive failure renders both
the database and backup unrecoverable. We recommend at least a monthly
backup to CD ROM. Again, in the early phases of setting up Investor/RT or
when instruments, charts, quote pages, and portfolios are being defined,
it is recommended that the database be backed up even more frequently
using the Backup Database menu item in the Control menu, under Database
If you experience a system or application failure while
Investor/RT is running there is a possibility that database integrity may
be compromised. Investor/RT guards against this by performing frequent
"checkpoints" to assure that all database updates are written to the hard
drive periodically and every time you "save" an object of some kind, e.g.
a quote page. Nevertheless, when a failure occurs, there is a possibility
of database damage. Investor/RT checks the integrity of the database when
the software is started again. If the database does have errors, it is
imperative that steps be taken to eliminate the errors before proceeding
to use the database. If you checkmark the option "Restore database at
startup if verify finds errors", the process is entirely automatic. This
is the recommended setting. If Investor/RT is setup as recommended to
backup daily, to verify before making a backup, and to automatically
restore if errors are detected at startup, then there should never be any
problem with database integrity.
Further, the MS Windows and Macintosh OS X versions of Investor/RT both
incorporate a "crash guard" feature. During execution of Investor/RT, if
the software encounters a fatal error and must be terminated for any
reason, the crash guard feature performs database housekeeping functions
to properly close the database files before Investor/RT is terminated.
This feature protects the database from damage due to application
Rebuilding a New Database
If you find it necessary to rebuild your Investor/RT database,
here are some tips that will assist you in salvaging as much data as
possible from the damaged database.
First, set aside the preference file. On Macintosh system, the file is
named preferences.d01 On the MS Windows platform the file name is
preferen.d01 This file contain registration information and other user
preferences. Make a copy of this file and place the copy on the desktop.
Be sure to leave the original file in the data_f folder. Check the
dbmaint.txt report to be sure that there were no errors reported in either
the object.d01 or the custcol.d01 files. If these files show no error, set
copies of them aside as well. These files contain chart, RTL objects
(scans, signals, custom indicators), and custom column definitions. In
summary here are the files you should copy and set aside on your desktop:
Start Investor/RT (i.e. running with the damaged database) and use the
File: Export command to export the .All Symbols quotepage and each user
defined quote page you wish to retain. The exported text files can be
imported into the new database later to define the quote pages and
If database errors were detected in the object.d01 file, you can save all
chart and RTL object definitions. Open each chart and RTL object and use
File: Export to save their definitions to text files. Repeat this process
for each RTL object and chart you wish to retain.
If you wish to save historical data, use the File: Export: Data command.
For example to save all daily historical data, choose File: Export: Data,
pick the ".All Symbols" quote page, choose Daily, and click the "Export
Quote Page" button. A text file containing all daily data on file for
every instrument in the ".All Symbols" quote page will be placed into a
text file. Tick Data, weekly data, and monthly data may be similarly
exported to text files. These files may later be used via the Import
command to establish the historical data in the new database.
Having saved quote pages, historical data, charts and RTL objects if
necessary, and having set aside the preferences, object, and custcols
files as described above, quit Investor/RT and install a "fresh"
database". To accomplish this you must first remove the data_f
folder/directory from the Investor/RT folder (remove it or rename it in
place if you like) then run the Investor/RT "Full Installer"; it will
install a new starter database, creating a new data_f folder.
Before starting Investor/RT on the freshly installed database, place the
"set aside" preferences, object and custcols files into the data_f folder,
replacing the three files having the same name. In addition you MUST run
the database maintenance function BEFORE starting Investor/RT. On the
Windows platform run the database maintenance from the "Start" Menu in
your Windows taskbar. On Macintosh, run the Investor/RT Database
Maintenance program located in the Investor/RT Folder. If the maintenance
utility continues to report errors, then contact Linn Software support for
assistance; attach the dbmaint.txt report file to the email.
If the new database (with your old preferences, object and custcols files)
verifies as okay, start Investor/RT. The application will start up with
your registration and other preferences. The charts, scans, trading
systems, custom columns will be available in the new database.
The next step is to import all of the instruments. Choose File: Import:
QuotePage and import the ".All Symbols" quotepage you exported earlier.
Repeat this same process for any user-defined quote pages exported
If your object.d01 file was unusable you can recover RTL objects and chart
definitions saved earlier using File: Import to import each of them.
Finally, import any exported historical data. Choose File: Import: Data
and pick "Daily Historical Data" as the Import Type and "ASCII Text File"
as the Import File Type, and "Import Format" as the File Format. Then
click the Import button and pick the file of daily data exported earlier.
Repeat this process for tick data, weekly or monthly data, if applicable.
At this point you should be up and running with a new Investor/RT
database. Before proceeding further, make sure the "Verify before Backup"
checkbox is marked in your Backup Preferences. Then make a backup of the
new database by choosing Backup Database from the Control: Database
At the present time, there is no provision for transferring portfolio
definitions and transactions. Portfolios will have to be created again in
the normal fashion. As you continue the setup process, make periodic
backups to assure all of your changes are backed up.
Version Consider this scenario: You employ all of the recommended
settings, and have a verified daily backup (made yesterday) and at the end
of the trading day, Investor/RT attempts to make a new backup and detects
database errors. A message appears:
"Database errors were encountered. The requested backup was not performed.
Press "Fix" to perform database maintenance, "Restore" to restore the most
Investor/RT gives you the choice of either attempting the fix the existing
database, or simply restoring the most recent backup. Attempting to fix
the existing database is certainly worth a try. The "Fix" button runs the
"Database Maintenance" utility which can very often fix minor database
issues while preserving all of the setup work done since the last backup
was made. If you have done little or no setup work or if reverting to the
prior day's backup is acceptable, then lick "Restore". If you elect to fix
the database, the database maintenance utility will run and inform you
whether the database is okay. In the event that the database maintenance
(fix) does not repair the database, you have two choices: either (1)
restore the previous backup using the Restore Database command in the
Control Menu, or (2) continue to troubleshoot the problem.
If you elect to continue troubleshooting, here are some tips. Quit
Investor/RT; run the database maintenance utility independently of
Investor/RT. MS Windows users can do this via the Start Menu, i.e.
Macintosh users should launch the "Database Maintenance" application
inside the Investor/RT Folder.
The Database Maintenance utility will then produce a text file named
dbmaint.txt inside the admin directory/folder. You can email this file to
Linn Software support for diagnosis and recommendations, or open the file
yourself and inspect the report. In the report you may find lines of text
reporting "Problems at record …", for example:
Problems at record 172:
* set SL_SYM_T_TICK(27) error:
* set SL_SYM_T_TICK(27) error:
If you see SL_SYM_T_TICK or SL_SYM_T_BAR, this indicates that the database
error relates to intra-day (tick) data in the database. If you see
SL_SYM_T_DAT, or SL_SYM_T_WKT, or SL_SYM_T_MOT, the error relates to
daily, weekly, or monthly historical data. If all of the errors relate to
either intra-day, or daily/weekly/monthly historical data, Investor/RT has
initialization utilities that can repair the database while preserving all
other data. The Initialize D/W/M Database utility will eliminate all
daily, weekly, and monthly data from the database and repair any damaged
linkages relating to the storage of daily, weekly, or monthly data for all
ticker symbols. Before using this utility, you may wish to use the Export
command to Export Daily (or weekly or monthly) data for all symbols to a
text file. After initializing the D/W/M database, you can then use the
Import command to bring the exported data back in. Similarly, the
Initialize Intra-day Database utility will eliminate all intra-day
historical data from the database and repair the linkages. Export one
minute bar data for all symbols before initializing if you like;
alternatively, you can download intra-day data as needed if your data
service supports a historical intra-day server.
After initializing and reimporting historical data, run the database
maintenance utility again to assure that the database now verifies okay.
Another type of database error you may see in the dbmaint.txt report
involves the linkage SL_SYM_T_NEWS. Problems of this kind indicate an
error in the news portion of the database. If you do not wish to restore
your backup, you can eliminate the problem by using the "Initialize News
Database" utility. All news items will be discarded from the database and
all linkages will be repaired. After initializing the news database, run
the database maintenance utility again to assure that the database now
Another type of database error you may see in the dbmaint.txt report
involves the linkage SL_SYM_T_NOTE. Problems of this kind indicate an
error in the trading notes portion of the database. If you do not wish to
restore your backup, you can eliminate the problem by using the
"Initialize Trading Notes" utility. All trading notes will be discarded
from the database and all linkages will be repaired. After initializing
the trading notes database, run the database maintenance utility again to
assure that the database now verifies okay.
If errors persist, contact Linn Software support for further assistance
All software products that provide charting must have some place to store the data.
Metastock, for example, a widely used charting program, stores each stock's data in a file on the hard drive. If you track 1000 stocks you have 1000 historical files, each one containing the historical data for one instrument. Investor/RT on the other hand uses an integrated database management system to record all information about the stocks you are following and the historical data for each one. Investor/RT includes an embedded database system that manages all of this information. For example, Daily/Weekly/Monthly historical data for all instruments whether you track 10, 100, 1000, or 10,000 ticker symbols is kept in a single file. The database software embedded in IRT keeps track of which history records belong to which instrument. Another file keeps track of each instrument. There are linkages between this file (the symbol file) and the history file. The record for IBM in the symbol file contains information that tells the database system where in the history file to find the historical data for IBM. This is what I mean by "the files are linked or interrelated".
There are many reasons for using an embedded database system rather than a more simplistic "one file per" approach. When IRT does a scan on 1000 stocks it only has to open the historical file one time instead of opening and closing 1000 individual files. Scans run very very quickly in IRT. Another reason is that a database system is capable of managing very complex data relationships easily. The IRT database keeps track of instruments, quote pages, charts and their contents, layouts, scans, custom indicators, trading signals and trading systems, and much more. Most of these things are interrelated and the database system allows IRT to quickly determine, for example, which tickers belong in which quotepages, which portfolios contain positions on a particular stock, which indicators and instruments belong in each named chart, which instruments make up a custom instrument or even which custom instruments are used in other custom instruments.
We use the database system to great advantage in IRT to record all of your preferences and definitions and all of the complex relationships among them. The flip side is that the care and maintenance of this small collection of inter related files is very important. That's why we employ daily verification and backup methods and have a wide variety of database utility functions aimed at maintaining the integrity of the "database".