TwoToneDetect Redundancy

Starting with v66, multiple instances of TwoToneDetect running in different locations can work together to provide a redundant backup alerting system.

For example, two copies of TwoToneDetect can be running at fire stations opposite sides of a county or district where they are on different power grids and have different internet service providers.  If the power or internet goes out at the primary location, the secondary location will detect that the primary location is down and will send the emails instead.

Or, two copies of TwoToneDetect can be running on the same computer with different email servers specified in their configurations.  If the primary email server is down and fails to send, the secondary copy of TwoToneDetect will send the emails using the secondary server instead.

Note:  If you don’t have two locations or instances of TwoToneDetect running but still want to be able to check the status of the program and automatically restart it if it shuts down or freezes, see here.

How it Works

Both instances of TwoToneDetect must have access to a common file that they can both read and write to.  This file can have any name, but in these examples we’ll call it CommonToneLogFile.txt. When TwoToneDetect is configured for redundant operation and detects a valid tone set, it will attempt to read CommonToneLogFile.txt to see if any other instance of TwoToneDetect has already detected the same tone and sent emails.  If it finds that no other instance of the program has already sent emails for the detected done set, it will send the emails and then update CommonToneLogFile.txt so that other instances of the program will not send email.

In order to ensure that multiple instances of the program are not trying to read or write to CommonToneLogFile.txt at the same time, each instance of the program is configured with a delay.  The delay for the primary instance of TwoToneDetect should be set to zero, while each additional instance should be configured with a delay of around 15-60 seconds to allow the primary instance to write to the file before the secondary reads the file.

Currently TwoToneDetect supports the ftp protocol for remote file access.  If both instances of TwoToneDetect are running on the same computer (for email server redundancy), a locally stored file can be used.

If the remote file is accessed via ftp, a second “common” text file is used by each instance of the program to check to see whether the other instances of the program are online.  In this way, if the email or power goes out at one location running TwoToneDetect, the other location will detect this and send an email to an administrator so that they are aware of the outage.  In these examples we’ll call this file CommonStatusFile.txt

To configure TwoToneDetect to look for this shared file, create a text file called redundant.cfg in the program directory.  This file will have the following format:

[REDUNDANT]
file_access = ftp
tone_tracking_file = CommonToneLogFile.txt
redundant_delay = 30
redundant_instance_alias = Backup at Station 2
ftp_server = ftp.twotonedetect.net
ftp_port = 21
ftp_username = ftp_username
ftp_password = ftp_password
TTD_tracking_file = CommonStatusFile.txt
redundant_down_alert_time = 400
redundant_admin_email = email@address.com

Explanation of file parameters:

  • file_access: options are ftp or local.
  • tone_tracking_file:  Name of the file used to track detected tone sets across multiple running instances of the program.  For local files, specify the entire path to the file i.e. c:\path\to\filename.txt.  For a common file stored on an ftp server, specify only the filename i.e. filename.txt.
  • redundant_delay:  Number of seconds to wait before checking the common file.  For the primary instance of TwoToneDetect, set this to 0.  For the secondary/backup instance of TwoToneDetect, set this to a reasonable delay.  30 seconds is suggested as a starting point.
  • redundant_instance_alias: This field allows you to specify an alias for this instance of TwoToneDetect for easier identification when emails are received.  For example, you could set it to something like “Primary”, “Backup”, “Station 1 Radio Room”, “Tower Site”, etc.
  • ftp_server:  If accessing a remote common file via ftp, specify the server location here.  If accessing a local common file, this line can be omitted.
  • ftp_port:  If accessing a remote common file via ftp, specify the ftp port here.  Default is port 21.  If accessing a local common file, this line can be omitted.
  • ftp_username:  If accessing a remote common file via ftp, specify the ftp username here.  If accessing a local common file, this line can be omitted.
  • ftp_password:  If accessing a remote common file via ftp, specify the ftp password here.  If accessing a local common file, this line can be omitted.
  • TTD_tracking_file: Name of the file used to track whether other instances of the program are online.  This parameter is only used if file_access = ftp.
  • redundant_down_alert_time: Amount of time to wait before sending an email to notify an administrator that one of the instances of TwoToneDetect is offline.  This parameter is only used if file_access = ftp.
  • redundant_admin_email: Email address to send an alert message to if an instance of TwoToneDetect is detected as being offline.  This parameter is only used if file_access = ftp.
Tips
  1. When setting a system up for redundant operation, it is probably a good idea to take advantage of TwoToneDetect’s remote tones.cfg capabilities to ensure that both copies of the program are accessing the same remotely-hosted tones.cfg file.  This will ensure that both the primary and backup instances of the program are using the same list of tones and users at all times.
  2. Make sure both the primary and redundant system are having their clocks automatically updated via NTP.  Most modern operating systems do this automatically, but double check to make sure it’s enabled on your system.  It’s important that the clocks of the two systems are synchronized regularly.
  3. If you reboot the systems regularly (this is recommended), have the systems reboot at different times of day so that one is always online.  Having the reboot times staggered also helps reduce the changes that both instances are always accessing the remote files at the same time.
  4. Differences in performance on the two systems can lead to different response times.  It may take one system longer to convert the WAV files to MP3/AMR format than the other, and it may take one system longer to send emails than the other.  Keep this in mind when choosing the redundant_delay time interval.  The longer this delay time is, the less likely it is that you will get duplicate emails for the same tone set.  On the other hand longer delay times will result in the message being delayed longer when the primary system is down.
A Word of Caution

While cloud drive services like Dropbox and Google Drive may be an option for hosting the common file (using the “local” option in the file_access parameter of the redundant.cfg file), the synchronization of files between remote computers is left to the whims of the cloud drive’s application.  Because TwoToneDetect redundant operation is time-sensitive, cloud-drive solutions may not perform well, and could result in multiple emails being sent for the same tone set.  Your mileage may vary.  If you need a place to host your common file, contact me for a low-cost hosting option.

Leave a Reply