Track & Trace Gateway Tracer Maintenance and Troubleshooting
Although the Container tracing is automatic and scheduled throughout the day, Track and Trace does require some ongoing maintenance, administration, and response to periodic alerts. It is suggested that a primary, and a backup user be assigned the task of maintaining the Tracer.
Login Credentials
The most common maintenance needed Terminal tracing is updating the Login Credentials. Most of the Terminal sites require periodic password changes. This is also the most common reason for tracing failures and site deactivation's, and can easily be updated in Track and Trace when the site credentials are changed.
On the Rail side of tracing credentials should not need to be changes as the credentials utilized are setup by an Envase Profit Tools Technician during implementation. If new Rail setup is required it is best to submit an assistance request to CustomerService@ProfitTools.net
Site Re-activation
Track and Trace will automatically deactivate sites when three successive traces have failed. This can be caused by invalid login credentials, failed internet connectivity, or local connectivity issues. The system can be set to automatically reactivate sites, but if the same site is deactivated multiple times, then user intervention is required. In many cases, a hung web session can be resolved by exiting and restarting the Tracer
Responding to alerts
Track and Trace alerts can be generated for a variety of reasons. These alerts can be Informational, Warnings or Errors, combined with a Low, Medium or High priority. This combination will provide an indication of how urgent the alert, and the type of response that may be needed. The message may also contain suggested steps to correct the issue, or may indicate that no user action is required.
Troubleshooting
The most common issue reported from a user’s perspective is that “Track and Trace is not tracing.” This can mean that one container has not traced as expected in the last half hour or that the application was closed several days ago and is non-operational.
Triage
The following general guidelines will help assess the scope of the issue the user is experiencing.
Note: It is important to verify any reported issues after closing and restarting the tracer. Always verify any results reported by the user, problems could have been transient, misinterpreted, or based on incorrect assumptions.
Messages
Evaluate the Tracer for Errors in the Message Logs, Email alert messages, an unresponsive Tracer instance or evidence the Tracer has not been running. These will provide the most direct clues to the nature of the failure.
Container Status
Review the Red/Green container status indicators to verify tracing. This can be used as an indication of the health of the tracer as well as the status of the data. Sorting by Reporting Rail or Terminal will allow a scan of the status indicators to see if any trends or patterns are visible such as all containers from a site are red.
In addition, there is also a Trace Result column that advises of the status of the last trace of the container.
Trace Times
Review both the last Auto trace Date/Time as well as the Last Success / Next Trace Date Times for individual containers. These will give a direct indication of how recent the data has been refreshed.
Scope
The scope of the issue will give a good indication of how to troubleshoot the issue
One container
The failure of a single container to trace is generally indicative of a data entry error, tracing the container on an incorrect site, no trace data available from the host site, or the data was updated since the last Auto Trace, but may rarely be indicative of a bug in the tracing logic. The testing of this level issue is generally to verify the container number and the tracing site, and to perform a manual trace of that container. If the manual trace also provides no results, then the user should verify their data and call the terminal for additional information. If the manual trace results differ from the Track and Trace results, a Force Trace should be done to refresh the Track and Trace results. If the Manual and Track and Trace results still differ then Profit Tools should be contacted to verify the issue and test for a tracing bug.
All containers for one site
A failure of an entire site to trace is most often indicative of an incorrect or expired password for the site, but may be due to the site being temporarily offline, site updates that inhibit the tracing, local internet security policy, or a bug in the tracing logic. The verification of this level issue is generally to verify the site login credentials and to manually log into the site and trace several sample containers. If the password is expired, or otherwise incorrect it should be corrected and updated in Track and Trace Site Setup. If the Manual and Track and Trace results differ then Profit Tools should be contacted to verify the issue and test for site changes or a tracing bug.
Multiple sites
A failure of multiple sites to trace will generally be caused by the tracer not running, internet connectivity issues, Windows modal popups, a site crashing during tracing. The best course of testing for this issue is to queue several containers for each trace site and observer the full trace cycle. If any popup messages appear outside of the browser session, then this is the most likely cause. The Click Off utility should be installed and configured to manage this popup. If all of the trace events appear to time out, then the LAN or internet connectivity should be investigated. If the trace results in a crash, Profit Tools should be contacted. If the partial tracing succeeds, then the test should be repeated with all containers in the queue.
Common Resolutions
Password Maintenance
Many of the sites that require passwords, will also require periodic password updates. If the person updating the site password is not the primary T&T user, it is common for the credentials stored in T&T not to get updated. For any site that is disabling frequently or unable to login, the credentials should be validated manually.
Solution – Update the credentials
Site Outages
Manually verify that the site is accessible, and a container can be traced manually. This will assist in validating both the login credentials and the operation of the site. For customers that are 100% reliant on automatic tracing, they may not be aware of a site outage.
Solution – Wait for the site to come online
Site Changes
Site changes will generally require code changes within the tracing application. We do monitor internally to attempt to be as responsive as possible to these changes, but due to the nature of their business, we are often alerted by an attentive customer to a site change before our monitoring system indicates sufficient errors to trigger an alert.
Solution – Notify Profit Tools of the site changes. Provide login credentials and sample containers.
Site Specific Setup
Several sites including BNSF, CPR, and Emodal may have settings changes that need to be made in the web site options in order to trace.
Solution – Investigate any site specific requirements, like access to a particular tracing page, allowed user permissions, or the number of containers that may be traced in one request.
Restart the Tracer
Due to the nature of the system and the multiple connections made to sites over the internet, it is possible for the site to unexpectedly terminate a connection in a way the tracing system cannot auto recover. In most situations it is advisable to close the Tracer, check in the Windows Task Manager to verify the railtrace.exe application is closed, and restart the Tracer. This will reset all internet connections and will often be enough to resolve an unidentified tracing problem.
Solution – Exit and restart the tracer
Python Service
Under certain circumstances the Python tracing Service could stop responding, or be in an invalid state due to PC or Tracing related issues. The first thing to check is from Help / Program Info. From here you can verify the version of the Python Service. Checking the version will verify that the service is running and responding. The service can be started or restarted using Task Manager. Look for Track And Trace Gateway under the Services Tab.
Hung Trace
Due to web, server, or tracing errors it is possible for a trace session to become hung up. This may cause the site not to trace or the tracer to appear frozen, even if the site is up and the credentials are correct.
Solution – Exit and restart the Tracer and Tracing Service, then verify all sites are active and the logins are correct.
Upgrade
When experiencing tracing issues, always run the Check for upgrade as a site update may have been posted. It is possible an upgrade has been released, but the system has not yet checked for an automatic update. It is also possible that the upgrade was attempted, but due to local, network, or internet security settings the upgrade was not able to complete. It is also possible for AV and UAC to prevent the automatic updates.
Solution – Complete the upgrade manually and address any IT issues necessary to allow the automatic updates
Tracer is not running
The system could be failing to trace individual sites as noted above, or could simply be not tracing at all. Some common things to check are noted below.
T&T Not Running – Make sure T&T tracer is up and running on the designated PC or terminal server session.
Solution – Start the T&T Tracer.
Not Running as Tracer – T&T can be started in either an active Tracing mode or a passive Client mode. If you cannot see the Trace Now buttons then T&T has been started in Client mode.
Solution – Make sure that T&T has been started using the ‘-tracer’ command line argument and was not just started by double clicking on the EXE.
Autotrace – If the Auto Trace schedule is not correctly configured or the auto tracing has not been turned on, the system will not trace automatically.
Solution – Verify the Auto Trace schedule is set up properly in Settings and that the Auto Trace for Port and Rails (as needed) are set to ON.
SQL Errors
Error -3: Row changed between retrieve and update
This error can reverence a variety of tables, but almost always indicates that two or more instances of the T&T Tracer are running at the same time and are causing a database conflict.
Solution – Close all open copies of T&T, reopen a single tracer instance.
System Errors
Fatal Disk Error
Fatal Disk Errors are System based errors that will adversely affect the tracing system and cause the application to fail with a variety of other cascade errors. These are typically caused by the disconnecting of the drive mapping, or by excessively slow server response that times out a disk read.
Solution – Fix the underlying cause of the error and then exit and restart the Tracer
Track & Trace Error Codes
General Severity
- Errors: 500 - 599
- Warnings: 600-699
- Information: 700 - 799
Defined Error Codes
Errors
- 500 - Escalated warning
- 502 - Site not suppported
- 505 - Failed Login
- 513 - Account locked
- 514 - Invalid username/password
- 515 - Unknown Error. Not logged in.
- 516 - Password Expired
Warnings
- 600 - Wait_for_element timeout
- 601 - Failed to click element
- 602 - Failed to send keys to element
- 603 - Failed to inject jquery
- 604 - Failed to trace extra details (Implemented on BNSF)
- 606 - Failed to scrape
- 607 - Failed to normalize
- 608 - Failed to remove containers (Not sure where this is used)
- 610 - Quit(for when TT ends SCSPA)
- 611 - Requested page not found
- 612 - Element not found
Informational
- 700 - EMODAL Failed Login