Service replay debugging

The service replay debugging feature allows you to replay actual service events that occurred in your service instance when your application ran in Symphony. You use service replay debugging when you want to debug without making changes to your application.

About service replay debugging

In the Symphony environment, the SIM starts a service instance and drives all of the service events that occur. The service replay debugging feature can capture those service events in a file, along with the original data that was passed to the service at each stage. The service replay debugging feature allows you to run your service from this file, no longer requiring a SIM to drive the service actions. Service replay debugging can drive the service instance in complete isolation, independent of Symphony or Symphony DE.

With service replay debugging, you can run your service binary directly from a command line, debugger, or integrated development environment (IDE) and step through the service code to find any errors in your service logic or in the service environment.

The service replay debugging feature makes it easy to debug a number of different scenarios:


To handle ...

The service replay debugging feature allows you to ...

Environment issues

Set the runtime environment for the service and run your service binary. In many cases, you will be able to see which dynamic libraries failed to load, which environment variables are not set as expected, and which other dependencies are missing.

Startup issues

Capture service startup and initialization issues without modifying your service code.

Miscellaneous issues

Capture every service event. For example, you could use the debugger to step through your service code to make sure everything works as expected.

Unexpected issues

Capture unexpected problems so that you can replay and debug them later on. For example, you could use service replay debugging to trace and resolve any new issues found in your service.


Without service replay debugging

The following is an example demonstrating debugging an error that occurs in the onCreateService( ) method when service replay debugging is not enabled.

Without service replay debugging enabled, the session manager and service instance manager logs indicate that a process exited while creating the service. You do not know why.

To debug, you use live service debugging (debugging while the service is running), and you modify the onCreateService code to:

  • Add an infinite while loop at the top of onCreateService

  • Attach to the live service instance process with a debugger, and set breakpoints

  • Modify the values of variables to exit the while loop

  • Debug the live process

  • Once resolved, remove the infinite while loop

With service replay debugging

The following is an example of debugging an error that occurs in the onCreateService( ) method when service replay debugging is enabled.

To debug, you:

  • Run the environment script in a command prompt or shell

  • Run the service binary from the same command prompt or shell, as the environment script. Either run it directly or from a debugger IDE. The environment script tells the service to run from the SERL file.

Service event replay log (SERL)

This file contains the data required to replay actual service events that occurred on your service instance when your application ran in Symphony.

Note:

More than one set ( a SERL file and an environment script) may exist, depending on the number of service instances that encountered errors. You can use any set to replay the service events that occurred.

Environment script

Environment scripts are used to simulate a service environment when you replay your service. When you run the environment script, the script sets the environment variables that were used when your service ran through Symphony.

An environment script is found together with its SERL file. The environment script is named as follows:
  • Windows:

    appName.serviceName.hostname.pid.timestamp.env.bat

  • UNIX bash:

    appName.serviceName.hostname.pid.timestamp.env.profile

  • UNIX csh:

    appName.serviceName.hostname.pid.timestamp.env.cshrc

Symphony adds a single reserved environment variable in the environment script called SOAM_SERVICE_EVENT_REPLAY_LOG, which specifies the SERL file that can be used to run your service. When the SOAM_SERVICE_EVENT_REPLAY_LOG variable is defined by the environment script, your service is driven by the service events logged in the SERL file that this variable references. If you do not run the environment script, this environment variable is not defined, and your service is driven by the service instance manager.

Location of SERLs and environment scripts

The SERLs and environment scripts are located as follows:

  • On Windows: %SOAM_HOME%\work\serl\appName\serviceName

  • On UNIX: $SOAM_HOME/work/serl/appName/serviceName

Scope

Service replay debugging can be used in both Symphony DE and Symphony on the grid,

Service replay debugging can be used on all platforms supported by Symphony.

Configuration to enable service replay debugging

Service replay debugging is enabled by setting the debugSetting attribute in the Service section of the application profile.


When you set debugSetting to ...

The behavior is ...

Use this ...

none

Disables service replay debugging. No environment script or service event replay log is created.

This is the default setting.

When you do not want to enable service replay debugging.

customized

Enables service replay debugging upon detection of specific, customizable error-handling events. Upon detection of a customized event, the environment in which the service is running is captured in one or more environment scripts, depending on the platform, and a service event replay log is created. You can configure the specific error-handling events that you would like Symphony to detect by specifying the customizedDebugAction for each of these events.

When you are developing and testing your service, to allow Symphony to detect common service problems, or when you are moving a clean service into production, and want to detect any unexpected problems that did not occur during testing.

full

Enables service replay debugging. For every service instance, the environment in which the service is running is captured in one or more environment scripts, depending on the platform, and a service event replay log is created capturing all service events.

When you cannot capture your problems using the customized debug setting, or when problems are cumulative.


Events captured when debugSetting is customized

For each method, the service events that are captured are listed below:


Method

Captured Service Events

Notes

Register

  • Register

“Register” means “All code that executes before calling the ServiceContainer’s run method”.

CreateService

  • Register

  • CreateService

SessionEnter

  • Register

  • CreateService

  • SessionEnter

SessionUpdate

  • Register

  • CreateService

  • SessionEnter

  • SessionUpdate

It is better to record all SessionUpdate events that had occurred between SessionEnter and SessionLeave. If invokes are interleaved between SessionUpdates, those Invoke events that occurred are discarded.

Invoke

  • Register

  • CreateService

  • SessionEnter

  • SessionUpdate

  • Invoke

If there were tasks that preceded the current task, they are not replayed. This assumes that the execution of those tasks did not contribute to the failure of the current task (i.e. the tasks do not alter the state of the service). Therefore, the omission of the other service events should not change the behavior of the Invoke when replayed.

All SessionUpdate events that had occurred for this session are retained.

SessionLeave

  • Register

  • CreateService

  • SessionEnter

  • SessionUpdate

  • SessionLeave

If there were tasks that preceded the SessionLeave, they are not replayed. This assumes that the execution of tasks do not contribute to the failure of the SessionLeave method (i.e. the tasks do not alter the state of the service). This also assumes that the SessionEnter is used correctly (user’s intention for that function is correct, but the functionality may not be correct) to perform any session-specific initialization and that SessionLeave is used correctly to do the corresponding session-specific uninitialization. Therefore, the omission of the other service events should not change the behavior of the SessionLeave when replayed.

All SessionUpdate events that had occurred for this session are retained.

DestroyService

  • Register

  • CreateService

  • DestroyService

If there were sessions bound to this SI, tasks executed on this SI, and sessions unbound from this SI before the DestroyService, these service events are not replayed. This assumes that the CreateService is used correctly to perform any initialization common to the whole service and that the DestroyService is used correctly to do the corresponding service-specific uninitialization. Therefore, the omission of the other service events should not change the behavior of the DestroyService when replayed.

ServiceInterrupt

An interrupt event follows a “normal” service event (those listed above this one in this table). InterruptEvents are preserved only if the service event that precedes it is preserved for replay.

Under service replay debugging, you have to control the interrupting thread and the thread that is performing the “normal” service event yourself to re-create whatever the scenarios you are experiencing.

Symphony preserves service events but cannot preserve the timing.


Related concepts
About debugging a service

Configuration to modify service replay debugging behavior

Service replay debugging behavior can be modified as follows:
  • Customize the error-handling events you want to detect and capture when debugSetting is set to customized.

  • Set the environment script execution mode to control the environment the service will replay in

Customize error-handling events for detection and capture

In the application profile, you can configure service replay debugging upon detection of the following error-handling events in a specific method:
  • Timeout—A method executes for longer than its configured duration

  • Exit—The service process exits while executing a method

  • Exception—The service throws an exception while executing a method

  • Return—The service returns from a method normally, possibly with a return code

You configure service replay debugging for an error-handling event by setting the customizedDebugAction attribute for the event.


Table 1. Effect of Setting customizedDebugAction

Event in Application Profile

customizedDebugAction Value

Behavior

Service > Control > Method > Timeout

writeServiceEventReplayFiles

Default. When Symphony detects that the specified method has timed out, it generates service event replay files to capture the relevant service events that led up to the timeout. This is the recommended setting if method timeout is an unexpected problem for your service.

none

When Symphony detects that the specified method has timed out, it does not generate service event replay files. This is the recommended setting if your service hangs as a normal occurrence.

Service > Control > Method > Exit

writeServiceEventReplayFiles

Default. When Symphony detects that the service process has exited (or crashed) while executing the specified method, it generates service event replay files to capture the relevant service events that led up to the exit. This is the recommended setting if the service process exiting or crashing in the specified method is an unexpected problem for your service.

none

When Symphony detects that the service process has exited (or crashed) during execution of the specified method, it does not generate service event replay files. This is the recommended setting if your service exits as a normal occurrence.

Service > Control > Method > Exception

writeServiceEventReplayFiles

Default. When Symphony detects that the specified method has thrown a particular exception (Fatal or Failure), as specified, it generates service event replay files to capture the relevant service events that lead up to the exception.

none

When Symphony detects that the specified method has thrown a particular exception (Fatal or Failure), as specified, it does not generate service event replay files.

Service > Control > Method > Return

writeServiceEventReplayFiles

When Symphony detects that the specified method has returned normally with or without a specific control code, it generates service event replay files.

none

Default. When Symphony detects that the specified method has returned normally with or without a specific control code, it does not generate service event replay files. This is the recommended setting, as this is typically a normal, successful occurrence.


Environment script execution mode

The environment under which the service replays is captured in the environment scripts. Depending on the command-line arguments used when running the script, the environment is different.


If the run the script with ...

The behavior is ...

Use this mode when ...

No command-line arguments

The script appends the value of the current shell’s environment to the service’s environment for the following environment variables:

  • PATH

  • LD_LIBRARY_PATH (UNIX only)

  • LD_PRELOAD (UNIX only)

Any other environment variables set in the service overwrite the values set in the shell.

You want to retain your development environment while debugging your service.

-pure-env (Windows)

--pure-env (UNIX)

All environment variables set in the service overwrite the values set in the shell.

You cannot reproduce the problem using the default service environment, typically when you are trying to replicate the exact environment for the service to debug a possible environment problem.