dynamicTAO

(Sorry, this is out of date, we have more new stuff that's not documented yet. If you need it, get in touch with us).

1. Introduction

dynamicTAO is an extension of the TAO ORB that allows users to dynamically change the ORB behavior. This modification of the behavior is achieved through a runtime reflective interface added to the standard TAO.

The standard TAO distribution (TAO, from here) has been designed and implemented using object oriented patterns. One of the patterns used, the strategy pattern, encapsulates the behavior of several key features of the ORB (concurrency, demultiplexing, etc.) in such a way that its behavior can be modified, but only when the ORB is first launched. Desired strategies are specified by editing a configuration file (svc.conf) where several parameters can be adjusted. When TAO starts its execution, it reads the file and configures itself accordingly to the parameters it finds.

TAO is built on top of ACE, and it uses the ACE Service Configurator to configure its services (TAO's strategies are instantiated by abstract factories [2] that are implemented as ACE Services).

dynamicTAO offers the possibility of reconfiguring the strategies used by TAO on the fly, while it is running, and not only when it is first launched. In order to achieve that goal, several classes have been created to allow dynamic reconfiguration.

Section 2 lists the classes defined for dynamicTAO, section 3 explains how the current version of dynamicTAO works, and finally, section 4 describes new functionality that is being added to dynamicTAO. This last section basically describes how a new IDL specified interface is offered to the servants running on top of the ORB so they are able to modify the behavior of their own ORB.

2. dynamicTAO classes and inheritance diagram

dynamicTAO is composed of the elements listed below:

    Application_Configurator                        Persistent_Repository                 TAO_Dynamic_Config

    Net_Broker                                            Dynamic_Service_Config            Dynamic_Server_Factory

    DCP_Broker                                          Application_Configurator

The inheritance diagram is the one that follows:

   --------------------------          ----------------------
   |Application_Configurator|<>------- |Application_Record  |
   --------------------------          ----------------------

             ------------------------
             |  ACE_Service_Object  |
             ------------------------
                         |
                         |                     -----------------------
                        / \             -------|Persistent_Repository|
                  ------------          |      -----------------------
                  |Net_Broker |<>-------|
                  ------------          |
                        |               |      -------------------------         -----------------
                        |               |------|Dynamic_Service_Config  |<>------|Application_List|
                        |                      -------------------------         -----------------
                       / \
                  ------------
                  |DCP_Broker|
                  ------------

                                  ------------------------
                                  |Simple_Strategy_Factory|
                                  ------------------------
                                             |
                                             |
                                            / \
                           ---------------------------------------
                           |                                      |
               --------------------------              ------------------------
               |Reactive_Strategy_Factory|             |Thread_Strategy_Factory|
               --------------------------              ------------------------

               ----------------------------            -------------------------
               |TAO_Server_Strategy_Factory|           |Application_Configurator|
               ----------------------------            -------------------------
                            |                                       |
                            |                                       |
                           / \                                     / \
                -----------------------                   -----------------------
                |Dynamic_Server_Factory|                  | TAO_Dynamic_Config  |
                -----------------------                   -----------------------
 

3. Current version of dynamicTAO (v0.1)

This sections gives an overview of how dynamicTAO changes the behavior of a running ORB. A more detailed explanation can be found in the Initial Design and Implementation Plan document. Before going throw the steps of reconfiguration, let's first see what each class of the dynamicTAO framework does:
 
 

• Application Configurator: Abstract class for application specific configurators built as an extension to the ACE layer. Subclasses of the ApplicationConfigurator contain knowledge about how to configure specific applications. Initially the only implemented application is the dynamicTAO ORB. ORB servants may use the same subsystem to reconfigure themselves. It offers a collection of methods for installing strategies in hooks accessing a list of hooks for the concrete application.

• Net_Broker: Abstract class that defines a bridge between remote network clients and the local Dynamic_Service_Configurator. Its pure virtual methods are:

• int initialize();
• int list_services()
• handle_request()

• DCP_Broker: Subclasses the class Net_Broker.

• Dynamic_Service_Config: Supplies common server operations for dynamic configuration of services during execution time. It spawns a thread that is responsible for listening for incoming requests. It uses the ACE_Service_Config internally.

• Dynamic_Server_Factory: It is a factory for TAO strategies. It can be dynamically configured to allow run-time adaptation. It receives commands from the Dynamic_Service_Config and provides creators for strategies loaded by the Dynamic_Service_Config

• TAO_Dynamic_Config: Encapsulates TAO_Specific knowledge of how to do the reconfiguration of services.

• Simple_Strategy_Factory: This is a wrapper that can be added to Strategies so they become ACE Services and, then, can be stored in the ACE Service Repository.

In order to understand how an application (in our case, TAO will be that application) can be reconfigured, it is necessary to describe Hooks, Services and implementations.

A hook is a reconfiguration point, where different strategies can be plugged, therefore, an application offers a number of hooks. The way it works is like this:

1. An application offers a number of hooks. TAO, for example, offers:

• Acceptor
• Concurrency_Strategy
• Connector
• Demarshalling_Strategy
• Demultiplexing
• Dispatching
• Marshalling_Strategy
• Scheduling_Strategy

2. Services are plugged in those hooks (a service is an ACE_Service)
3. Finally comes the implementation, that by now is implemented as a dynamically loadable library. Implementations are assigned to a specific service, and services are plugged into hooks.

3. New version of dynamicTAO (v0.2)

Expected release date: Nov/10/1998.

1. Dynamic Configurator

The previous version of dynamicTAO only allowed the reconfiguration of the ORB by means of a client/server connection. It would be also necessary to allow applications running on top of the ORB to reconfigure its own ORB, since they have the knowledge of how to adapt to environmental changes. If the same model of reconfiguration is maintained, it means that the servant that executes the CORBA object, would have to connect to the net_broker by using a socket, which would be inefficient, since both of them (the servant and the net_broker) belong to the same process. Besides being inefficient, it is also non secure, since any other application would be able to reconfigure any ORB.

What is needed is a way of integrating the dynamic reconfiguration functionality within the ORB, in such a way that a meta-interface could be exported to the servant. A good way of exporting this meta-interface is by adding a new component to the ORB, and specifying its behavior by means of IDL. Following this approach, the meta-interface is being implemented as a CORBA object (in the same way that the object adapter is implemented). From here on, the component (CORBA object) that will implement this meta-interface, will be called Dynamic Configurator.

A servant will be able to ask for a reference to the Dynamic Configurator by following the same procedure used to get a reference to the portable object adapter (POA) or the ORB interface. This approach not only allows seamless integration with the ORB, but it also lets applications decide who else can modify the behavior of its ORB by exporting the reference of the Dynamic Configurator to whatever client they consider appropriate.

The ongoing work of building the new component requires:

1. Defining the IDL interface. It can be done by following the functionality offered by the DCP_Broker.
2. Modifying TAO's ORB_Core in order to add the new component. It can be done by looking where the POA is defined and how can a servant access it.

2. Multiple Versions

Dynamic TAO is based in the Dynamic_Service_Config pattern in order to allow run-time configuration of the ORB. This pattern uses the ACE_Service_Config to store the strategies that are hooked into the ORB. Previous version of DynamicTAO allowed the user to have only one implementation of a strategy loaded into memory at the same time, because the name used to store different implementations of  a service in the ACE_Repository was always the same (i.e. Thread_Strategy, Reactive_Strategy, etc. were stored in the ACE_Repository under Concurrency). This behaviour presents some problems:

1. When the user decides to load a new strategy for a service,  the Dynamic_Service_Config looks for an existing instance of  that service within the ACE_Repository, deletes it, unloads the associated library (strategy), loads the new library into memory and creates an instance of it inside the ACE_Repository. The problem is that until the user decides to hook the new implementation of the service, the hook for that service contains an invalid reference (the library has been unloaded).
2. It is possible to have several servants running on top of an ORB, and each of them use a different strategy, or it is possible to have a servant that changes very frequently a specific strategy. In both cases if only one strategy can be at memory at the same time, it means that dynamically linked libraries will have to be loaded and unloaded every time, making the system run very slow.
3. Finally, it would be possible to have several versions of the same strategy, so it would be needed to have all of them loaded into memory in order to switch from one to another, depending on the servant issuing the request (this problem is similar to the previous one).

1. Subclass the ACE_Repository and specialize it, so it can work with several services with the same name (by adding extra information to the nodes stored in the list).
2. Keep the same ACE_Repository and add extra functionality to the Dynamic_Service_Config pattern, in order to cope with several versions of the same service.

3. Protect dynamicTAO so the currently hooked service cannot be removed

DynamicTAO has been modified in such a way that it is not possible to remove a service if it is the one that is hooked. Allowing that has no sense since the hook would be unstable, and the whole ORB could crash in case of an incoming request.
 

4. Avoiding loading a strategy that has already been loaded

In order to to avoid problems with dlls that have global variables (static methods), the method load_library checks wheter or not the dll has already been loaded into memory. It uses the list of versions that has been explained in section 3.2 to store the name of the implementations already loaded.
 
 

4. New version of dynamicTAO (v0.3)

(Expected release date: January/99) 

(Most of the next points refer to the concurrency strategy. As soon as new strategies are developed, new information will be added)

1. Removing strategies from memory

There are some strategies that cannot be removed from memory inmediatly. This is the case of the thread-pool-strategy, which spawns threads that execute a method contained in the dll. Althoug the strategy can be unhooked, existing threads must continue executing by following the original strategy. The problem is that when the threads finish, they need to execute some code contained in the dll, so if it has been removed, the ORB will crash. A new mechanism is required in order to check whether a strategy can be unloaded or not.

The problem with unloading a strategy consists on having some thread that needs to execute code from that strategy at some point. That means that the strategy cannot be unloaded until all resources that need it (basically threads) have finished their execution or don't need to execute code from the strategy any more.

A generic solution consists on adding a new method to the strategy base class:

unload

This method informs the strategy that its unloading has been requested. If the method returns TRUE, then the strategy can be inmediately unloaded. If the answer is FALSE, then the strategy becomes responsible for interacting with the strategy_remover in order to get unloaded.

STUDY THIS CASE: Thread_Pool_Strategy loaded into memory. All threads waiting at the semaphore. Someone requests the removing of the service: The fini method is called, so the "release" method from the semaphore is called as many times as threads waiting. Threads start exiting. PROBLEM: The library is unloaded before all the threads have finished--> DEFINITELY THE PROTOCOL FOR UNLOADING LIBRARY MUST BE IMPLEMENTED!!  What must be done: First call the unload method. If TRUE is returned then unload it because for shure all the threads will be finished. Else do what it is specified before.
 

2. Changing a strategy and pending requests

When a strategy is being used, it normally receives incoming requests that must be managed. It is possible that depending on the strategy, some of the events get buffered until the strategy can take care of them. In this situation, a client could request the change of the strategy, which means that something should be done with those buffered requests.

It is important to realize that not all the strategies will have requests being buffered. For example, the thread-per-connection strategy, as soon as it receives a request, it serves it by activating the thread, so in this situation cannot be requests buffered. Maybe there will be requests blocked for a period of time while the strategy finishes activating the thread, but this blocked requests are not considered to be buffered, because the acceptor while take care of the activation of theses requests as soon as the strategy is ready to process another request. An example of a strategy that can have buffered requests is the thread-pool-strategy in the case where new requests arrive, but there are no more threads in the pool. In this case, is the own thread-pool strategy the one that blocks the requests because it cannot serve them. This is the case that will be discussed:

There are three possible solutions:

          Hold the strategy change request and the new incoming requests until the buffered requests have been managed by the strategy.

This solution could be the most interesting if we considered some kind of causal ordering. That is, it is possible that the user that issues the requests, knows about the strategy being used by the ORB, so it wants to have the requests served by that strategy. The user (or client application) knew about the strategy, so if there is another client that decides to change the strategy, and decides to do so AFTER the first client has sent the messages, the change of the strategy should be postponed until the buffer empties. Of course, new requests coming after this change event will be blocked and served by the new strategy. Maybe a new kind of strategy could be used in order to decide  how to change strategies (though we will have the same problem when changing this strategy. Recursive problem!!).
 

Change the strategy and return exceptions for those requests that were in the buffer (and don't serve them).

This would be the easiest solution, and would leave all the work to the client, since it would have to decide what to do in case of this type of exceptions. The client should be unaware of this topic, since it is something that can be done in the ORB.

         Change the strategy and provide to it a list with the buffered requests.

The idea behind a reflective ORB is to provide the best possible service to the user, so if the decision of changing a strategy is taken, it is because some heuristic estimated that changing the strategy was needed in order to provide the performance expected by the user. From this point of view, buffered requests SHOULD be served with the new strategy. We must think from the point of view of the ORB and not from the user's perspective. In order to manage the buffered incoming requests properly, it is required that the strategy about to be unhooked notifies the new strategy about the fact that there are requests that should be handled. This notification must be done in a very generic way, because a strategy will now nothing about the next strategy that is going to be hooked. Basically, the scenario is this: there are several requests that are waiting to be served, and this requests are nothing more than SVC_HANDLERs pointers, stored in a data structure. In order to overcome this problem in a generic way, changing the base class of the strategies is needed (for example, ACE_Concurrency_Strategy). Two new methods must be added to this base class:

waiting_requests

This method returns an ACE list with the SVC_HANDLERs that must be still processed. The class that defines the nodes of the list will be provided in order to standardize the list

process_previous_waiting_requests

Receives an ACE list as a parameter and it is responsible for serving the requests stored in the list.

With these two methods, the only thing that is needed is that when a new strategy is going to be hooked by a subclass of application configurator, before hooking it, call the "waiting_requests" method of the current hooked strategy, hook the new strategy and pass it the requests list by calling "process previous waiting requests". The only problem of this approach is that each subclass of Application_Configurator must follow this two method invocation procedure, but this does not have to be considered a problem, since the user must know about this fact and must be free to handle this situation as it wants. The truth is that by following this approach, the user can decide to do any of the three points stated before: it could use the current strategy to serve the requests and after it will hook the new strategy, it could return an exception to the users, or it could handle the requests by itself.

3. Added a new DCP command

In order to learn about the currently hooked service, a new command has been added to the protocol: get_hooked_service
 

4. Synchronize strategy switching

If a request to change a strategy arrives while the strategy is doing some work, the change should be postponed until the strategy finishes. This can be done by using a mutex, in such a way that when the strategy starts executing one of its methods, it acquires the semaphore and releases it before leaving the method. In order to change the strategy, the application configurator needs to acquire the mutex first.

The mutex must be provided by the application configurator, and the strategies must get the mutex from that object.  THERE MUST BE ONE MUTEX PER HOOK!! We cannot block all the strategies!!!

Attention: It is not needed to synchronize the method execution. Because the only thing that could happen is that we will call the previous strategy. The only thing that must be taken into consideration is the removing of a strategy.
 

Please, send you suggestions and comments to Manuel Roman (mroman1@uiuc.edu).