5.1.2 Shuttles

A shuttle is a set of property values, including a processor context. It is the only schedulable entity provided by the kernel. Shuttles run when installed in the run queue of a given processor (as we saw in section ). Each shuttle has its own ShtlPropAllocator which can be accessed by the accessor get_allocator.

<Off shuttle. >= (U->)
// A shuttle. 
//
class off_Shtl : public off_AbsResUnit {
public:

  // Returns the property allocator for this shuttle.
  off_ShtlPropAllocator *get_allocator(void);

  <Other public methods of off_Shtl. >
protected:
  <Other protected methods of off_Shtl. >
private:
  off_mdepShtl s_mdep;          // Processor context.
  off_ShtlPropAllocator s_palloc; // Property allocator.
  <Other private members of off_Shtl. >
};

Defines off_Shtl (links are to index).

<Off shuttle dependencies. >= (U->) [D->]
#include <klib/AbsResUnit.h>     // for off_AbsResUnit

<Off shuttle dependencies. >+= (U->) [<-D->]
#include <shtl/ShtlPropAllocator.h>

<off_Shtl::get_allocator implementation. >= (U->)
// Returns the property allocator for this shuttle.
off_ShtlPropAllocator *off_Shtl::get_allocator(void)
{
  return &s_palloc;
}

Shuttle properties are entities used to specify which kind of resources (e.g. DTLBs, IO maps, etc.) should be readily available for a shuttle to run. Specific resource instances are identified by property values.

There are three different kinds of properties:

1.

Properties folded into the shuttle. In those cases where a given property is almost always used we hardwire it into the shuttle code (e.g. we will be seeing how processor registers are handled right away by the shuttle itself).

2.

Properties which are serviced into the kernel. In those cases where a resource has to be made available by the kernel for a shuttle to run, the property involved is handled by a function call inside the kernel.

3.

Properties which are serviced by the user. Every user can define its own properties and arrange for handlers to be called (in the property user context) to service them.

Only the last two kinds of properties will appear into the s_props member.

A shuttle is created with a prespecified protection and domain. Besides, a prototype shuttle can be given to initialize the set of properties. Only for the root (the very first) shuttle is such prototype not supplied; in any other case the current shuttle is always used as a prototype. The program counter and (user^5.1) stack pointer must be always supplied.

<Other public methods of off_Shtl. >= (<-U) [D->]
// Creates a shuttle. 
off_Shtl( const off_Protection &p, 
          const off_prtl_id_t &domain,
          const off_shtl_id_t &id,
          off_ShtlSrv     *srv,
          register_t       pc,
          register_t       sp,
          register_t       ksp,
          off_Shtl        *proto );

<Off shuttle dependencies. >+= (U->) [<-D->]
#include <klib/err.h>           // for err_t 
#include <klib/prot.h>          // for off_Protection
#include <klib/Magic.h>         // for magic numbers
class off_ShtlSrv;              // to break the circular dependency
#include <shtl/mdep/mShtl.h>    // for off_mdepShtl

System users go through a different constructor, because most of the shuttle fields are pre-initialized.

<Other public methods of off_Shtl. >+= (<-U) [<-D->]
// Creates a shuttle.
off_Shtl(const off_Protection &p, 
         const off_prtl_id_t &domain,
         register_t       pc,
         register_t       sp,
         register_t       ksp,
         off_Shtl        *proto );

The kernel stack pointer and user stack pointer and program counter is sent to the machine dependent shuttle information, which includes the processor context.

<off_Shtl::off_Shtl implementation. >= (U->) [D->]
// Creates a shuttle.
off_Shtl::off_Shtl(const off_Protection &p, 
                   const off_prtl_id_t &domain,
                   register_t       pc,
                   register_t       sp,
                   register_t       ksp,
                   off_Shtl   *proto ) :
  off_AbsResUnit(p,domain),
  s_mdep(pc,sp,ksp),
  <Initialize Shtl::s_palloc. >,
  <Initialize other aggregate members of off_Shtl. >
{
  <Initialize other private members of off_Shtl according to proto. >
}

Remaining shuttle data is pre-initialized. First, a zero-argument constructor permits declaration of uninitialized shuttles.

<Other public methods of off_Shtl. >+= (<-U) [<-D->]
//  Allows uninitialized shuttles.
off_Shtl(void);

<off_Shtl::off_Shtl implementation. >+= (U->) [<-D->]
//  Allows uninitialized shuttles.
off_Shtl::off_Shtl(void) :
  off_AbsResUnit(off_Protection(), OFF_PRTL_NULL, 
                 OFF_SHTL_NULL, OFF_MAGIC_SHTL, NULL)
{;}

Later, when shuttle servers are being created, another constructor is used to pre-initialize those shuttle fields which are unlikely to change.

<Other public methods of off_Shtl. >+= (<-U) [<-D->]
// Pre-initializes a shuttle.
off_Shtl(const off_shtl_id_t &id, const off_magic_t &m, off_ShtlSrv *c);

<off_Shtl::off_Shtl implementation. >+= (U->) [<-D]
// Pre-initializes a shuttle.
off_Shtl::off_Shtl(const off_shtl_id_t &id, 
                   const off_magic_t &m, off_ShtlSrv *c) :
  off_AbsResUnit(id,m,(off_AbsCompResource*)c)
{;}

Where the magic number has to be defined.

<Off magic numbers. >+= [<-D]
OFF_MAGIC_SHTL, // Shtl magic nb.

We can now add another case to the implementation of Magic::nameof.

<off_Magic::nameof case for m_numbers. >+= [<-D]
case OFF_MAGIC_SHTL:
  return "off_Shtl";

Shuttle users can check their references to processors using this method.

<Other public methods of off_Shtl. >+= (<-U) [<-D->]
// Does this look like a Shtl?
inline boolean_t valid(void) const;

<off_Shtl::valid implementation. >= (U->)
// Does this look like a Shtl?
inline boolean_t off_Shtl::valid(void) const 
{ return get_magic() == OFF_MAGIC_SHTL; }

The new container can also implement its get_umagic method.

<Other protected methods of off_ShtlSrv. >= (<-U) [D->]
virtual off_magic_t get_umagic(void) { return OFF_MAGIC_SHTL; }

\subsection{Shuttle allocation}

A shuttle allocator is a block allocator instantiated to maintain shuttles.

<Off shuttle allocator. >= (U->)
// A Shuttle allocator.
//
class off_ShtlAllocator : public off_TBlockAllocator<off_LShtl> {
public:
  // Allows declarations of uninitialized shuttle allocators
  off_ShtlAllocator(void){;}

  const off_ShtlAllocator &operator=(const off_ShtlAllocator &other) { 
    return (*this=other);
  }

  <Other public methods of off_ShtlAllocator. >
private:
  <Other private members of off_ShtlAllocator. >
};

Defines off_ShtlAllocator (links are to index).

As the block allocator uses a liked free-list, the array nodes must subclass DLinkedNode.

<Off linkable shuttle. >= (U->)
// A linkable off_Shtl.
//
class off_LShtl : public DLinkedNode {
public:
  off_Shtl l_shtl;              // The shuttle

  // To allow uninitilized LShtls.
  off_LShtl(void) {;}

  <Other public methods of off_LShtl. >
};

Defines off_LShtl (links are to index).

<Off shuttle allocator dependencies. >= (U->) [D->]
#include <klib/BlockAllocator.h> // for off_TBlockAllocator
#include <shtl/Shtl.h>          // for off_Shtl

An indexer is maintained by the allocator. Besides, a kernel virtual memory region is provided by the creator.

<Other private members of off_ShtlAllocator. >= (<-U) [D->]
static off_Indexer<off_LShtl> s_idx; 

<Off shuttle allocator static members. >= (U->)
off_Indexer<off_LShtl> off_ShtlAllocator::s_idx; 

<Off shuttle allocator dependencies. >+= (U->) [<-D]
#include <klib/BlockAllocator.h>   // for off_BlockAllocator
#include <shtl/Shtl.h>          // for off_Shtl et al.

The allocator is initialized as follows.

<Other public methods of off_ShtlAllocator. >= (<-U) [D->]
// Creates a shuttle allocator.
off_ShtlAllocator(off_Exhausted *r, off_KVMRegion *reg);

The region is used to provide memory for the (growing) array of shuttles. A pointer to the start of such array is kept inside the shuttle allocator so we could redefine the operator +.

<Other private members of off_ShtlAllocator. >+= (<-U) [<-D]
off_LShtl *s_shtls;

<off_ShtlAllocator::off_ShtlAllocator implementation. >= (U->)
// Creates a shuttle allocator.
off_ShtlAllocator::off_ShtlAllocator(off_Exhausted *r, off_KVMRegion *reg) :
  off_TBlockAllocator<off_LShtl>(r,(const off_Indexable*)&s_idx,reg),
  s_shtls((off_LShtl*)reg->get_start())
{;}

<Other public methods of off_ShtlAllocator. >+= (<-U) [<-D->]
inline off_LShtl *operator +(vm_offset_t at);

<off_ShtlAllocator::operator+ implementation. >= (U->)
inline off_LShtl *off_ShtlAllocator::operator +(vm_offset_t at)
{
  return s_shtls+at;
}

The allocator must be initialized when the shuttle server is started, not when it is instantiated as there might be no KVM by that time. The revocator is the shuttle server itself, as set at construction time; the region is also created at construction time.

<Initialize ShtlSrv::s_alloc. >= (<-U)
new(&s_alloc)  off_ShtlAllocator((off_Exhausted*)this,kvm.create_region());

<Off shuttle server implementation dependencies. >= (U->) [D->]
#include <dmm/KVM.h>            // for kvm

Where we used the new operator trick.

<Other public methods of off_ShtlAllocator. >+= (<-U) [<-D]
void *operator new(size_t s, off_ShtlAllocator *a) { (void)s; return a; }

The notify method must thus be provided by IO banks to implement the resource revocator signature. However, such events are silently discarded in this implementation.

<Other public methods of off_ShtlSrv. >+= (<-U) [<-D->]
// Notifies that shuttles are exhausted.
void notify(void);

Once the allocator has been initialized, the Shuttle server can be indexed to obtain references to shuttles; they can be allocated and deallocated.

<Other public methods of off_ShtlSrv. >+= (<-U) [<-D->]
//Gets a Shuttle from its distributed address
inline off_Shtl *operator+(off_offset_t s);
// Allocates a shuttle.
off_Shtl *alloc(const off_shtl_id_t &at, boolean_t use_revocation=FALSE);
// Allocates any shuttle.
off_Shtl *alloc(boolean_t use_revocation=FALSE);
// Deallocates a shuttle
void free( const off_Shtl *s );

<off_ShtlSrv::operator+ implementation. >= (U->)
//Gets a Shuttle from its distributed address
inline off_Shtl *off_ShtlSrv::operator+(off_offset_t s)
{ 
  assert(valid());
  return &(s_alloc+s)->l_shtl;
}

Shuttles are preinitialized when they are allocated.

<off_ShtlSrv::alloc and free implementation. >= (U->)
// Allocates a shuttle.
off_Shtl *off_ShtlSrv::alloc(const off_shtl_id_t &at, boolean_t use_revocation)
{
  off_LShtl *ls;
  assert(valid());
  w_lock();
  ls=s_alloc.allocate(at,use_revocation);
  w_unlock();
  if (ls) {
    (void) new(ls) off_LShtl(at,this);
  }
  return (ls)?(&ls->l_shtl):(off_Shtl*)NULL;
}

// Allocates a shuttle.
off_Shtl *off_ShtlSrv::alloc(boolean_t use_revocation)
{
  off_LShtl *ls;
  assert(valid());
  w_lock();
  ls=s_alloc.allocate(use_revocation);
  w_unlock();
  if (ls) {
    off_shtl_id_t sid(get_id(),s_alloc.pos(ls));
    (void) new(ls) off_LShtl(sid,this);
  }
  return (ls)?(&ls->l_shtl):(off_Shtl*)NULL;
}

// Deallocates an Shuttle
void off_ShtlSrv::free( const off_Shtl *s ) 
{ 
  off_LShtl *ls=(off_LShtl*)(((char*)s)-(int)&off_LShtl::l_shtl);
  assert(valid());
  w_lock(); 
  s_alloc.deallocate(ls); 
  w_unlock(); 
}

<Off shuttle server dependencies. >+= (U->) [<-D->]
#include <assert.h>             // for assert

Shuttle preinitialization use a new operator for LShtl returning its single argument.

<Other public methods of off_LShtl. >= (<-U) [D->]
void *operator new(size_t s, off_LShtl *p) { (void)s; return (void*) p; }

A preinitialization constructor for LShtls is also used.

<Other public methods of off_LShtl. >+= (<-U) [<-D]
off_LShtl(const off_shtl_id_t &id, off_ShtlSrv *ssrv) :
  l_shtl(id,OFF_MAGIC_SHTL,ssrv)
{;}

Shuttles are allocated and deallocated with new and delete operators. The operator new uses the shuttle server alloc method.

<Other public methods of off_Shtl. >+= (<-U) [<-D->]
void *operator new(size_t s, const off_shtl_id_t &at,
                   boolean_t use_revocation=FALSE);
void *operator new(size_t s, boolean_t use_revocation=FALSE);
void operator delete(void *p);

<off_Shtl::operator new implementation. >= (U->)
void *off_Shtl::operator new(size_t s, boolean_t use_revocation=FALSE)
{
  off_Shtl *sh=nd.get_shtl().alloc(use_revocation);
  (void)s;
  assert(s && nd.get_shtl().valid());
  assert(!sh || sh->valid());
  return (void*)sh;
}
void *off_Shtl::operator new(size_t s, const off_shtl_id_t &at,
                             boolean_t use_revocation=FALSE)
{
  off_Shtl *sh=nd.get_shtl().alloc(at,use_revocation);
  (void)s;
  assert(s && nd.get_shtl().valid());
  assert(!sh || sh->valid());
  return (void*)sh;
}

<off_Shtl::operator delete implementation. >= (U->)
// Deallocates this page frame.
void off_Shtl::operator delete(void *p)
{
  off_Shtl *sh=(off_Shtl*)p;
  assert(sh && sh->valid());
  nd.get_shtl().free(sh);
}

\subsection{Shuttle revocation}

Each shuttle servers delivers revocation events to its exception portal or r_domain, in case that portal has been set. The owner of that portal is the referee for shuttle revocation inside the shuttle server.

<off_ShtlSrv::notify implementation. >= (U->)
// Notifies that shuttles are exhausted.
void off_ShtlSrv::notify(void)
{
  assert(valid());
  if (get_domain()==OFF_PRTL_NULL)
    return;
  else {
    ;
#if 0                           // XXX fix this
    off_ExhaustReq m(OFF_MAGIC_SHTLSRV,get_id());
    prtl.kpct(get_domain(),off_Shtl::self(),sizeof(m),0,&m,NULL,0);
#endif
  }
}

<Off shuttle server implementation dependencies. >+= (U->) [<-D->]
#include <prtl/ex.h>            // for off_ExhaustReq
//#include <prtl/PrtlSrv.h>       // for prtl.            XXX fix this
#include <hw/Processor.h>       // for Processor and self.

\subsection{Shuttles, stacks and SMP}

On SMP machines, self can be used to determine the shuttle being run in the current processor.

<Other public methods of off_Shtl. >+= (<-U) [<-D->]
// Returns the current shuttle in the current processor.
static off_Shtl *self(void);

The implementation is machine dependent and uses information found in the shuttle kernel stack.

<off_Shtl::self implementation. >= (U->)
// Returns the current shuttle in the current processor.
off_Shtl *off_Shtl::self(void) 
{
  return off_Processor::self()->get_current();
}

<Off shuttle late dependencies. >= (U->)
#include <hw/Processor.h>

Our current stack pointer is a value so commonly referenced that shuttles provide means to access it without a reference to the current shuttle registers. Note that it returns kernel addresses, because it is called within the kernel.

<Other public methods of off_Shtl. >+= (<-U) [<-D->]
// Get or set the PC/SP.
static inline register_t get_sp(void);

<off_Shtl::get_pc implementation. >= (U->)
// Get or set the PC/SP.
inline register_t off_Shtl::get_sp(void) {return off_mdepShtl::get_sp();}

\subsection{Shuttle properties}

Properties are used to specify which resources (e.g. DTLBs, IO maps, etc.) should be readily available for the shuttle to run. Specific resource instances are identified by a property value (of type off_id_t), resource kinds are identified by properties (of type off_prop_t).

<Off shuttle property identifer. >= (U->)
// Properties
typedef natural_t off_prop_t;
const off_prop_t OFF_PROP_NULL=(off_prop_t)-1;

Defines OFF_PROP_NULL, off_prop_t (links are to index).

The operational meaning of a property is clear if we consider that the shuttle server code is guaranteed to:

• Call a property switch function, pswitch, supplying the property value as an argument every time a shuttle using it is about to run.
• Call a property set function, pset, supplying the value being set every time the property value is (re)set in a shuttle.
• Call a property cleanup function, pclr, supplying the value being cleaned up every time the property value is about to be set. \end{itemize}

  The first function is intended to be used as a command^5.2 or callback for context switches; The last two functions, are provided for those cases when the hardware is switching the context automatically but some work should be done to install (or deinstall) a property value in a given shuttle.

  Thus, a property can be defined as follows:

  <Off shuttle property. >= (U->)
  // A shuttle property.
  //
  class off_ShtlProp {
  public:
    <Other public methods of off_ShtlProp. >
  private:
    off_id_t p_val;                // Property value.
    off_ShtlPropSrv *p_srv;        // Property server.
    off_ShtlPropSrv *p_swsrv;      // Property server for switch() (cached).
  };
  
  

  Defines off_ShtlProp (links are to index).

  <Off shuttle property dependencies. >= (U->) [D->]
  #include <klib/err.h>           // for err_t 
  #include <klib/ids.h>           // for OFF_PRTL_NULL et al.
  #include <flux/types.h>         // for boolean_t natural_t et al.
  

  Where the property server must implement this interface\footnote{An abstract class could do the job, but GNU C++ signatures provide subtype polymorphism independently from the inheritance hierarchy.:

  <Off shuttle property server. >= (U->)
  // The interface of a property server. 
  //
  signature off_ShtlPropSrv {
  
    // Switches property values. 
    // Returns either 0 or an error code.
    err_t pswitch(const off_id_t &to, const off_shtl_id_t &s);
    
    // Set or clear the property at a given shuttle. 
    err_t pset(const off_id_t &pval,const off_shtl_id_t &s);
    void  pclr(const off_id_t &pval, const off_shtl_id_t &s);
  
    // Is pswitch used?
    boolean_t needs_switch(void);
  };
  
  

  Defines off_ShtlPropSrv (links are to index).

  When a property is being created (i.e. defined), its value and server must be provided.

  <Other public methods of off_ShtlProp. >= (<-U) [D->]
  // Creates this property.
  off_ShtlProp(off_ShtlPropSrv *srv, const off_id_t &val);
  

  Although another constructor is provided to declare uninitialized properties.

  <Other public methods of off_ShtlProp. >+= (<-U) [<-D->]
  // Creates this property as unitialized
  off_ShtlProp(void) : 
    p_val(OFF_ID_NULL)
  {;}
  

  The property calls the server needs_switch to determine if the switch function is needed or not, as there are properties which do not need a switch function (eg. those properties handled by the hardware, like the IOPL, and those properties which are passive, like user ids).

  <off_ShtlProp::off_ShtlProp implementation. >= (U->)
  off_ShtlProp::off_ShtlProp(off_ShtlPropSrv *srv, const off_id_t &val) :
    p_val(val), p_srv(srv)
  {
    if (srv->needs_switch())
      p_swsrv=srv;
    else
      *(void**)&p_swsrv=NULL;
  }
  
  

  The preperty value can be changed later at any time and can be consulted.

  <Other public methods of off_ShtlProp. >+= (<-U) [<-D->]
  // Sets a new value for this property.
  inline void set_val(const off_id_t &val);
  // Returns the property value.
  inline const off_id_t &get_val(void) const;
  
  

  <off_ShtlProp::set_val and get_val implementation. >= (U->)
  // Sets a new value for this property.
  inline void off_ShtlProp::set_val(const off_id_t &val) { p_val = val; }
  // Returns the property value.
  inline const off_id_t &off_ShtlProp::get_val(void) const { return p_val; }
  
  

  The property server, implementing the functions servicing the property, is reachable through the property.

  <Other public methods of off_ShtlProp. >+= (<-U) [<-D->]
  // Switches property values. 
  // Returns either 0 or an error code.
  inline err_t pswitch(const off_shtl_id_t &s);
    
  // Set or clear the property at a given shuttle. 
  err_t pset(const off_id_t &pval, const off_shtl_id_t &s);
  
  

  <off_ShtlProp::pswitch implementation. >= (U->)
  // Switches property values. 
  // Returns either 0 or an error code.
  inline err_t off_ShtlProp::pswitch(const off_shtl_id_t &s)
  {
    if ((void*)p_swsrv!=NULL)
      return  p_swsrv->pswitch(p_val,s);
    else
      return EOK;
  }
  

  <off_ShtlProp::pset and pclr implementation. >= (U->)
  // Set or clear the property at a given shuttle. 
  err_t off_ShtlProp::pset(const off_id_t &pval, 
                           const off_shtl_id_t &s)
  {
    p_srv->pclr(p_val,s);
    p_val=pval;
    return p_srv->pset(p_val,s);
  }
  
  

  There are two kind of property servers, one using procedure calls for in-kernel properties and another using portal calls for off-kernel properties. The first one can be implemented using a pointer to an object matching the property server interface shown above. The second one can use this wrapper.

  <Off off-kernel shuttle property server. >= (U->)
  // A user-serviced property
  //
  class off_uShtlPropSrv {
  public:
    <Other public methods of off_uShtlPropSrv. >
  
    // Switches property values. 
    // Returns either 0 or an error code.
    err_t pswitch(const off_id_t &to, const off_shtl_id_t &s);
    
    // Set or clear the property at a given shuttle. 
    err_t pset(const off_id_t &pval,const off_shtl_id_t &s);
    void  pclr(const off_id_t &pval, const off_shtl_id_t &s);
  
    // Is pswitch used?
    boolean_t needs_switch(void);
    
  };
  
  

  Defines off_uShtlPropSrv (links are to index).

  Properties handled inside the kernel must be well-defined at every node in the network, user properties do not need to. For user defined properties, the portal servicing property switches is carried along with the property value in every shuttle. Every kernel can then blindly jump through that portal at any node. Of course, it is wise to have those portals replicated at every node because they will be used every shuttle switch.

  The user portal servicing a property is given when the user property is being created.

  <Other public methods of off_uShtlPropSrv. >= (<-U)
  // Creates a user property server dispatching to the given portal.
  off_uShtlPropSrv(const off_prtl_id_t &uobj, boolean_t needs_switch=FALSE);
  

  A flag is also given so that the shuttle server could know if it is necessary at all to callpswitch (to prepare the property for use).