5.1.4.1 Multiprocessing on Intels

On SMP system we need to locate the current shuttle. We have two methods:

1.

A word in the bottom of the kernel stack can be used to keep a pointer to the current shuttle. If we maintain kernel stacks aligned to a multiple of a given value (OFF_MDEP_KSTK_MAX) we can find the bottom of the stack by simple arithmetic.

<Other public methods of off_mdepShtl with fixed kernel stacks. >=
static off_Shtl *self(void) { 
  return (off_Shtl*)( (get_esp()&OFF_MDEP_KSTK_MSK) +
                       OFF_MDEP_KSTK_MAX-sizeof(off_Shtl*) );
}

2.

Alternatively, the current processor is asked for the pointer to the current shuttle. It can keep such pointer.

We employ the last one.

To support multiprogramming on Intels we use a single Intel task state segment (TSS). Such TSS is provided by the OSKit (in base_tss, of type x86_tss) and it is encapsulated in the machine dependent shuttle service.

The TSS is only used to setup the kernel stack automatically on traps and interrupts, thus the OSKit base_tss is shared and register frames are kept in the kernel stacks.

Context switch is done by hand. It has to be concerned only about kernel state, because user state is handled automatically by the interrupt procedure with some help from the hardware.

When about to context switch, we push the kernel registers into the kernel stack and store both the new (kernel) stack pointer and the pointer to the saved kernel registers in the machine dependent shuttle.

<Other private members of off_mdepShtl. >= (<-U)
off_mdepStck      s_ksp;        // Kernel stack pointer
off_mdepProcRegs *s_kregs;      // Saved kernel registers

<Off machine dependent shuttle dependencies. >+= (U->) [<-D]
class off_mdepProcRegs;
#include <klib/mdep/mStck.h>

Then (after adjusting shuttle properties) we switch the machine dependent part of a shuttle by reloading a previously saved set of kernel registers. We can find them by looking at the s_ksp member stored before.

This process has to be done with interrupts cleared and the esp0 (ring 0 stack) member of the base_tss needs to be updated before interrupts are enabled to the real (i.e. current) value.

Machine dependent context switch is performed by switch_to. It is called from the outgoing shuttle with the machine dependent part of the incoming shuttle.

<Other public methods of off_mdepShtl. >= (<-U) [D->]
// Does a context switch to the other shuttle. 
void switch_to(off_mdepShtl *other);

<off_mdepShtl::switch_to implementation. >= (U->)
// Does a context switch to the other shuttle. 
void off_mdepShtl::switch_to(off_mdepShtl *other)
{
  off_mdepProcRegs kregs;       // Buffer to save registers. 
  off_Processor::clri();
  s_kregs=&kregs;
  if (!off_mdep_save_kregs(&kregs)){
    off_mdep_load_kregs(other->s_kregs);
    otsan();
  }
  off_Processor::seti();
}

<Off machine dependent shuttle implementation dependencies. >= (U->)
#include <shtl/mdep/kregs.h>    // for save_ and load_kregs

Where we used a couple of low-level routines to do the dirty work.

<Off machine dependent kernel register switching. >= (U->)
// Save current registers in the buffer at kregs.
// Returns TRUE when called to save registers, 
// FALSE when registers are reloaded. 
int off_mdep_save_kregs(off_mdepProcRegs *kregs);

// Load previously saved registers from kregs.
// It should never return and the previous off_mdep_save_kregs
// will return with FALSE right after reload. 
void off_mdep_load_kregs(off_mdepProcRegs *kregs);

<Off machine dependent kernel register switching dependencies. >= (U->)
#include <flux/types.h>         // for boolean_t natural_t et al.

The implementation uses a little bit of assembly glue which saves every register into a mdepProcRegs buffer. It returns 1, i.e. TRUE to the caller.

<off_mdep_save_kregs implementation. >= (U->)
ENTRY(off_mdep_save_kregs)
ENTRY(_off_mdep_save_kregs)
    movl 4(%esp),%eax           /* fetch buffer (trap_state) */
    movl %edi,16(%eax)          /* save edi in trap_state->edi */
    movl %esi,20(%eax)          /* save esi in trap_state->esi */
    movl %ebp,24(%eax)          /* save ebp in trap_state->ebp */
    movl %ebx,32(%eax)          /* save ebx in trap_state->ebx */
    movl %edx,36(%eax)          /* save edx in trap_state->edx */
    movl %ecx,40(%eax)          /* save ecx in trap_state->ecx */
    movl $0,44(%eax)            /* save 0   in trap_state->eax */
    pushf                       /* save flags...  */
    popl %ecx
    movl %ecx,64(%eax)          /*       ...in trap_state->eflags */
    popl %ecx                   /* save caller eip...             */
    movl %ecx,56(%eax)          /*       ...in trap_state->eip    */
    movl %esp,28(%eax)          /* save esp in trap_state->esp    */
    movl $1,%eax                /* return TRUE...                 */
    jmp  *%ecx                  /*       ...to the caller         */

Defines off_mdep_save_kregs (links are to index).

Note how off_mdep_load_kregs will reload a register set as saved by the previous routine. It returns 0, i.e. FALSE, to the caller of mdep_save_kregs.

<off_mdep_load_kregs implementation. >= (U->)
ENTRY(off_mdep_load_kregs)
    movl   4(%esp),%eax         /* fetch buffer (trap_state)         */
    movl  16(%eax),%edi         /* fetch edi from trap_state->edi    */
    movl  20(%eax),%esi         /* fetch esi from trap_state->esi    */
    movl  24(%eax),%ebp         /* fetch ebp from trap_state->ebp    */
    movl  32(%eax),%ebx         /* fetch ebx from trap_state->ebx    */
    movl  36(%eax),%edx         /* fetch edx from trap_state->edx    */
    movl  40(%eax),%ecx         /* fetch ecx from trap_state->ecx    */
    pushl 64(%eax)              /* fetch     from trap_state->eflags */
    popf                        /*       eflags                      */
    movl  28(%eax),%esp         /* fetch esp from trap_state->esp    */
    pushl 56(%eax)              /* set the caller eip to trap_state->eip */
    movl  $0,%eax               /* return FALSE                      */
    ret                         /*    to the caller eip              */
ENTRY(_off_mdep_load_kregs)

Defines off_mdep_load_kregs (links are to index).

To startup a new shuttle, a kernel stack is allocated and a fake kernel register frame is pushed on it. Such frame corresponds to the execution of the upcall routine. The method in charge of it all is setup_stack.

<Other protected methods of off_mdepShtl. >= (<-U)
// Setup an initial kernel stack for this shuttle so that it enters
// userland with the program counter and stack pointer specified. 
void setup_stack(register_t sp, register_t pc);

The arguments (pushed by hand on the stack) will make it to call upcall to transfer the control to the user level entry point using the supplied stack and program counter (which are those specified as initial values by the shuttle creator). Upon upcall return, the terminated kernel routine is called, which simply destroys the shuttle--this happens only when the user main procedure is terminated.

We are now in position to create a shuttle for Intel architectures.

<Other public methods of off_mdepShtl. >+= (<-U) [<-D]
// Creates a machine dependent shuttle. 
off_mdepShtl(register_t pc=0, register_t sp=0, register_t ksp=0);

We setup the kernel stack and

<off_mdepShtl::off_mdepShtl implementation. >= (U->)
// Creates a machine dependent shuttle. 
off_mdepShtl::off_mdepShtl(register_t pc, register_t sp, register_t ksp) :
  s_ksp(ksp),s_kregs(0)
{
  if (sp)
    setup_stack(sp,pc);
}

The party is at setup_stack.

<off_mdepShtl::create_stack implementation. >= (U->)

void upcall(void)
{
  otsan();
}

// Creates an initial kernel stack for a shuttle so that it enters
// userland with the program counter and stack pointer specified. 
void off_mdepShtl::setup_stack(register_t sp, register_t pc)
{
  register_t inital_sp;
  off_ProcRegs *r;
  s_ksp.push(sp);               // User sp    (shtl_init arg)
  s_ksp.push(pc);               // User pc    (shtl_init arg)
  s_ksp.push(0);                // fake return eip for shtl_init
  inital_sp=s_ksp;
  r = new(s_ksp.alloca(sizeof(off_ProcRegs))) off_ProcRegs; 
  r->p_mdep.set_flags(get_eflags());
  r->p_mdep.set_pc((register_t)upcall);
  r->p_mdep.set_sp(inital_sp);
  s_ksp=inital_sp;
  s_kregs=&r->p_mdep;

}

After the stack is setup, the new shuttle is ready for execution.