C-MPI 1.

1 DESIGN DOCUMENT

System overview

The Virtual Interface Architecture is targeted as a communication infrastructure

for System Area Networks (SANs). It seeks to enable a wave of new cluster applications
that are able to harness the communication performance of these emerging networks. The
core of the specification is a set of principles outlining user-level networking -- where the
operating system is involved only in infrequent setup operations -- which establish the
structures for hardware protection checks and direct access to the network.
The VI Architecture eliminates the system-processing overhead of the traditional
model by providing each consumer process with a protected, directly accessible interface
to the network hardware - a Virtual Interface. Each VI represents a communication
endpoint. VI endpoint pairs can be logically connected to support bi-directional, point-to-
point data transfer. A process may own multiple VIs exported by one or more network
adapters. ParamNet-II NIC performs the endpoint virtualization directly and subsumes
the tasks of multiplexing, de-multiplexing, and data transfer scheduling normally

1
performed by an OS kernel and device driver. ParamNet-II NIC completely ensures the
reliability of communication between connected VIs.

The VI Architecture is comprised of four basic components:

1. Virtual Interfaces
2. Completion Queues
3. VI Providers,
4. VI Consumers.

Virtual Interface.
A Virtual Interface is the mechanism that allows a VI Consumer to directly
access a VI Provider to perform data transfer operations. A Vi consists of a set of
Receive/Transmit Queue.

Completion queue.
1. Means to select several completions among several VI’s.
2. The one-to-one connection model of the VI Architecture requires a large
number of VI's per user to achieve full connectivity between nodes in a
cluster. Polling or taking interrupts from each individual VI does not scale as
the number of VI's increases. Completion queues alleviate this problem by
providing a single notification point for an arbitrary number of work queues.

VI Provider
1. VI Provider is the set of hardware and software components responsible for
instantiating a Virtual Interface. The VI Provider consists of a ParamNet-II
(NIC) , Alteon Acenic and Device Driver.

2. ParamNet-II NIC implements the Virtual Interfaces and Completion Queues

and directly performs data transfer functions. ParamNet-II NIC device driver
performs the setup and resource management functions needed to maintain a
Virtual Interface between VI Consumers and VI NICs. These functions
include the creation/destruction of VIs, VI connection setup/teardown;
interrupt management and/or processing, management of system memory
used by the VI NIC, and error handling. Device Driver Agents interact with
VI NICs through standard operating system device management
mechanisms.

However Alteon Acenic NIC is a VI emulated device and does not have any
native hardware support for VI.

VI Consumer
1. The VI Consumer component consists of those objects that utilize the
resources of the VI Provider. The Consumer can either be a user application
that uses native VI primitives to conduct communication, or other
communication protocols (e.g. TCP/IP, Sockets, MPI) and OS communication
services layered over the VI Provider methods.

2
2. Integral to the VI Consumer is a VI user agent that is typically in the form of a
library, known as the Virtual Interface Provider Library (VIPL), linked into the
consumer application. The user agent abstracts the hardware and OS interface
details of different VI Provider implementations to provide a consistent
interface to the programmer. Legacy protocol stacks might employ the user agent
through native VI calls at their lower edge.
1.0 Acronyms, terms and definitions Used in CMPI Module

CCP-III

C-DAC communication Co-Processor-III

ParamNet
ParamNet-II is the prime interconnect fabric for connecting Compute
Nodes, File Servers and Graphics Nodes in C-DAC's range of parallel
computing systems
VIPL

Virtual interface provider library

Kernel Agent

A component of the operating systems required by the VI

architecture that subsumes the role of the device driver for the VI
NIC and includes the kernel software needed to register
communication memory and manage VIs.
Memory Registration
The act of creating a memory region. The memory registration operation
returns a Memory Handle that the process is required to provide with any
virtual address within the memory region.
VI
Virtual Interface. An interface between a VI NIC and a process
allowing a VI NIC direct access to process memory.
VIPL
A software component that enables an operating system
communication facility to utilize a particular VI provider.
VI Kernel Agent
The device driver for the VI NIC and include the kernel software needed
to register communication memory and manage Vis.

VI Provider
The combination of VI NIC and a VI Kernel Agent.
VI Consumer
A software process that consists of an application program, an operating
system communication facility, and a VI User Agent

3
CRE
Sun Cluster Runtime Environment(CRE) is a program
execution environment that provides basic job launching and
load balancing capabilities.

LSF Platform Computing’s Load Sharing Facility(LSF).

SCI Scalable Coherent Interface an interconnection technology.

2.0 Assumptions and constraints

6.1 VIA

Hardware assumptions: VIA assumes that the underlying network hardware

is stable and reliable.

Software assumptions: The source code base of M-VIA is from Linux

platform .We assume that equivalent kernel calls in the target platform (AIX) are
available.

Constraints : The number of VI’s supported and the total memory user can
register depend on the capabilities of NIC.

6.2 C-MPI

Hardware assumptions: Availability of ParamNet-II.

Software assumptions: MPI assumes the availability of Stable VIPL according to VIA
specification.

Constraints: Total number of MPI jobs are restricted by availability of VI at

hardware level and amount of memory registered.
3.0 Basic Design approach
Structured Design methodology is used.

4.0 Risks
Refer Risk management plan.

Hardware
1. IBM p630/p690 Power4 CPU based nodes
2. PARAMNet-II & Gigabit Ethernet (Alteon Acenic Tigon-II)

4
Data Communications

Paramnet-II brochures [10].

Software

Platform - P630/AIX5L.
Compiler - C compiler./ Fortran compiler.
Microcode - Protocol Controller, Receiver Macrocode for ParamNet-II NIC and
. Alteon Acenic Tigon-II.
Library- VIPL .

5
10.2 C-MPI Architecture.

Application

MPI API support

Plugin
Progress Engine
Transport
modules
TCP SHMEM VIA

VIPL
TCP Stack

ParamNet-II Gigabit Ethernet drivers

Network Drivers Drivers

Network Adaptors Memory ParamNet-II Gigabit ethernet

Fig 3.C- MPI Architecture.

The overall Sun MPI architecture is shown in Figure 1 above. Interconnect-

specific transport protocols are found in the bottom layer. Based on the interfaces
available at run time, the library picks the most efficient protocol to use between
each pair of processes.

6
10.3 C-MPI ADI architecture.

Fig 4. C-MPI ADI Architecture model.

MPI Applications
This is the place where user interact with the system. It consists of user MPI application
programs.
Point-to –Point Communication
The basic communication mechanism of MPI is classified as Point-to-Point
communication and collective communication. In Point-to-Point communication the
transmittal of data taken place between a pair of processes, one side sending, the other
receiving. Further this is classified as Blocking and Non-blocking calls.

7
Collective Communication
Collective communications transmit data among all process in a group specified by an
intra communicator object. One function, the barrier, serve to synchronize process without
passing data.
VI User Agent
A software component that enables an operating system communication facility to utilize
a particular VI Provider. The VI User Agent abstracts the details of the underlying VI NIC
hardware in accordance with an interface defined by the operating system communication
facility.
VI Kernel Agent
The Kernel Agent is a privileged part of the operating system, usually a driver, which
performs to setup and resource management functions needed to maintain a VI between
VI consumers and VI NICs. These functions include the creation/destruction of VIs, VI
connection setup/teardown, management of system memory used by the VI NIC and error
handling.
Virtual Interface(VI)
An interface between a VI NIC and a process allowing a VI NIC direct access to the
process memory. A VI consists of a pair of Work Queues and for Send operations and one
for receive operations. The queues store a descriptor between the time it is posted and the
time it is done. A pair of VIs are associated using the connect operation to allow packets
sent at one VI to be received at the other
Completion Queues

Notification of completed requests can be directed to a completion queue on a per-

VI Work queue basis. This association is established when a VI is created. Once a
VI Work Queue is associated with a Completion queue, all Completion
synchronization must take place on that Completion Queue.

8
10.4 Flow chart for C-MPI - ADI

START

IS
1 1.Transport
operation 3
2.Initialize
3.Finalize
?

2
1
IS
* Yes
1.Send
IS Inititalized
2.Recv
?

No
2

Initialize VIA Close VIA

environment Environment
+
(Open (Disconnect
NIC,Register VI,Deregister
Mem ory,Create Memory,Close NIC
VI,Connect VI etc.) etc.)

Yes IS Success
?

No IS Success
No ?

Yes
Rport Error

STOP

9
*

Yes No
Is Blocking Call
?

Is any Active Call

Yes
requests m pip_via_progress
? _nblk

No
Call
m pip_via_progress
_blk
Call
m pip_via_fsend

Is m sg_len
<= Yes
No
M AX_SIZE
?

Call long_send Call eager_send

No
Is Any ERROR

yes

Report Error S

10
+

No
Yes Is Blocking call
?

No Is Unexpected Is Unexpected No
? ?

Yes Yes
Yes Is any active Is any active No
requests requests
? ?
Call
Call
mpip_via_urecv
Post request and m pip_via_uirecv
Yes
No
call
mpip_via_progress Call
_blk Post request and m pip_via_frecv
call
Call m pip_via_progress
mpip_via_frecv _nblk

No Is m sg_lemgth
Yes
<= MAX_SIZE
?

Call long_recv
Call eager_recv

No
Is any ERROR
?

Yes

Report error S

11
CONTROL FLOW FOR VIA MODULE

USER TASK
(MPI APPLICATION
PROGRAM )

API(MPI Library)
Collective
communication

Point to point communication

Mpi/api

PROTOCOL MODULE
Mpi/engine/
SHARED
VIA TCP/IP MEMORY

Mpi/engine/via tcp shm

VIPL Library
/usr/lib/libvipl.a

Function flow from API to Engine

MPI_Init(&argc, &argv)
PMPI_Init(int *argc, char ***argv)
mpip_init(argc, argv)
initconns()
RTE_get_jtbl(&cid, &task.ti_jtbl, 0, -1)
makeconns(taskinfo_t *task)
mpip_rte_taskconns(task)
getconns(task, connif, connproto)

mpip_map_functions(char *pmname, protocol_t *pt)

mpip_probe_control()
int mpip_pmname_initconns(taskinfo_t *, void **);
int mpip_pmname_makeconns(taskinfo_t *, void *);
int mpip_pmname_addconns(taskinfo_t *, void *);

12
MPI_Send

int PMPI_Send(void *buf, int count, MPI_Datatype type, int dest, int tag, MPI_Comm
comm)

MPIP_Send(void *buf, size_t count, MPI_Datatype type, int dest, int tag,
MPI_Comm comm)
mpip_getproc(c, r)
mpip_send_self(void *buf, size_t count, MPI_Datatype type, int tag,
MPI_Comm comm, int mode)
mpip_req_set_sendready(r)
mpip_req_initlock(&req)
mpip_inc_nactvproto(proc->p_proto)
mpip_atominc(mpip_nactvreqs, 1);
mpip_enqsend(proc, &req);
mpip_progress_blk(&req)
mpip_pmname_progress_blk(MPI_Request rq,int pollall)

MPI_Recv

Int PMPI_Recv(void *buf, int count, MPI_Datatype type, int src, int tag, MPI_Comm
comm, MPI_Status *status)

Int MPIP_Recv(void *buf, int count, MPI_Datatype type, int src, int tag,
MPI_Comm comm, MPI_Status *status)
mpip_search_unexpected(src, tag, comm, TRUE)
mpip_getproc(comm, src)
mpip_recv_self(msg, buf, count, type, comm, status)
pt_ urecv(process_t *, unexmsg_t *, char *, int, MPI_Datatype,
MPI_Comm, MPI_Status *)
mpip_pmname_urecv(process_t *, unexmsg_t *, char *, int,
MPI_Datatype, MPI_Comm, MPI_Status *)
mpip_inc_nactvproto(srcproc->p_proto)
mpip_atominc(mpip_nactvreqs, 1);
pt_ postedreceive(srcproc)
mpip_pmname_postedreceive(srcproc)
mpip_postedrcvany(comm)
mpip_progress_blk(&req)
mpip_pmname_progress_blk(&req)

13
MPI_Wait

PMPI_Wait(MPI_Request *request, MPI_Status *status)

mpip_badrequest(*request)
MPIP_Wait(request, status);
mpip_req_test_init(*req)
mpip_status_setempty(status)
mpip_progress_nblk()
mpip_pmname__progress_nblk()
mpip_progress_blk(*req)
mpip_pmname__progress_blk()

MPI_Finalize

PMPI_Finalize(void)
pt_closeconns(0, 0, 0)
mpip_pmname_closeconns(process_t **procs,int nprocs,void *pminfo)

All the functions name having “pmname” as a variable, are mapped to respective protocol
module by the function mpip_map_functions.For example in VIA protocol module
“pmname” is replaced by “via”.

int mpip_via_initconns(taskinfo_t *task,void **pminfo)

{
Allocate and initialize the internal data structures for new connections.
This function will create a new process table taking information from the
task_info structure.

I task_info structure process are stores depending on protocol that they are using, Unique
array for respective protocol(ex via,tcp & shm),So we choose process from the task
structure which are using VIA & build a new process table in VIA.

Than newproc table created is assigned to pminfo

int mpip_via_makeconns(taskinfo_t *task,void *pminfo)

{
call do_setup(task) to make connections
}

14
do_setup()
{

•Get the NodeID & node information structure form process table cluster
table of the task_info structure .
• Fill up tha viadev structure for rank & number of process form the task
structure
• VipOpenNic(VIADEV_DEVICE, &viadev.nic)
• VipQueryNic(viadev.nic, &nicattrs) ;
• VipCreateCQ(viadev.nic, VIADEV_CQ_SIZE, &viadev.cq)create the
completion queue for the receive
• VipCreateCQ(viadev.nic, VIADEV_CQ_SIZE, &viadev.scq)create the
completion queue for send
• VipCreatePtag(viadev.nic, &viadev.ptag)
• Allocate pool of vbufs to work with and register memory by calling
allocate_vbufs(viadev.nic, viadev.ptag,VIADEV_VBUF_POOL_SIZE)
This function will allocate chunk of vbuffs &
VIADEV_VBUF_POOL_SIZE =viadev.np*VIADEV_PREPOST_DEPTH +
VIADEV_VBUF_SPARES
So pool size depends on number of process which are using the via.Connection
with another process is done through a single VI – one per process. We need a bit
of per-connection state in addition to the VI itself. This per-connection state is
kept in a viadev_connection_t structure
• Create the VIs here, but they are not yet connected to other processes. This
is done below, in viadev_setup_communication.

for(i=0;i<mpip_nworld;i++)- For all in MPIP_COMMWORD

{
if(t->ti_procs[i].pi_proto!=via_protocol && viadev.me!=i)if the process
is not using via protocol or it is not root process(mean to say ME) than
continue;
Create the VIs here, but they are not yet connected to other
processes. This is done below, in viadev_setup_communication
Fill up the connection struct
if(viadev.me != i)
{
c->vi = viutil_create_vi(viadev.nic, viadev.cq, viadev.ptag);
prepost receives i.e prepost descriptor to receive queue
VIADEV_PREPOST_DEPTH = 20
PREPOST_VBUF_RECV(connection)
This function will post the receive on the queue for the respective
rank I & increment the local credits & prposts.
all nodes execute this code so everyone starts off with the same
amount of credit.

15
c->remote_credit = VIADEV_PREPOST_DEPTH;
c->local_credit = 0;
c->preposts = VIADEV_PREPOST_DEPTH;
}
else
{
these should be ignored for self-communication
c->vi = NULL;
c->remote_credit = 0;
c->local_credit = 0;
}
}end of for loop
• Setup the local & remote address length from the NIC attributes
• fill in the nic address and connect
• VipNSInit(viadev.nic,(VIP_PVOID)0)
• VipNSGetHostByName(viadev.nic,proc_table[viadev.me].hostname,local
_addr,0)
• ESATBLISHING THE CONNECTION
for(i=0;i<mpip_nworld;i++)
{
if(t->ti_procs[i].pi_proto!=via_protocol && viadev.me!=i)
continue;

if(viadev.me == i) /* CONNECTWAIT_TIMEOUT */
{
for(j=i+1;j<mpip_nworld;j++)
{
if(t->ti_procs[j].pi_proto!=via_protocol)
continue;
remoteid=t->ti_procs[j].pi_proc->p_gid.pg_node;
remotenode=t->ti_ctbl.node[remoteid];

VipConnectRequest(viadev.connections[j].vi,local_addr,
remote_addr, VIP_INFINITE, &remote_attrs);
}
}
else if(viadev.me > i)
{
Set up the discriminator length in local address from NIC
attributes
VipConnectWait(viadev.nic, local_addr,
VIP_INFINITE, remote_addr, &remote_attrs, &conn) ;
VipConnectAccept(conn, viadev.connections[i].vi);
}
}End for for loop i
}end of do_setup function

16
int mpip_via_addconns(taskinfo_t *task,void *newproctbl)
{
Add connections to internal structures
For all task->ti_proto[via_protocol] check whether via_flag is set to
VIA_NOT_ATTACHED if it is do than increment the count newconnections .
If the via process table itself is not created than create it & make connection.
Other wise expand the existing one.
}

int mpip_via_closeconns(process_t **procs,int nprocs,void *pminfo)

{
Do the actual work (e.g. closing sockets) of severing the physical connections to
the disconnected processes
Assert for all –no of global active request, via active requests & any receive is
pending
Call do_stop()
}
int do_stop()
{

for (i = 0; i < mpip_nworld; i++) For all in communicator world

{
if the rank is my self then continue
call VipDisconnect(viadev.connections[i].vi)
Dequeue the all preposted receives by calling
VipRecvDone(viadev.connections[i].vi,(VIP_DESCRIPTOR **)&desc)
VipDestroyVi(viadev.connections[i].vi)
}
VipDestroyPtag(viadev.nic, viadev.ptag)
VipDeregisterMem(viadev.nic,vbuf_malloc_start,vbuf_mem_handle)
vbuf_malloc_start = NULL
VipDestroyCQ(viadev.cq)
VipNSShutdown(viadev.nic)
VipCloseNic(viadev.nic)

}End of do_stop()

17
MPIP_Send(void *buf, size_t count, MPI_Datatype type,
int dest, int tag, MPI_Comm comm)
{
mpip_req_t req;
process_t *proc;
int rc;a
Get the process from the remote group for intercommunication else from local group
from by calling malloc mpip_getproc by passing communicator as an argument.
If the got process is equal to mpip_myproc than call
mpip_send_self(buf, count, type, tag, comm, 0)
{
mpip_search_posted(comm, &env) – search for the posted queue
copy data directly from send buffer to receive buffer using
mpip_typetype_copy(recvrq->r_buf, recvrq->r_count,
recvrq->r_type, buf, count, type, length
set the status to one in MPIP_Req structure by calling mpip_req_done(&recvrq).
No matching receive. Pack the message and put it on the unexpected queue.
mpip_pack(packbuf, buf, count, type)
mpip_enqunexpected(comm, &env, packbuf, 0)
if (mode == MPIP_MODE_SYNC)
return mpip_progress_blk(&req)
}
else {
Fill up the request data structure as fallows
req.r_error = MPI_SUCCESS;
req.r_state = 0;
req.r_flags = MPIP_REQ_SEND;
req.r_tag = tag;
req.r_comm = comm;
req.r_peer = dest;
req.r_buf = buf;
req.r_count = count;
req.r_type = type;
req.r_length = type->d_size * count;
mpip_req_set_sendready(&req);
req.r_next = 0;
Request structure also consists of buffer to be send, which is void pointer in
request structure.
Increment the number of active request using the number of active request by one
in protocol struct by calling mpip_inc_nactvproto(p)
Increment the mpip_nactreqs (global variable for a process)byone.
Enque the request in the send queue by calling mpip_enqsend(proc, &req)
Which will set the senhead & sendtail in process struct.
Now call mpip_progress_blk(&req)

18
}

mpip_via_progress_blk(MPI_Request rq,int pollall)

{
calls mpip_via_progress_nblk(pollall) umtill the request is satisfied, this
is checked by calling mpip_req_test_done(rq) which checks for status
flag in request struct should be set to MPIP_REQ_DONE.
}

mpip_via_progress_nblk(int pollall){

This function attempts to progress all connections with requests, without blocking
on any one connection.
For all process using via as an protocol (viaproctblsz – global variable updated by
mpip_via_addconns)
{
Get the via process from via proctable entry.
mpip_check_transport(process)- this function checks for whether
process->p_flagsis set to MPIP_PROC_TRANSPORTERR.
Otherwise receive the packet & process them by calling
Rc=via_recv(viaproc_t* ap)
If rc = MPI_SUCCESS than set the progress flag
Mean while Send Packets if any by process send head (p->p_sendhead)
via_send_from_mpip_queue(process_t *destproc)
than return (err==MPI_SUCCESS) ? progress:err
}

via_recv(viaproc_t* ap)
{

rtn_status=MPID_VIA_DeviceCheck();
return rtn_status;
}

MPID_VIA_DeviceCheck()
{
• Till there are entry in Completion queue of receive by calling VipCQDone
(viadev.cq, &vi, &onrecvq), call VipRecvDone(vi, (VIP_DESCRIPTOR
**)&desc) to dequeue the send queue & get descriptor
• Get the data from the descriptor (desc->DS->Local.Data.Address )& store in
the vbuf.
}

19
int via_send_from_mpip_queue(process_t *destproc){

rq=destproc->p_sendhead

IF(
Here they will check whether requested message length + size of tranfer type exceeds
MAX_DAT_SIZE
OR
viadev_eager_ok(rq->r_length + sizeof(txfr_type),&viadev.connections[mote proc
rank])
here they will check whether the remote credits after sending the buffer will become less
than VIADEV_CREDIT_PRESERVE (=10)
)
(#define MAX_DATA_SIZE ((1024*16)-64-sizeof(viadev_packet_eager_start))

viadev_packet_eager_start is a sturct holding the header, envelope and

bytes_in_this_packet)
{
if above condition is true than they will set the rendezvous send & clear the
synchronous send from r_state flag in request.

Than set the packect len to zero

}
else
set the packet_len to request->r_length

For Eagar send

For a message length that is larger than 0, load up the contents to buffer.
If the data type is contigues than call mpip_pack() which will copy the data to buffer
depending on the data type, other wise just mem copy the data to buffer.

To handle the cancel reply message set the envelope mode to CANCEL type like this
env->ev_sndreq = rq->r_cancelreq
send the data using MPID_VIA_eager_send than free the buffer.
Other wise
Fill up with request struct.
Other wise Set the envelope mode depending on the req->r_state is whether synchronous
Or ready.
Send the data using
MPID_VIA_eager_send(&(tmp_buf->snd_buf),rq->r_length,rq->r_comm->c_rank,rq-
>r_ta
g,rq->r_comm->c_sndctxt,destproc->p_gid.pg_wrank,env->ev_mode,(reqaddr_t)rq,env-
>ev_rcvreq);

20
For rendezvous send
Set the envelope for rendezvous mode & fill up the other fields of envelope than call
send_req_long_data(rq->r_length,rq->r_comm->c_rank,rq->r_tag,rq->r_comm-
>c_sndctxt,destproc->p_gid.pg_wran
k,env->ev_mode,(reqaddr_t)rq);

After sending any of the above type just call the bellow calls for the clearance
rq->r_error=MPI_SUCCESS;- set the req, r_error field to MPI_SUCCESS.
mpip_req_done(&rq);- set the status to done
mpip_atomdec(mpip_nactvreqs, 1);- decriments the no of req
mpip_dec_nactvproto(mpim_via_prototbl);- decriment the no of active
protocol
mpip_deqsend(destproc);- remove the message form send queue.

end of the via_send_from_mpip_queue

________________________()_________________________________

int MPID_VIA_eager_send(void *buf,

int64_t len,
int src_lrank,
int tag,
int context_id,
int dest_grank,
int mode,
reqaddr_t rq,
reqaddr_t rrq)
{

Get the vbuff buffer from the free pool, by calling get_vbuf()
Structure of vbuf

viadev_packet_header + Data to be
viadev_packet_envelope + transferred (buff VIP_DESCRIPTOR
bytes_in_this_packet from API) (64 bytes)
BUFFER

21
Packet header consisting of fallowing fields
typedef struct {
viadev_packet_type type; /* what type of packet is this? */
packet_sequence_t id; /* sequence number of packet. Always checked for sanity
* and later for reliability */
int vbuf_credit; /* piggybacked vbuf credit */
int source;
} viadev_packet_header;

Packet envelop consisting of fallowing field

typedef struct {
int context; /* context id -- identifies communicator */
int tag; /* MPI message tag. */
int64_t data_length; /* how many bytes in MPI message (not header). */
int src_lrank; /* to help receiver */
int packet_no; /* to help packet sequencing */
int mode ; /* Send or Recv mode */
reqaddr_t sreq;
reqaddr_t rreq;

} viadev_packet_envelope;

Fill the envelop & header information ,set the header type to
VIADEV_PACKET_EAGER_START

Fill up the descriptor information in the vbuf (as shown above) by calling
vbuf_init_sendrecv(vbuf *v, unsigned long len)

Finally send it off by calling viadev_post_send (viadev_connection_t *c, vbuf *v)

This function will decrement the remote credits & posts the descriptor to the Send Queue
by calling VipPostsend()

Than call viadev_process_send(c->vi) to send the message by polling the send

completion queue by calling VipCQDone & completing the send operation by calling
VipSendDone

________________________________()_________________________________

send_req_long_data(int64_t len,int src_lrank,int tag,int cntxt,int dest_rank,int

mode,reqaddr_t rq)

This function is same as MPID_VIA_eager_send but here we will set the hader type as
VIADEV_PACKET_LONG_REQUEST & we wont send the data only request is made

22
for transfer.

MPI_Recv

int
PMPI_Recv(void *buf, int count, MPI_Datatype type,
int src, int tag, MPI_Comm comm, MPI_Status *status)

int
MPIP_Recv(void *buf, int count, MPI_Datatype type,
int src, int tag, MPI_Comm comm, MPI_Status *status)
{
check if( comm->c_unexhead) that means check for unexpected queue head is
present
To search unexpected queue list call mpip_search_unexpected(src, tag,
comm, TRUE)
This function searches by taking the src & tag as parameter, if they matched than it
returns the msg.
Than get the rank of the process from which process wanted to recv from msg struct
(unexmsg_t).
By using the rank get the process by calling srcproc=mpip_getproc(comm, rank)
if (srcproc == mpip_myproc)
than call mpip_recv_self(msg, buf, count, type, comm, status)
else
pt_urecv(srcproc, msg, buf, count, type, comm, status);
which is nothing but mpip_via_urecv

If at all unexpected queue head is empty than setup the receive request struct then call
mpip_req_set_recvhead(&req) to set the r_state field of the request struct to r_state
MPIP_REQ_RECVHEAD

Post the receive request

If ( rank >= 0)
{
if (srcproc != mpip_myproc) {
mpip_inc_nactvproto(srcproc->p_proto); - increment act protocol count
mpip_atominc(mpip_nactvreqs, 1);- increment the no of req count
srcproc->p_proto->pt_postedreceive(srcproc) this calls
mpip_via_ postedreceive
}
}
else

23
{
mpip_postedrcvany(comm)
Called when a receive is posted with a source of MPI_ANY_SOURCE. This
function handles MPI_ANY_SOURCE using two different methods. In the first method,
mpip_nrcvany is incremented and all the PM's that are part of the communicator are
instructed to turn on receives from everywhere. In the second method, the PM's are
instructed to turn on receive from all connections that are part of the communicator.
This function inturn calls pt_postedrcvany(comm.) for all protocols that are part of that
communicator, that is mpip_via_postedrcvany(comm.)

check if there is anything in posted recv tail

than update the queue by adding the present request to the tails
otherwise make head & tail point to same request

Progress until the receive is complete

mpip_progress_blk(&req)
this same as calling mpip_via_progress_blk

set the fields of status struct

}end of MPIP_Recv

________________________________()__________________________

mpip_via_postedreceive(process_t *p){
Update polling state in response to API posting a receive.
Process->p_nrecvs++;
}

mpip_recv_self(msg, buf, count, type, comm, status)

{
Here they will calculate the length of the data to be received by using
length = count * type->d_size
than copy the envelope from msg
than compare the envelope length & calculated length, if not equal than there is
truncation in recv.
mpip_unpack(buf, umsg->um_buf, length, count, type)
this function will unpack the packet & copies the data to buffer
Set the status structure
if (mpip_mode_test_sync(env->ev_mode)) {
whether envelop mode is synchronous
sndreq = (MPI_Request) env->ev_sndreq;
mpip_req_done(&sndreq);this will set the r_state flag of the request struct
}

24
}

mpip_via_postedrcvany(MPI_Comm comm)
{

update every connection that is part of this communicator

For all the process under the group
if (process->p_protoidx == via_protocol) {
mpip_inc_nactvproto(process->p_proto);this will increment the number of
active request for the particular protocol.
process->p_nrecvs++;
}
}

____________________________________()___________________________

int mpip_via_urecv(process_t *srcproc,unexmsg_t *umsg,char *buf,int count,

MPI_Datatype type,MPI_Comm comm,MPI_Status *status)
{
Complete a blocking receive which matched an unexpected msg
Get the envelop from the msg & calculate the length from type->d_size*count
If calculated length is more than or equal to envelop length than set request error
field to MPI_SUCCESS or give truncation error.

if(mpip_mode_test_rts(env->ev_mode))
{

Set up the request structure

mpip_req_set_recvhead(&req) this function will set the request->r_state |
= MPIP_REQ_RECVHEAD

mpip_inc_nactvproto(mpim_via_prototbl);-increment the numaber of

active request for a via protocol.
mpip_atominc(mpip_nactvreqs,1);- increments the global variable
number of requests.

Allocate a new txfr_type buffer for receiving the data & set the envelop of
that buffer.

Set the envelop mode to MPIP_MODE_ACK

Initialize the req.r_data fields like such as

req.r_data->bytes_recvd=0;
req.r_data->packets_recvd=0;
req.r_data->packer=0;

25
Depending on the type of data contagious or or other set up the send buffer
of txfr_type buffer.

Set up the connection table from the process connections(viadev.connection)

table for rank in MPI_COMM_WORLD entery (viadev.connection[ srcproc-
>p_gid.pg_wrank])

Now we will send the reply for the long data message by calling
send_reply_for_long_data( int src_lrank,int dest_rank,viadev_connection_t
*c,int context_id,int mode,reqaddr_t sreq,reqaddr_t rreq)
This function will setup the packet header type as
VIADEV_PACKET_LONG_REPLY
& send the ack msg to req.r_peer by using
viadev_post_send(c,v);-which calls VIpSendPost
rc = viadev_process_send(c->vi);-VipCQDone & VipSendDone

mpip_via_postedreceive(srcproc);
mpip_via_progress_blk(&req,1);

Check the status & assign the status field from the req uest struct returned by
progress_blk call.

Free the buffer

Return the request status
}
else
{
Check the calculated length with the envelop length if miss match there is
truncation error.
Set the status fields by using the envelop fields.
Check for whether unexpected message is contiguous else call
mpip_unpack(buf, umsg->um_buf, len, count, type)
Check whether envelop mode is set to synchronous
So, synchronous message, return an acknowledgment
Set return envelop
mpip_req_set_sync(rq);
mpip_req_set_sendack(rq);
mpip_mode_set_ack(renv->ev_mode);
mpip_mode_set_sync(renv->ev_mode);
Set up the connection structure for sending the acknowledgement
send_ack(renv->ev_rank,env->ev_rank,c,renv->ev_sndreq,renv->ev_rcvreq)
Free the temp buffer
}

26
int mpip_via_fsend(process_t *dest,char *buf,size_t count,MPI_Datatype dtype,int tag,
MPI_Comm comm,int mode)
{
Do a fast send of data.Do not return until send completes.Note that when fsend is called, no
active request outstanding.
Do a fast send of data.Do not return until send completes.Note that when fsend is
called, no active request outstanding
• Check for group whether it is locl group or remote group depending on
that assign the group from comm. Struct.
• For all the group process find the destination process
• Get the via process information from the protocol private information of
destination process
• Calculate the length from count & data type
• Fill up the request structure from arguments
• Set the request r_state to MPIP_REQ_SENDREADY ready by calling
mpip_req_set_sendready(&req)
• Check the mode whether it is sync or ready send depending on that set the
request flag
• mpip_inc_nactvproto(mpim_via_prototbl);
• mpip_atominc(mpip_nactvreqs, 1);
• mpip_enqsend(dest,&req);
• via_send_from_mpip_queue(dest)
• Till the request flag is set to MPIP_REQ_DONE call via-recv(via_proc)
}

int mpip_via_frecv(process_t *srcproc,char *buf,int count,MPI_Datatype dtype,

int tag,MPI_Comm comm,MPI_Status *status)

{
This function will do blocking receives of data. It will not return until all the data has
been read. This function will not be called from the threadsafe library.

Functionality of this code is disabled

}

27
5.0 Data Design

• Internal software data structure.

11.1b C-MPI.

typedef struct{
char buffer[VBUF_BUFFER_SIZE];
VIP_DESCRIPTOR desc; /* 64 Bytes*/
}vbuf;

typedef struct viaproc{

process_t *via_proc;
int via_flags;
via_unex_t *head;
int sequence_sent;
int sequence_recv;
}viaproc_t;

typedef struct _viadev_connection_t

{
VIP_VI_HANDLE vi;
int global_rank;
int remote_credit; /* how many vbufs I can consume on remote end.*/
int local_credit; /* accumulate vbuf credit locally here */
int preposts; /* number of vbufs currently preposted */
/* These are not necessary for reliable connections, but we use
* them anyway as a sanity check
* They will become necessary when we handle unreliable connections
*/

int next_packet_expected; /* for sequencing (starts at 1) */

int next_packet_tosend; /* for sequencing (starts at 1) */
int bytes_received;
int bytes_sent;
/* This field is used for managing preposted receives. It is the
* number of rendezvous packets (r3/rput) expected to arrive for
* receives we've ACK'd but have not been completed. In general we
* can't prepost all of these vbufs, but we do prepost extra ones
* to allow sender to keep the pipe full. As packets come in this
* field is decremented. We know when to stop preposting extra
* buffers when this number goes to zero.
*/
int rendezvous_packets_expected;
/* these fields are used to remember data transfer operations
* that are currently in progress on this connection. The

28
* send handle list is a queue of send handles representing
* in-progress rendezvous transfers. It is processed in FIFO
* order (because of MPI ordering rules) so there is both a head
* and a tail.
* The receive handle is a pointer to a single
* in-progress eager receive. We require that an eager sender
* send *all* packets associated with an eager receive before
* sending any others, so when we receive the first packet of
* an eager series, we remember it by caching the rhandle
* on the connection. */
MPIR_SHANDLE *shandle_head; /* "queue" of send handles to process */
MPIR_SHANDLE *shandle_tail;
MPIR_RHANDLE *rhandle; /* current eager receive "in progress" */
/* these two fields are used *only* by MPID_DeviceCheck to
* build up a list of connections that have received new
* flow control credit so that pending operations should be
* pushed. nextflow is a pointer to the next connection on the
* list, and inflow is 1 (true) or 0 (false) to indicate whether
* the connection is currently on the flowlist. This is needed
* to prevent a circular list.
*/
struct _viadev_connection_t *nextflow;
int inflow;
struct unexpected *unexpectedQ;
} viadev_connection_t;

typedef struct
{
int np; /* number of processes total */
int me; /* my process rank */
int global_id; /* global id of this parallel app */
VIP_NIC_HANDLE nic; /* single NIC */
VIP_CQ_HANDLE cq; /* single completion queue for this nic */
VIP_PROTECTION_HANDLE ptag; /* single protection tag for all memory
reistration */
VIP_ULONG maxtransfersize; /* max send/rdma size */
viadev_connection_t *connections; /* array of VIs connected to other processes */
}viadev_info_t
viadev_info_t viadev;

typedef struct
{
viadev_packet_type type; /* what type of packet is this? */
packet_sequence_t id; /* sequence number of packet. Always check ed for */
/*sanity and later for reliability */
int buf_credit; /* piggybacked vbuf credit */
int source;
} viadev_packet_header;

typedef struct {
int context; /* context id -- identifies communicator */

29
int tag; /* MPI message tag. */
int data_length; /* how many bytes in MPI message (not header). */
int src_lrank; /* to help receiver */
int packet_no; /* to help packet sequencing */
int mode ; /* Send or Recv mode */
reqaddr_t sreq;
reqaddr_t rreq;
} viadev_packet_envelope;

typedef struct {
viadev_packet_header header;
viadev_packet_envelope envelope;
int bytes_in_this_packet;
} viadev_packet_eager_start;

typedef struct
{
viadev_packet_header header;
viadev_packet_envelope envelope;
#ifdef VIADEV_RGET_SUPPORT
void *buffer_address; /* address of pinned buffer on sender.
* NULL if not pinned */
VIP_MEM_HANDLE memhandle; /* memory handle for buffer (one handle
* for whole buffer) */
#endif
reqaddr_t sreq; /* identifies the send handle and is sent
* by receiver in reply */
int descriptor_required ;
} viadev_packet_rendezvous_start;

typedef struct {
viadev_packet_header header;
#ifdef VIADEV_RPUT_SUPPORT
void *buffer_address; /* address of pinned buffer on receiver.
* NULL if not pinned */
VIP_MEM_HANDLE memhandle; /* memory handle for buffer */
#endif

reqaddr_t rreq; /* identifies receive handle */

reqaddr_t sreq; /* identifies send handle (echoed)*/
} viadev_packet_rendezvous_reply;

typedef struct {
viadev_packet_header header;
reqaddr_t rreq; /* identifies receive handle */
reqaddr_t sreq;
int context_id;
int src_lrank;
int mode;
} viadev_packet_long_reply;

30
• Global data structure

11.2b C- MPI .

struct envelope {
int ev_ctxt ;
int ev_mode ;
int ev_rank ;
int ev_tag ;
int 64_t ev_len;
reqaddr_t ev_sendreq ;
reqaddr_t ev_rcvreq ;
};

struct procinfo {
struct process *pi_proc; /* process table entry */
char pi_addr[RTE_NETADD_LEN]; /* IP address */
struct procinfo *pi_next; /* next in protocol */
struct procinfo *pi_master; /* next host master */
int pi_proto; /* protocol */
int pi_ptrank; /* rank in protocol */
int pi_nodeid; /* cluster nodeid */
};

struct taskinfo {
int ti_flags;
#define TASK_WORLD 0x1
#define TASK_MERGE 0x2
#define TASK_CLIENT 0x4
#define TASK_SERVER 0x8
#define TASK_PARENT 0x10
#define TASK_CHILD 0x20
#define TASK_ROBUST 0x40
MPI_Comm ti_comm;
MPI_Comm ti_icomm;
int ti_sd;
rte_ctbl_t ti_ctbl;
rte_proc_t *ti_jtbl;
rte_port_t *ti_port;
int ti_index;
int ti_nlocal;
int ti_nremote;

31
int ti_nhosts;
int ti_nprotos;
struct procinfo *ti_procs;
struct procinfo *ti_proto[NPROTO];
int ti_nproto[NPROTO];
int *ti_lenvsizes[NPROTO]; /* server environment sizes */
void *ti_lenvs[NPROTO]; /* server environments */
int *ti_renvsizes[NPROTO]; /* client environment sizes */
void *ti_renvs[NPROTO]; /* client environments */
int ti_myenvsize[NPROTO];
void *ti_myenv[NPROTO];
MPI_Info ti_info;
};

struct mpip_req {
MPI_Comm r_comm; /* communicator */
MPI_Datatype r_type; /* message datatype */
int r_tag; /* message tag */
int r_peer; /* peer process rank in comm */
int r_flags; /* request flags */
int r_state; /* request state */
int r_fhandle; /* Fortran handle */
size_t r_count; /* message count */
union vf r_bufquery; /* message buffer or queryfn */
int64_t r_length; /* data length received/sent */
reqaddr_t r_peereq; /* peer request */
#define r_cancelreq r_peereq /* request to be cancelled */
union vf r_datafree; /* PM specific data or freefn */
struct mpip_req *r_next; /* for linking in queues */
struct process *r_proc; /* peer process */
int r_error; /* error class */
int r_lock; /* request spin lock */
struct mpip_req *r_bsend; /* background bsend request */
int r_pertag; /* persistent tag */
uint r_cookie; /* error checking cookie */
void_fn *r_cancelfn; /* cancelfn */
#ifndef __sparcv9
char r_pad[32];
#endif
};

struct mpip_comm
{
int c_flags; /* state flags */
int c_fhandle; /* Fortran handle */
int c_sndctxt; /* send context */
int c_rcvctxt; /* receive context */
int c_rank; /* caches rank in local group */
int c_size; /* caches size of local group */
#define MPIP_COMM_INTERCOMM 0x100 /* intercommunicator */
#define MPIP_COMM_CARTESIAN 0x200 /* cartesian topology */

32
#define MPIP_COMM_GRAPH 0x400 /* graph topology */
#define MPIP_COMM_MARKED 0x800
#define MPIP_COMM_TRANSPORTERR 0x1000
#define MPIP_COMM_ROBUST 0x2000
#define MPIP_COMM_SUPER_WIN 0x4000
struct mpip_group *c_group; /* local group */
struct mpip_group *c_rgroup; /* remote group */
MPI_Request c_posthead; /* posted queue head */
MPI_Request c_posttail; /* posted queue tail */
struct unexmsg *c_unexhead; /* unexpected queue head */
struct unexmsg *c_unextail; /* unexpected queue tail */
struct mpip_comm *c_intra; /* local group intracomm */
int c_refcnt; /* reference count */
int c_inuse; /* in use count */
uint c_colloffset; /* general buffer pool offset */
long long c_localbval; /* local barrier value */
void *c_gbp; /* general buffer pool */
cluster_t c_cluster; /* collective cluster info */
int c_latency; /* network latency (usec) */
int c_bandwidth; /* network bw (Mb/sec) */
int c_tcp; /* TCP in use */
#ifdef THREADSAFE
mutex_t c_queuelock; /* posted/unexpected Q lock */
mutex_t c_collock; /* collective operation lock */
#endif
int c_proto[NPROTO]; /* protocols in use */
int c_nprotos; /* # of protocols in use */
int c_depth; /* binary tree depth */
struct attributes c_attrs; /* attributes */
int c_nedges; /* # of graph edges */
#define c_ndims c_nedges /* # of cartesian dimensions */
int *c_index; /* graph indices */
#define c_dim c_index /* cartesian dimensions */
int *c_period; /* cartesian periods */
int *c_edge; /* graph edge */
#define c_coord c_edge /* cartesian coordinates */
struct mpip_errhdlr *c_errhdlr; /* error handler */
void *c_super; /* object the comm underlies */
#ifdef THREADSAFE
mutex_t c_statelock; /* lock for updating state */
#endif
char c_name[MPI_MAX_OBJECT_NAME];
uint c_cookie; /* error checking cookie */
};

struct unexmsg {
struct envelope um_env; /* message envelope */
char *um_buf; /* message buffer */
struct unexmsg *um_next; /* next in queue */
void *um_data; /* PM specific data */
};

33
struct protocol {
uint pt_nactv; /* # active requests */
uint pt_nprocs; /* # of processes */
uint pt_mtu;
int pt_loaded; /* 0=PM not loaded, 1=PM loaded */
uint pt_flags;
int pt_flags_revs[32];
#ifdef THREADSAFE
mutex_t pt_lock;
#endif
int (*pt_initialize)(void);
int (*pt_initconns)(taskinfo_t *, void **);
int (*pt_makeconns)(taskinfo_t *, void *);
int (*pt_addconns)(taskinfo_t *, void *);
int (*pt_completeconns)(taskinfo_t *);
int (*pt_remconns)(process_t **, int, void **);
int (*pt_closeconns)(process_t **, int, void *);
int (*pt_undoconns)(taskinfo_t *, void *);
int (*pt_fsend)(process_t *, char *, size_t, MPI_Datatype,
int, MPI_Comm, int);
int (*pt_frecv)(process_t *, char *, int, MPI_Datatype,
int, MPI_Comm, MPI_Status *);
int (*pt_urecv)(process_t *, unexmsg_t *, char *, int,
MPI_Datatype, MPI_Comm, MPI_Status *);
int (*pt_uirecv)(process_t *, unexmsg_t *, MPI_Request);
int (*pt_progress_blk)(MPI_Request, int);
int (*pt_progress_nblk)(int);
void (*pt_postedreceive)(process_t *);
void (*pt_unpostedreceive)(process_t *);
void (*pt_queuedsend)(process_t *);
void (*pt_dequeuedsend)(process_t *);
void (*pt_postedrcvany)(MPI_Comm);
void (*pt_unpostedrcvany)(MPI_Comm);
void (*pt_matchedrcvany)(MPI_Comm, process_t *);
int (*pt_getevents)(void **, uint *, uint *, uint *);
int (*pt_getpolls)(struct pollfd **, int *, int *);
void (*pt_error_request)(MPI_Request);
int (*pt_finalize)(void);
int (*pt_ping)(MPI_Comm, int);
};

34
Component Design

Name of the component/module : CMPI/Connections.

• Reference to requirement number : CMPI-REQ-1.a

• Function name. : mpip_via_initconns

 Description of the functionality : Allocate and initialize the
internal data structures for new
connections.
• Procedure for component/module: None.
• Component interface description : int mpip_via_initconns( taskinfo_t *task,
void *pminfo)
Parameters : task - task descriptor.
pminfo - PM specific information(out)
Return : MPI_Error class

• Algorithmic description
1. Allocate Memory for data structures.
2. Initialize data structures

• Restrictions/limitations : None.

• Global data definitions : None

• Local data structures : procinfo_t,rte_proc_t,viaproc_t,rte_conn_t
• Description in pseudo code : None.
• Reusable Modules/Code (if any) : None.

10. Name of the component/module : CMPI/Connections.

• Reference to requirement number : CMPI-REQ-1.b
• Function name. : mpip_via_addconns
• Description of the functionality : Add connections to internal structures
• Procedure for component/module : None.
• Component interface description : int mpip_via_addconns( taskinfo_t *task, void
*newproctbl)
Parameters : task - task descriptor.
newproctbl - Process table contains new
connections .
Return : MPI error class

• Algorithmic description

35
Once new connections are made add this to internal structures.
• Restrictions/limitations: None.

• Global data definitions : None

• Local data structures : procinfo_t.
• Description in pseudo code : None.
• Reusable Modules/Code (if any) : None.

11. Name of the component/module : CMPI/Connections.

• Reference to requirement number : CMPI-REQ-1.c
• Function name. : mpip_via_makeconns
• Description of the functionality : Implements connection part of
initialization which intern calls do_setup
function.

• Procedure for component/module : None.

• Component interface description : int mpip_via_makeconns( taskinfo_t *task,
void *pminfo)
Parameters : task - task descriptor.
pminfo - Used to get the list of process
Return : MPI_SUCCESS

• Algorithmic description
• Initialize device specific structures.
• Call do_setup function for VIA initialization
• Restrictions/limitations : None.

• Global data definitions : mpip_prototbl

• Local data structures : viaproc_t,procinfo_t
• Description in pseudo code: None.
• Reusable Modules/Code (if any): None.

12. Name of the component/module : CMPI/Initialize.

• Reference to requirement number : CMPI-REQ-2
• Function name. : do_setup
• Description of the functionality : This function performs necessary
initialization for C-MPI ADI and
establishes the connection between VI’s.
• Procedure for component/module : None.
• Component interface description : int do_setup( taskinfo_t *t)

Parameters : task - task descriptor.

Return : VIP_SUCCESS or Error class

• Algorithmic description

36
1) Opening of Network interface to get NIC handle
2) Create Completion queue.
3) Create Protection tag .
4) Register memory of specified size.
5) Create Protection tag .
6) Create VI’s
7) Prepost descriptor onto the receive queue.
8) Generate unique discriminator.
9) Request for connection
10) Wait for connect request.
11) Accept the connect request and establish the connection
between VI’s.

• Restrictions/limitations : The size of memory to be registered and number of

VI’s are restricted by amount of memory availability
with machine.

• Global data definitions : viadev

• Local data structures :
VIP_NET_ADDRESS,viadev_discriminator_t,
VIP_NIC_ATTRIBUTES,
VIP_VI_ATTRIBUTES,
VIP_CONN_HANDLE
• Description in pseudo code : None.
• Reusable Modules/Code (if any) : None.

13. Name of the component/module : CMPI/Connections.

• Reference to requirement number : CMPI-REQ-1.d
• Function name. : mpip_via_remconns,
• Description of the functionality : In this function PM must
remove from its internal table all
processes in comm. Which have become
disconnected, i.e. whose connection
count has become zero. The physical
connections are not actually broken
here.
• Procedure for component/module: None.
• Component interface description : int mpip_via_remconns( process_t **procs,
int nprocs , void *pminfo)
Parameters : procs - process related information.
Nprocs – Number of processes.
pminfo - PM specific information(out)
Return : Opaque PM specific information.

• Algorithmic description
1 Check for connection count.

37
1. If connection count is zero removes the process information from the
table i.e. logically disconnect the processes.

• Restrictions/limitations : None.

• Global data definitions : None

• Local data structures : viaproc_t
• Description in pseudo code: None.
• Reusable Modules/Code (if any): None.

14. Name of the component/module : CMPI/Connections

• Reference to requirement number : CMPI-REQ-1.e
• Function name. : mpip_via_closeconns

• Description of the functionality : Physically disconnect the

endpoints Which intern calls do_stop
function.
• Procedure for component/module: None.
• Component interface description : int mpip_via_closeconns( process_t
**procs, int nprocs void *pminfo )
Parameters : procs- Array of procs to close connection
nprocs - # connection to be disconnected
pminfo - PM specific information(out)
Return : MPI_Error class

• Algorithmic description
1. Check the internal table for logically disconnected processes.
2. Call do_stop for physically disconnect the connections.

• Restrictions/limitations: None.

• Global data definitions : None

• Local data structures : viaproc_t
• Description in pseudo code: None.
• Reusable Modules/Code (if any): None.

15. Name of the component/module : CMPI/Finalize.

• Reference to requirement number : CMPI-REQ-5
• Function name. : do_stop
• Description of the functionality : This function perform reverse
operation as done in do_start( )
• Procedure for component/module: None.
• Component interface description : int do_stop()
Parameters : None
Return : MPI_SUCCESS or Error class
• . Algorithmic description

38
1 Disconnect VI’s.
2 Destroy VI’s.
3 Destroy protection tag.
4 Deregister memory.
5 Destroy completion queue.
6 Close NIC

• Restrictions/limitations : None.
• Global data definitions : viadev
• Local data structures : VIP_DESCRIPTOR
• Description in pseudo code: None.
• Reusable Modules/Code (if any): None.

16. Name of the component/module : CMPI/Send

• Reference to requirement number : CMPI-REQ-3
• Function name. : mpip_via_fsend
• Description of the functionality : This is a fast send function. This
do not return until send completes.
When this function is called no
outstanding active requests.
• Procedure for component/module: None
• Component interface description : int mpip_via_fsend(process_t *dest,char
*buf,size_t count,MPI_Datatype dtype,int
tag,MPI_Comm comm.,int mode )
Parameters : dest – destination process
buf – message buffer
count – message count
dtype – message data type
tag – message tag
comm. – communicator
mode – ready, synchronous, standard

Return : MPI Error class

• Algorithmic description
• Select the protocol
• Call for eager send or long send

• Restrictions/limitations : None.

• Global data definitions : viadev

o Local data structures : viaproc_t,envelope_t, mpip_req_t
• Description in pseudo code: None.
• Reusable Modules/Code (if any): None.

39
17. Name of the component/module : CMPI/Send.
• Reference to requirement number : CMPI-REQ-3
• Function name. : MPID_VIA_long_send
• Description of the functionality : This function is used to send messages
of long length. Before sending actual
data sender sends hand shaking message
to receiver. Once sender got
acknowledgement sender starts sending
data packets to destination process.

• Procedure for component/module: None.

• Component interface description : int MPID_VIA_long_send(void *buf , int
len,int src_lrank, int tag, int context_id, int
dest_grank, reqaddr_t rq,reqaddr_t rrq) )
Parameters : buf – message buffer
len – message length
src_lrank – sender local rank
tag – message tag
context_id – sender context
dest_grank – destination process global rank
rq – sender request
rrq – receiver request
Return : MPI Error class

• Algorithmic description
1. Send handshaking message to receiver.
2. Wait for acknowledgement.
3. Divide the long message as packets
4. If receiver is ready post descriptors on send queue.

• Restrictions/limitations: None.

• Global data definitions : viadev, viadev_connection_t

• Local data structures : viadev_packet_eager_start,
viadev_packet_rendezvous_reply
• Description in pseudo code: None.
• Reusable Modules/Code (if any): None.

18. Name of the component/module : CMPI/Send.

• Reference to requirement number : CMPI-REQ-3
• Function name. : MPID_VIA_eager_send

• Description of the functionality : This function is used to send small messages .

40
Data is to be transferred with single packet
• Procedure for component/module: None.
• Component interface description : int MPID_VIA_eager_send(void *buf , int
len,int src_lrank, int tag, int context_id, int
dest_grank, int mode reqaddr_t rq,reqaddr_t
rrq) )
Parameters : buf – message buffer
len – message length
src_lrank – sender local rank
tag – message tag
context_id – sender context
dest_grank – destination process global rank
mode – synchronous, ready, standard
rq – sender request
rrq – receiver request
Return : VIP_SUCCESS or Error class

• Algorithmic description
1. Get the buffer.
2. Copy to internal buffer
3. Post descriptor.

• Restrictions/limitations: None.

• Global data definitions : viadev, viadev_connection_t

• Local data structures : viadev_packet_eager_start,
viadev_packet_rendezvous_reply
• Description in pseudo code : None.
• Reusable Modules/Code (if any) : None.

19. Name of the component/module : CMPI/Recv

• Reference to requirement number : CMPI-REQ-4
• Function name. : mpip_via_frecv
• Description of the functionality : This is blocking receive function . It will
not return until complete data has been
received.
• Procedure for component/module: None.
• Component interface description : int mpip_via_frecv(process_t *srcproc, char
*buf, int count, MPI_Datatype dtype, int tag,
MPI_Comm comm., MPI_Status *status )
Parameters : srcproc – Message source process
buf – receiver message buffer
count – message count
dtype – message data type
tag – message tag
comm. – communicator
status – message received status.
Return : MPI Error class

41
• Algorithmic description
1. Poll on CQ
2. Get buffer.
3. Check for acceptability.
4. If accepted select protocol else put it in unexpected queue.

• Restrictions/limitations: None.

• Global data definitions : None

• Local data structures : viaproc_t, mpip_req_t
• Description in pseudo code: None.
• Reusable Modules/Code (if any): None.

20. Name of the component/module : CMPI/Recv.

• Reference to requirement number : CMPI-REQ-4
• Function name. : viadev_incoming_long_data
• Description of the functionality : This function is used to receive long
messages . If sender wants reply it
prepost the descriptors and send an
acknowledgement
• Procedure for component/module: None.
• Component interface description : int viadev_incoming_long_data(vbuf *v
,viadev_connection_t *c,
viadev_packet_eager_start * header)
Parameters : v – VIA buffer structure
c– VIA connection structure.
header – Packet header
Return : VIP_SUCCESS or Error class

• Algorithmic description
• Search Unexpected queue.
• If found call mpip_via_urecv()
2. Send ACK to sender
3. Receive packets
4. Get buffer
5. Copy data from internal buffer to user buffer.
6. Prepost descriptors.

• Restrictions/limitations: None.

• Global data definitions : MPI_Request

• Local data structures : process_t, reqaddr_t
• Description in pseudo code : None.
• Reusable Modules/Code (if any) : None.

42
21. Name of the component/module : CMPI/Recv.
• Reference to requirement number : CMPI-REQ-4
• Function name. : viadev_incoming_eager_start
• Description of the functionality : This function is used to receive small
messages.
Parameters : v – VIA buffer structure
c– VIA connection structure.
header – Packet header
Return : VIP_SUCCESS or Error class
• Procedure for component/module: None.
• Component interface description : int viadev_incoming_eager_start(vbuf *v
,viadev_connection_t *c,
viadev_packet_eager_start * header)
• Algorithmic description
1. Search Unexpected queue.
2. If expected call function mpip_via_urecv().
3. If not found get data from descriptor.
4. Copy data from internal buffer to user buffer.
5. Prepost descriptors.

• Restrictions/limitations : None.

• Global data definitions : MPI_Request

• Local data structures : process_t, reqaddr_t,
envelope_t,viaproc_t
• Description in pseudo code : None.
• Reusable Modules/Code (if any) : None.

22. Name of the component/module : CMPI/Recv.

• Reference to requirement number : CMPI-REQ-4
• Function name. : mpip_via_urecv
• Description of the functionality : Completes the unexpected matched
receive. This is blocking receive
messages.
• Procedure for component/module: None.
• Component interface description : int mpip_via_urecv(process_t *srcproc,
unexmsg_t *umsg, char *buf, int count,
MPIP_Datatype type,MPI_Comm
comm.,MPI_Status *status)
Parameters : srcproc – Source process
Umsg – Unexpected message.
buf – Message buffer.
count – Message count
type – Message data type
comm. – Communicator
status – Message received status (out)
Return : VIP_SUCCESS or Error class

43
• Algorithmic description
1. If long data send ACK to sender.
2. Get the data from receive queue.
3. Get buffer
4. copy data from internal buffer to user buffer
5. Prepost descriptor on receive queue

• Restrictions/limitations: None.

• Global data definitions : MPI_Request

• Local data structures : viaproc_t, mpip_req_t,envelope_t
• Description in pseudo code: None.
• Reusable Modules/Code (if any): None.

23. Name of the component/module : CMPI/Recv.

• Reference to requirement number : CMPI-REQ-4
• Function name. : mpip_via_uirecv
• Description of the functionality : This function to be called when a non
blocking receive matches an unexpected
message.
• Procedure for component/module: None.
• Component interface description : int mpip_via_uirecv(process_t *srcproc,
unexmsg_t *umsg, MPI_Request rq)
Parameters : srcproc – Source process
umsg – matched unexpected message .
rq - non blocking receive request
Return : VIP_SUCCESS or Error class

• Algorithmic description
1. If long data send ACK to sender.
2. Get the data from receive queue.
3. Get buffer
4. copy data from internal buffer to user buffer
5 Prepost descriptor on receive queue

• Restrictions/limitations: None.

• Global data definitions : MPI_Request

• Local data structures : viaproc_t, envelope_t.
• Description in pseudo code: None.
• Reusable Modules/Code (if any): None.

44
24. Name of the component/module : CMPI/Send/Recv
• Reference to requirement number : CMPI-REQ-4,CMPI-REQ-5
• Function name. : mpip_via_progress_blk
• Description of the functionality :
This is blocking function which keep
calling mpip_via_progress_nblk till
request is completed.
• Procedure for component/module: None.
• Component interface description : int mpip_via_progress_blk(MPI_Request

Parameters : rq – Request to block on.

Pollall – poll all connections.
Return : Error class
• Algorithmic description
1. Check for completion.
2. If not completed call mpip_via_progress_nblk()
3 If completed return Success.

• Restrictions/limitations : None.

• Global data definitions : viaconfig

• Local data structures : None

• Description in pseudo code: None.

• Reusable Modules/Code (if any): None.

25. Name of the component/module : CMPI/Send/Recv

• Reference to requirement number : CMPI-REQ-4,CMPI-REQ-5
• Function name. : mpip_via_progress_nblk
• Description of the functionality : This is nonblocking function which
attempt to progress all connections with requests without blocking on any one
connection.
Parameters : pollall – Poll all the connections.
Return : 0 – No progress
-1 – progress made no errors
>0 - An error has occurred and return value is MPI error
class
• Procedure for component/module: None.
• Component interface description : int mpip_via_progress_nblk(int pollall)
• Algorithmic description
• Poll on completion queue.
• Select the protocol.
• Call necessary function depending upon data size

• Restrictions/limitations : None.

• Global data definitions : None

45
• Local data structures : process_t, viaproc_t
• Description in pseudo code : None.
• Reusable Modules/Code (if any) : None.

6.0 Interface Design

13.2 C-MPI
• External machine interfaces: None
• External system interfaces: None
• Human interface: None.
• User interface design
User can create an executable using Makefile.
e.g .Make example
Once the executable is available then user can run the application by
following command.
mprun –np number of processes –Mf hostfile executable name
Host file contains the nodes in which user wants to run the application.
e.g. mprun –np 2 –Mf host example

46